Limit size of article

[ptah-alexs]

Some large pages don't import completly.

Instance http://www.fanfics.me/read.php?id=46429

rules file:

title://title                                                                                                                   
strip://table[@class = 'ReadTable']                                                                                             
strip://table[@class = 'horizontal-bar']                                                                                        
strip://*[contains(@class,'hidden')]                                                                                            
strip://div[@id = 'text-width-container']                                                                                       
strip://div[@id = 'new_comment']                                                                                                
strip://table[@class = 'horizontal-bar']                                                                                        
strip://div[@id = 'read_footer']                                                                                                
strip://div[@class='totop']                                                                                                
body://h2//p[@class = 'fict']

[tcitworld]

Which parts do you miss ? I don't speak a word of russian, but it seems I fully have it (from the Chapter One title to the « Chapter published: 04/13/2014 » end).

Even though if I import it via the « add link » button inside wallabag and it tells me import has failed.
Note that I haven't got specific rules for this website, and that my server timeout is set as default.

[ptah-alexs]

Hm, maybe difference in a options. I had only approx 1/4 of original size.

[tcitworld]

Try :

• removing your rules
• (more likely) See if you can modify some Apache parameters like KeepAliveTimeout or RequestReadTimeout and increase them.

Else, there's not much you can do, except changing hosting provider.

[nicosomb]

Can you run wallabag_compatibility_test.php please?

Maybe tidy is not enabled.

[ptah-alexs]

|Test | Should Be | What You Have|
|PHP | 5.3.3 or higher | 5.5.11-2|
|XML | Enabled | Enabled, and sane|
|PCRE | Enabled | Enabled|
|Data filtering | Enabled | Enabled|
|Tidy | Enabled | Enabled|
|cURL | Enabled | Enabled|
|Parallel URL fetching | Enabled | Enabled|
|allow_url_fopen | Enabled | Enabled|
|gettext | Enabled | Enabled|

[nicosomb]

Without rules file, all the article is fetched by wallabag.
Can you give more information about your environment please?

[ptah-alexs]

what information is you need?

[tcitworld]

All the informations stated here : https://github.com/wallabag/wallabag/blob/dev/CONTRIBUTING.md#you-found-a-bug
Sorry for not replying earlier.

[tcitworld]

@ptah-alexs : Have you seen my last message ? :)

[ptah-alexs]

Sorry for long silence.

Retrieve page http://www.fanfics.me/read.php?id=46429 within and without rulefile : Fatal error: Maximum execution time of 120 seconds exceeded in /var/www/wallabag/inc/3rdparty/libraries/readability/Readability.php on line 877.

Another page http://www.fanfics.me/read.php?id=60257 save approx 30-40% of text either with and without rulefile.

Version of wallabag 1.6.1
Dedicated server with Debian GNU/Linux Sid
PHP 5.5.12-2 (cli) (built: May 12 2014 13:02:34)
Used MySQL storage
wallabag_compatibility_test.php without warnings:

Test Should Be What You Have
PHP 5.3.3 or higher 5.5.12-2
XML Enabled Enabled, and sane
PCRE Enabled Enabled
Data filtering Enabled Enabled
Tidy Enabled Enabled
cURL Enabled Enabled
Parallel URL fetching Enabled Enabled
allow_url_fopen Enabled Enabled
gettext Enabled Enabled

[tcitworld]

Can you edit your php.ini and increase the number set for the max_execution_time value until it works fully ?

Also, your system seems a bit weak, even though the page is very long, my own installation (VPS with 2 vcores) takes about 1 minute to save it. What are your hardware specifications ?

[ptah-alexs]

it's a strange - i had edit /etc/php5/apache2/php.ini, set max_execution_time in 240. In result i had see "Fatal error: Maximum execution time of 120 seconds exceeded in /var/www/wallabag/inc/3rdparty/libraries/readability/Readability.php on line 877", what's wrong?

[ptah-alexs]

I have a seedbox with AMD Athlon(tm) II X3 425 processor and 12G of memory.

[ptah-alexs]

With 1.7.0 are same things.

[tcitworld]

Hum, my fault, it's the max_input_time value you may need to increase.

Don't forget to restart/reload Apache after editing the php.ini file. ;)

[ptah-alexs]

Hmm. max_execution_time = 120, max_input_time = -1, after restart apache: Fatal error: Maximum execution time of 120 seconds exceeded in /var/www/wallabag/inc/3rdparty/libraries/readability/Readability.php on line 877. It's strange.

[tcitworld]

Have you got safe_mode enabled ? Are you sure Apache uses the right php.ini file ?

Else, I see no other option than to add set_time_limit(0); into the Readability.php file, either at the top, either just before line 877. But if that doesn't work (like if it's an infinite loop), it may freeze your server until Apache own timeout comes.

Sorry for not finding a solution. :(

[ptah-alexs]

How do I know that safe_mode enabled? I'm sure that used right php.ini. I'll try your advice.

[ptah-alexs]

With set_time_limit(0); script done successful, but save only first 50-60kb of text.

[tcitworld]

How do I know that safe_mode enabled?

It should be in php.ini if set. But it's rather unusual to have it.

Is first 50-60kb of text better than when you started with ~1/4 of the text ?

Final try : edit your Apache configuration, either directly in Apache configuration (/etc/apache2/apache2.conf) or better inside your vhost :
Add the line (or modify it) : TimeOut 300 and see if increasing the time changing anything.

[ptah-alexs]

1. Php.net says safe_mode is deprecated.
2. 60kb of 950kb text :(
3. with TimeOut 300" in apache2 config i'd seen: Fatal error: Maximum execution time of 120 seconds exceeded in /var/www/wallabag/inc/3rdparty/libraries/readability/Readability.php on line 452
   3.1 with TimeOut 600: Fatal error: Maximum execution time of 120 seconds exceeded in /var/www/wallabag/inc/3rdparty/libraries/readability/Readability.php on line 448
   3.2 with TimeOut 900: Fatal error: Maximum execution time of 120 seconds exceeded in /var/www/wallabag/inc/3rdparty/libraries/readability/Readability.php on line 452

[tcitworld]

Well, I'm sorry to say I've got no ideas left. Maybe someone will see this and have the solution, but I think you will be luckier on Stack Overflow.
Good luck anyway.

[ptah-alexs]

OK, thank you.

[lbivens]

The problem is on line 33 of ./inc/3rdparty/makefulltextfeed.php where it says "@set_time_limit(120);"
Changing the value to something like 600 has worked to upload my 481 items list.

Maybe tweaking the import function on Poche.class.php would be better than changing the time-out value. How about making a pool with the articles to import and using multiple threads so the time-out affects only a single item? I am a bit rusty on php...

Update: I just imported, with the same 600 s setting, 1771 items from another reading list... It worked like a charm!

[j0k3r]

The given site seems to be dead.
Anyway, I don't think we need to limit the size of an article because the main goal of wallabag is to save the article content to be able to read it later. So if you want to save only some parts of this article, you should better use your bookmark instead.

Closing the issue for now.
Feel free to arg if you think we are wrong and we'll re-open the issue.