Concurrently crawling

[palmiak]

This is the implementation of the discussion #762

It would be great if you could take a second look.

There are some things I'm having problems with:

• there are three places that throw a warning about too long lines
• I missed this part, so this is something that should be re-add

try {
            $response = $this->client->send( $request );
        } catch ( TooManyRedirectsException $e ) {
            WsLog::l( "Too many redirects from $url" );
        }

• also crawlURL() method is no longer used - so we should either remove it or deprecate

Currently the concurrency rate can be set with apply_filters( 'wp2static_concurrent_crawl_rate', 1 ) and is set to 1 as default.

I'm using the new crawler on production for WordPressowka.pl and wpowls.co.

[john-shaffer]

Excellent work!

I resolved the issues you mentioned and added a crawlConcurrency option on the advanced options page. I changed the filter name to wp2static_crawl_concurrency.

On some very small VMs (t3a.micro), I'm seeing improvements from ~10.5s to ~7.5s on a small site with crawlConcurrency=64, and from ~8m to ~6m on a larger slow site with crawlConcurrency=8. Higher concurrency values are slightly slower, but not by much.

[palmiak]

This is really an amazing improvement. So it looks like this 25-30% is a quite repeatable thing. sob., 6 lis 2021, 02:46 użytkownik John Shaffer ***@***.***> napisał:

…

Excellent work! I resolved the issues you mentioned and added a crawlConcurrency option on the advanced options page. I changed the filter name to wp2static_crawl_concurrency. On some very small VMs (t3a.micro), I'm seeing improvements from ~10.5s to ~7.5s on a small site with crawlConcurrency=64, and from ~8m to ~6m on a larger slow site with crawlConcurrency=8. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#834 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR34OQVIV46WFQXG3AQF5TUKSJJVANCNFSM5HJQN7DQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.