New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blitz timeouts with EasyEngine [pagespeed issue] #497

Closed
htpcBeginner opened this Issue Apr 16, 2015 · 30 comments

Comments

Projects
None yet
3 participants
@htpcBeginner

htpcBeginner commented Apr 16, 2015

Did a quick comparison of the same site on two instances:

  1. with W3TC Page (memcached), Object Cache (memcached), and W3TC Browser cache enabled. - My current setup

  2. with EasyEngine Nginx fastcgi cache, Nginx Browser cache, and w3tc object cache (memcached).

Test site was a copy of real full wordpress site with more than 500 posts.

Attached are the Blitz results. EE Nginx settings are default. I am not sure if EE Nginx settings should be optimized but I am surprised to see the big difference. @rahul286 any thoughts would be much appreciated.

new-1

@rahul286

This comment has been minimized.

Show comment
Hide comment
@rahul286

rahul286 Apr 16, 2015

Member

Did you verify fastcgi cache was working correctly?

Please share URL you tested.

Member

rahul286 commented Apr 16, 2015

Did you verify fastcgi cache was working correctly?

Please share URL you tested.

@cm896

This comment has been minimized.

Show comment
Hide comment
@cm896

cm896 Apr 16, 2015

https://www.blitz.io/
I'll try to incorporate some of these tests/results into my workflow. Thanks.

cm896 commented Apr 16, 2015

https://www.blitz.io/
I'll try to incorporate some of these tests/results into my workflow. Thanks.

@htpcBeginner

This comment has been minimized.

Show comment
Hide comment
@htpcBeginner

htpcBeginner commented Apr 16, 2015

www.htpcbeginner.com was the test site.

Yes, fastcgi cache is working:
capture

Here are the links to full blitz reports:

  1. https://www.blitz.io/report/671aa8181fe71068232f458e2f6eef05
  2. https://www.blitz.io/report/671aa8181fe71068232f458e2f4bdd13
@rahul286

This comment has been minimized.

Show comment
Hide comment
@rahul286

rahul286 Apr 16, 2015

Member
  1. with W3TC Page (memcached), Object Cache (memcached), and W3TC Browser cache enabled. - My current setup

What was web server config in other case? Was it nginx/apache?

  1. with EasyEngine Nginx fastcgi cache, Nginx Browser cache, and w3tc object cache (memcached).

Did you test with pagespeed on/off? What was command used to create EE site?

If you have both setup available can I check wordpress admin and server config for both cases?

Member

rahul286 commented Apr 16, 2015

  1. with W3TC Page (memcached), Object Cache (memcached), and W3TC Browser cache enabled. - My current setup

What was web server config in other case? Was it nginx/apache?

  1. with EasyEngine Nginx fastcgi cache, Nginx Browser cache, and w3tc object cache (memcached).

Did you test with pagespeed on/off? What was command used to create EE site?

If you have both setup available can I check wordpress admin and server config for both cases?

@htpcBeginner

This comment has been minimized.

Show comment
Hide comment
@htpcBeginner

htpcBeginner Apr 16, 2015

  1. Nginx, MySQL, PHP5-FPM, Memcached, and Opcache.
  2. Nginx with pagespeed (but disabled in site config), MariaDB, PHP5-FPM, FastCGI cache, Memcached, and Opcache. ee site create htpcbeginner.com --wpfc.

I found that with pagespeed the CPU usage was 5-6 times higher. That was a brief test and did not dig deeper.

pagespeed still included in nginx.conf. Should I exclude that?

htpcBeginner commented Apr 16, 2015

  1. Nginx, MySQL, PHP5-FPM, Memcached, and Opcache.
  2. Nginx with pagespeed (but disabled in site config), MariaDB, PHP5-FPM, FastCGI cache, Memcached, and Opcache. ee site create htpcbeginner.com --wpfc.

I found that with pagespeed the CPU usage was 5-6 times higher. That was a brief test and did not dig deeper.

pagespeed still included in nginx.conf. Should I exclude that?

@rahul286

This comment has been minimized.

Show comment
Hide comment
@rahul286

rahul286 Apr 16, 2015

Member
  1. Nginx, MySQL, PHP5-FPM, Memcached, and Opcache

Was it custom nginx config or generated by EE?

pagespeed still included in nginx.conf. Should I exclude that?

Ya. Please turn off pagespeed and test again.

Member

rahul286 commented Apr 16, 2015

  1. Nginx, MySQL, PHP5-FPM, Memcached, and Opcache

Was it custom nginx config or generated by EE?

pagespeed still included in nginx.conf. Should I exclude that?

Ya. Please turn off pagespeed and test again.

@htpcBeginner

This comment has been minimized.

Show comment
Hide comment
@htpcBeginner

htpcBeginner Apr 16, 2015

Default EE site conf. OK. testing again with ps off.

I have the EE setup (2) active right now which I can give you access. If you want the other setup (1), I will have to flip the DNS (but I can provide shell access without flipping DNS).

If so, let me know how I can forward the details privately.

htpcBeginner commented Apr 16, 2015

Default EE site conf. OK. testing again with ps off.

I have the EE setup (2) active right now which I can give you access. If you want the other setup (1), I will have to flip the DNS (but I can provide shell access without flipping DNS).

If so, let me know how I can forward the details privately.

@htpcBeginner

This comment has been minimized.

Show comment
Hide comment
@htpcBeginner

htpcBeginner Apr 16, 2015

Wow!!! Big difference with pagespeed completely off. Now EE results much better and I had to run it twice just to make sure.

new-2

With EE I had 34 more errors but the response times were much improved and stable. Not sure why pagespeed slows the site down so much.

https://www.blitz.io/report/671aa8181fe71068232f458e2f6cb3bb

htpcBeginner commented Apr 16, 2015

Wow!!! Big difference with pagespeed completely off. Now EE results much better and I had to run it twice just to make sure.

new-2

With EE I had 34 more errors but the response times were much improved and stable. Not sure why pagespeed slows the site down so much.

https://www.blitz.io/report/671aa8181fe71068232f458e2f6cb3bb

@rahul286

This comment has been minimized.

Show comment
Hide comment
@rahul286

rahul286 Apr 16, 2015

Member

I just verified an EE site and it works nice!

Before pagespeed update - https://www.blitz.io/report/671aa8181fe71068232f458e2f04567c

I will be running same test with pagespeed soon.

Member

rahul286 commented Apr 16, 2015

I just verified an EE site and it works nice!

Before pagespeed update - https://www.blitz.io/report/671aa8181fe71068232f458e2f04567c

I will be running same test with pagespeed soon.

@htpcBeginner

This comment has been minimized.

Show comment
Hide comment
@htpcBeginner

htpcBeginner Apr 16, 2015

Yours is even better. How do you have such low response times, is it because rtcamp.cloudapp.net is a blank WP site? Curious to know your test with PS on Ragul. @rahul286

And why does pagespeed slow the site down - opposite of what it is expected to do.

htpcBeginner commented Apr 16, 2015

Yours is even better. How do you have such low response times, is it because rtcamp.cloudapp.net is a blank WP site? Curious to know your test with PS on Ragul. @rahul286

And why does pagespeed slow the site down - opposite of what it is expected to do.

@cm896

This comment has been minimized.

Show comment
Hide comment
@cm896

cm896 Apr 16, 2015

I think it would depend on what pagespeed is doing. I will test with pagespeed off. I know I am probably asking it to do too much. It has to parse all the output before sending it to the browser, but thought that if the content was cached, the time getting the file to PS would be minimal, or PS could cache that output, so it's not crunching each and every time.

cm896 commented Apr 16, 2015

I think it would depend on what pagespeed is doing. I will test with pagespeed off. I know I am probably asking it to do too much. It has to parse all the output before sending it to the browser, but thought that if the content was cached, the time getting the file to PS would be minimal, or PS could cache that output, so it's not crunching each and every time.

@rahul286

This comment has been minimized.

Show comment
Hide comment
@rahul286

rahul286 Apr 16, 2015

Member

Blank site should not matter as once a page is cached, it matter of nginx reading a static file from fastcgi-cache which is stored in memory (tmpfs) in EE.

Lower response time could be because of location of server may be closer to blitz test server.

Member

rahul286 commented Apr 16, 2015

Blank site should not matter as once a page is cached, it matter of nginx reading a static file from fastcgi-cache which is stored in memory (tmpfs) in EE.

Lower response time could be because of location of server may be closer to blitz test server.

@rahul286

This comment has been minimized.

Show comment
Hide comment
@rahul286

rahul286 Apr 16, 2015

Member

I also faced similar error rate with pagespeed - https://www.blitz.io/report/671aa8181fe71068232f458e2f4b10bc

I will try to check pagespeed config.

By default when ee enables pagepseed, it only enable support. One need to play with pagespeed filters and this part cannot be automated as it depends on site's markup.

Member

rahul286 commented Apr 16, 2015

I also faced similar error rate with pagespeed - https://www.blitz.io/report/671aa8181fe71068232f458e2f4b10bc

I will try to check pagespeed config.

By default when ee enables pagepseed, it only enable support. One need to play with pagespeed filters and this part cannot be automated as it depends on site's markup.

@htpcBeginner

This comment has been minimized.

Show comment
Hide comment
@htpcBeginner

htpcBeginner Apr 16, 2015

@cm896 @rahul286 Thanks for spending time on this.

htpcBeginner commented Apr 16, 2015

@cm896 @rahul286 Thanks for spending time on this.

@rahul286

This comment has been minimized.

Show comment
Hide comment
Member

rahul286 commented Apr 16, 2015

@rahul286

This comment has been minimized.

Show comment
Hide comment
@rahul286

rahul286 Apr 16, 2015

Member

Meanwhile, if anyone finds anything concrete in thread https://groups.google.com/forum/#!msg/ngx-pagespeed-discuss/vzSq8XQPGmM/9mE11D-9U8YJ or anywhere else on the web, please let us know.

Member

rahul286 commented Apr 16, 2015

Meanwhile, if anyone finds anything concrete in thread https://groups.google.com/forum/#!msg/ngx-pagespeed-discuss/vzSq8XQPGmM/9mE11D-9U8YJ or anywhere else on the web, please let us know.

@cm896

This comment has been minimized.

Show comment
Hide comment
@cm896

cm896 Apr 16, 2015

@htpcBeginner Thanks for sharing your information, the insight into the added overhead PS is adding to the average response time will help me debug and I will test with it disabled, even though the client loves the boost to the page rank score shown.

All the credit goes to @rahul286. I am only a bystander, trying to read, learn and improve. I am using the debug slides/suggestion from @rahul286 to find my client's site speed issues. Thanks @rahul286 for all the great work and suggestions, it's appreciated.

cm896 commented Apr 16, 2015

@htpcBeginner Thanks for sharing your information, the insight into the added overhead PS is adding to the average response time will help me debug and I will test with it disabled, even though the client loves the boost to the page rank score shown.

All the credit goes to @rahul286. I am only a bystander, trying to read, learn and improve. I am using the debug slides/suggestion from @rahul286 to find my client's site speed issues. Thanks @rahul286 for all the great work and suggestions, it's appreciated.

@htpcBeginner

This comment has been minimized.

Show comment
Hide comment
@htpcBeginner

htpcBeginner Apr 16, 2015

@rahul286 One last question. EE by default sets worker_rlimit_nofile to 100000. Should this be customized based on hardware?

I also have worker_processes to 2 (2 core VPS) instead of EE's auto and worker_connections to 2048 instead of EE's default 4096.

htpcBeginner commented Apr 16, 2015

@rahul286 One last question. EE by default sets worker_rlimit_nofile to 100000. Should this be customized based on hardware?

I also have worker_processes to 2 (2 core VPS) instead of EE's auto and worker_connections to 2048 instead of EE's default 4096.

@htpcBeginner

This comment has been minimized.

Show comment
Hide comment
@htpcBeginner

htpcBeginner Apr 16, 2015

Because of the above results, I will be moving to EE. Next up is removing W3TC and settling on object cache solution with either memcached or redis.

htpcBeginner commented Apr 16, 2015

Because of the above results, I will be moving to EE. Next up is removing W3TC and settling on object cache solution with either memcached or redis.

@rahul286

This comment has been minimized.

Show comment
Hide comment
@rahul286

rahul286 Apr 16, 2015

Member

I guess this thread has answer - https://developers.google.com/speed/pagespeed/module/downstream-caching. I will try to test this tomorrow. But if anyone tries and gets it working, please share config/results.

EE's copy of nginx already has ngx_cache_purge purge module compiled in it so that step can be skipped.

About other questions:

I also have worker_processes to 2 (2 core VPS) instead of EE's auto and worker_connections to 2048 instead of EE's default 4096.

Auto always means equal to number of cores, which is recommended. In your case, it would have been 2 only with auto.

EE by default sets worker_rlimit_nofile to 100000. Should this be customized based on hardware?

Ya. I do not have formula for it but I prefer to set higher limits. My idea is to not let a server hit its own limit before OS limits. But again, I set it randomly and it worked. Before that my load tests were failing quite early.

Next up is removing W3TC

We also want to remove W3TC and hence added support for pagespeed. If pagespeed works nicely, we can use it for minify and CDN.

I loved W3TC in early days but it seems bloated now. So I also want to remove it from my servers asap.

Member

rahul286 commented Apr 16, 2015

I guess this thread has answer - https://developers.google.com/speed/pagespeed/module/downstream-caching. I will try to test this tomorrow. But if anyone tries and gets it working, please share config/results.

EE's copy of nginx already has ngx_cache_purge purge module compiled in it so that step can be skipped.

About other questions:

I also have worker_processes to 2 (2 core VPS) instead of EE's auto and worker_connections to 2048 instead of EE's default 4096.

Auto always means equal to number of cores, which is recommended. In your case, it would have been 2 only with auto.

EE by default sets worker_rlimit_nofile to 100000. Should this be customized based on hardware?

Ya. I do not have formula for it but I prefer to set higher limits. My idea is to not let a server hit its own limit before OS limits. But again, I set it randomly and it worked. Before that my load tests were failing quite early.

Next up is removing W3TC

We also want to remove W3TC and hence added support for pagespeed. If pagespeed works nicely, we can use it for minify and CDN.

I loved W3TC in early days but it seems bloated now. So I also want to remove it from my servers asap.

@htpcBeginner

This comment has been minimized.

Show comment
Hide comment
@htpcBeginner

htpcBeginner Apr 16, 2015

Thanks @rahul286. Exactly the reason why I tried pagespeed - minifying and CDN.

htpcBeginner commented Apr 16, 2015

Thanks @rahul286. Exactly the reason why I tried pagespeed - minifying and CDN.

@rahul286

This comment has been minimized.

Show comment
Hide comment
@rahul286

rahul286 Apr 16, 2015

Member

minifying and CDN

BTW basic CDN can be done using nginx subs_filter

For minify, if pagespeed refuses to work again, I am planning to try some wp plugin.

But I really wish pagespeed to work!

Member

rahul286 commented Apr 16, 2015

minifying and CDN

BTW basic CDN can be done using nginx subs_filter

For minify, if pagespeed refuses to work again, I am planning to try some wp plugin.

But I really wish pagespeed to work!

@htpcBeginner

This comment has been minimized.

Show comment
Hide comment
@htpcBeginner

htpcBeginner Apr 16, 2015

Good to know. My second option after pagespeed for minifying is autoptimize.https://wordpress.org/plugins/autoptimize/

But like you I hope pagespeed works.

htpcBeginner commented Apr 16, 2015

Good to know. My second option after pagespeed for minifying is autoptimize.https://wordpress.org/plugins/autoptimize/

But like you I hope pagespeed works.

@cm896

This comment has been minimized.

Show comment
Hide comment
@cm896

cm896 Apr 16, 2015

During my clients testing, we pulled all the files to AWS's cloudfront/S3 bucket. From Digital Ocean, I didn't notice any speed gain in the images being served. When I profiled those resources, the CDN didn't offer any advantages, it had a few negative pagespeed impacts though, 1, images from the CDN could not be converted to webp, or inlined, or optimized for sizes. GTmetric, Page Rank were both dinged when I served images from the CDN rather than from the site under page speeds control. We backed the CDN out and allow page speed to serve those images in favor of the better score, since there wasn't a content delivery improvement. For this client, SPDY and SSL were the improvements. I think removing the CDN and plugin to manage it, although we were handling it through W3tc plugin, and then testing with having page speed handling it with:

pagespeed MapRewriteDomain cdn.example.com www.example.com;

Link to one analysis of SPDY + SSL over CDN: https://thethemefoundry.com/blog/why-we-dont-use-a-cdn-spdy-ssl/

cm896 commented Apr 16, 2015

During my clients testing, we pulled all the files to AWS's cloudfront/S3 bucket. From Digital Ocean, I didn't notice any speed gain in the images being served. When I profiled those resources, the CDN didn't offer any advantages, it had a few negative pagespeed impacts though, 1, images from the CDN could not be converted to webp, or inlined, or optimized for sizes. GTmetric, Page Rank were both dinged when I served images from the CDN rather than from the site under page speeds control. We backed the CDN out and allow page speed to serve those images in favor of the better score, since there wasn't a content delivery improvement. For this client, SPDY and SSL were the improvements. I think removing the CDN and plugin to manage it, although we were handling it through W3tc plugin, and then testing with having page speed handling it with:

pagespeed MapRewriteDomain cdn.example.com www.example.com;

Link to one analysis of SPDY + SSL over CDN: https://thethemefoundry.com/blog/why-we-dont-use-a-cdn-spdy-ssl/

@cm896

This comment has been minimized.

Show comment
Hide comment
@cm896

cm896 Apr 16, 2015

@htpcBeginner Have you looked at this one: https://wordpress.org/plugins/wp-minify-fix/
This one was updated and email blast sent out today - it has a pro version too:
https://wordpress.org/plugins/wp-smushit/

I'd rather not have a WP plugin handle this stuff, and allow PS to handle it. I have a feeling my WP install is already over bloated with plugins and it's killing the performance.

cm896 commented Apr 16, 2015

@htpcBeginner Have you looked at this one: https://wordpress.org/plugins/wp-minify-fix/
This one was updated and email blast sent out today - it has a pro version too:
https://wordpress.org/plugins/wp-smushit/

I'd rather not have a WP plugin handle this stuff, and allow PS to handle it. I have a feeling my WP install is already over bloated with plugins and it's killing the performance.

@htpcBeginner

This comment has been minimized.

Show comment
Hide comment
@htpcBeginner

htpcBeginner Apr 16, 2015

Good stuff about CDN. That is what I am finding with CDN tests and hence why I am not using CDN currently.

I am going to leave minify off for now until we know something more about pagespeed, which is my preference.

htpcBeginner commented Apr 16, 2015

Good stuff about CDN. That is what I am finding with CDN tests and hence why I am not using CDN currently.

I am going to leave minify off for now until we know something more about pagespeed, which is my preference.

@cm896

This comment has been minimized.

Show comment
Hide comment
@cm896

cm896 Apr 16, 2015

The problem we have encountered with pagespeed concatenating and minifying js/css is the url length limit, so all scripts, and all css don't make it into one file. That's a bummer, and not all the plugins are not playing nice. Some suggestions to help PS handle these were to:

  1. remove the query strings appended to the css and js enque'ed files.
  2. remove the id's for the css includes/calls.

A custom filter worked to resolve both these issues.

// Remove query string from static files
function remove_cssjs_ver( $src ) {
if( strpos( $src, '?ver=' ) )
$src = remove_query_arg( 'ver', $src );
return $src;
}
add_filter( 'style_loader_src', 'remove_cssjs_ver', 10, 2 );
add_filter( 'script_loader_src', 'remove_cssjs_ver', 10, 2 );

function remove_style_id($link) {
return preg_replace("/id='.*-css'/", "", $link);
}
add_filter('style_loader_tag', 'remove_style_id');

I thought to maybe re-write the file names so more could be smushed together in one call to the server, but we eventually had to back this out since the smushed JS sometimes didn't completely function correctly. So now we are back to 20 css requests, and 26 js requests, which surprisingly still delivers a 94% GTmetric score. :(

cm896 commented Apr 16, 2015

The problem we have encountered with pagespeed concatenating and minifying js/css is the url length limit, so all scripts, and all css don't make it into one file. That's a bummer, and not all the plugins are not playing nice. Some suggestions to help PS handle these were to:

  1. remove the query strings appended to the css and js enque'ed files.
  2. remove the id's for the css includes/calls.

A custom filter worked to resolve both these issues.

// Remove query string from static files
function remove_cssjs_ver( $src ) {
if( strpos( $src, '?ver=' ) )
$src = remove_query_arg( 'ver', $src );
return $src;
}
add_filter( 'style_loader_src', 'remove_cssjs_ver', 10, 2 );
add_filter( 'script_loader_src', 'remove_cssjs_ver', 10, 2 );

function remove_style_id($link) {
return preg_replace("/id='.*-css'/", "", $link);
}
add_filter('style_loader_tag', 'remove_style_id');

I thought to maybe re-write the file names so more could be smushed together in one call to the server, but we eventually had to back this out since the smushed JS sometimes didn't completely function correctly. So now we are back to 20 css requests, and 26 js requests, which surprisingly still delivers a 94% GTmetric score. :(

@rahul286

This comment has been minimized.

Show comment
Hide comment
@rahul286

rahul286 Apr 21, 2015

Member

Update:

Please check for related discussion - https://groups.google.com/forum/#!msg/ngx-pagespeed-discuss/vzSq8XQPGmM/9mE11D-9U8YJ

Current way is to configure downstream caching in pagespeed. But in my test it did not perform well and was too complex to setup.

Pagespeed developer is working on simplifying this - https://groups.google.com/d/msg/ngx-pagespeed-discuss/vzSq8XQPGmM/vAOyCN_JH0cJ

I am not sure how long it will take but I will prefer to wait till then.

Member

rahul286 commented Apr 21, 2015

Update:

Please check for related discussion - https://groups.google.com/forum/#!msg/ngx-pagespeed-discuss/vzSq8XQPGmM/9mE11D-9U8YJ

Current way is to configure downstream caching in pagespeed. But in my test it did not perform well and was too complex to setup.

Pagespeed developer is working on simplifying this - https://groups.google.com/d/msg/ngx-pagespeed-discuss/vzSq8XQPGmM/vAOyCN_JH0cJ

I am not sure how long it will take but I will prefer to wait till then.

@rahul286 rahul286 changed the title from Blitz timeouts with EasyEngine to Blitz timeouts with EasyEngine [pagespeed issue] Apr 24, 2015

@rahul286

This comment has been minimized.

Show comment
Hide comment
@rahul286

rahul286 Jul 22, 2015

Member

Update:

http://www.jefftk.com/downstream-caching-draft

So far downstream caching seems to be only workaround.

Need to check:

  • weather downstream-caching works at all with php/mysql crash
  • how it works with full-page redis and fastcgi-cache
  • scale - HTML response rate should be similar to non-pagespeed variant
  • multiple sites - can it work nicely for 100s of sites on a single server. Will each site need two nginx config blocks?
  • cache purging through nginx-helper
Member

rahul286 commented Jul 22, 2015

Update:

http://www.jefftk.com/downstream-caching-draft

So far downstream caching seems to be only workaround.

Need to check:

  • weather downstream-caching works at all with php/mysql crash
  • how it works with full-page redis and fastcgi-cache
  • scale - HTML response rate should be similar to non-pagespeed variant
  • multiple sites - can it work nicely for 100s of sites on a single server. Will each site need two nginx config blocks?
  • cache purging through nginx-helper
@rahul286

This comment has been minimized.

Show comment
Hide comment
@rahul286

rahul286 May 29, 2018

Member

We no longer support pagespeed so we won't be able to look further in this. Sorry.

Member

rahul286 commented May 29, 2018

We no longer support pagespeed so we won't be able to look further in this. Sorry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment