New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nginx ModPagespeed with CloudFront : 5 mins expiry issue #1585

Open
abieri opened this Issue Jul 6, 2017 · 28 comments

Comments

Projects
None yet
3 participants
@abieri
Copy link

abieri commented Jul 6, 2017

Hi,

I have two load balanced web servers that are running pagespeed and one CDN server that is also running pagespeed to take advantage of image optimizations.

On top of that CDN server I have AWS cloudfront. I've made to extra rules on CloudFront to make sure that the images that are generated by pagespeed are regularly pulled as I read that if you have the 5 mins expiry issue it's mostly because pagespeed can optimize those images further.

I am still getting those 5 mins expiry. I use pagespeed insight as a tool to monitor my performances and as long as I added the images optimisation none of them got flaggued anymore but I get the expiry issue. Is there any way to prevent that behavior since the performance are acceptable on my end?

For example I still get issues for this image //cdn.cigars.com/images/category/main/1487002303387-0.jpeg/280/200: However I can see those logs on the cdn server :
13895#0: [ngx_pagespeed 1.11.33.4-0] ipro: Shrinking image http://cdn.cigars.com/images/category/main/1487002303387-0.jpeg/280/200' (41376 bytes) to http://cdn.cigars.com/images/category/main/1487002303387-0.jpeg/280/x200.pagespeed.ic.YVeabFbeIZ.jpg' (15683 bytes)

Website is : https://www.cigars.com
CloudFront Distribution : https://d18md549cxuah8.cloudfront.net

Thanks a lot in advance

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented Jul 6, 2017

http://cdn.cigars.com/images/category/main/1487002303387-0.jpeg/280/200 gives me:
Cache-Control:max-age=31544077, public
If you still get the short TTL, the issue probably is related to CF's caching policy. If you don't, the problem probably is transient. The module emits responses with a short cache ttl when it's not fully done optimizing them yet, and may CF is caching those.

@oschaaf oschaaf closed this Jul 6, 2017

@abieri

This comment has been minimized.

Copy link

abieri commented Jul 6, 2017

@oschaaf thanks, I ended up using the cloudfront object caching option based on the sent headers, therefore it should behave as normal. I keep you posted if I still have the issue after a couple of days.

Thanks

screen shot 2017-07-06 at 8 51 06 pm

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented Jul 6, 2017

@abieri thanks for the update

@abieri

This comment has been minimized.

Copy link

abieri commented Jul 7, 2017

@oschaaf after over a day of traffic I'm not getting any improvement. Is there anything in the logs i should look for that might prevent from optimizing images further?

@oschaaf oschaaf reopened this Jul 7, 2017

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented Jul 7, 2017

I can see that https://d18md549cxuah8.cloudfront.net/images/banner/1480293162163-1.png/640/x130.pagespeed.ic.vYsLBK0bWP.jpg?PageSpeedNoop=<random> keeps responding with the short TTL. PageSpeedNoop=XX should bust the CDN caches, and assuming it's not cloudflare overwriting the CC header, it's mod_pagespeed generating them.

Would it be possible for you to upgrade the module to the latest stable? I'm curious if the problem has been resolved in the latest release.

@abieri

This comment has been minimized.

Copy link

abieri commented Jul 7, 2017

@oschaaf thanks for your prompt reply actually we did perform a major update on one of our other site that follows the exact same architecture and the issue seems to be gone for that site. We will plan an update at the appropriate time and come back to you. Should we plan the update on both the webservers and the cdn?

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented Jul 7, 2017

Re: "Should we plan the update on both the webservers and the cdn?"
I think that would be best, yes. Closing this issue again, let's re-open it if the ugprade does not resolve the issue

@oschaaf oschaaf closed this Jul 7, 2017

@abieri

This comment has been minimized.

Copy link

abieri commented Jul 20, 2017

@oschaaf We have have successfully updated our pagespeed versions on all server but we are still experiencing the same expiry issue :

https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fwww.cigars.com%2F&tab=mobile : you can see the mobile version suffers from 5 mins expiry images.

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented Jul 20, 2017

@abieri Thanks for the update, re-opening.

I can reproduce it when I request the index page with User-Agent set to Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko; Google Page Speed Insights) Chrome/27.0.1453 Mobile Safari/537.36.

Could you try disabling resize_mobile_images and/or resize_images to see if disabling either one or both of them help with getting long cache TTL headers?

@oschaaf oschaaf reopened this Jul 20, 2017

@abieri

This comment has been minimized.

Copy link

abieri commented Jul 23, 2017

@oschaaf I've disabled both filters and I'm still experiencing the same issue. Weirdly i'm seeing good results after a cloudfront invalidation but then re-experiencing the problem.

@abieri

This comment has been minimized.

Copy link

abieri commented Aug 7, 2017

@oschaaf any updates regarding that issue?

@jmarantz

This comment has been minimized.

Copy link
Contributor

jmarantz commented Aug 7, 2017

Question about your CloudFront setup. Does it send request-headers through? mod_pagespeed's image-serving depends on client request headers to know whether to send a jpeg or a webp. In particular, it needs the 'Accept' and (if provided) 'Save-Data' headers. I think I remember in the past having trouble with CDNs -- maybe CloudFront -- being configured by default to strip out request-headers when proxying requests from origin, but I think you should be able to override that.

@abieri

This comment has been minimized.

Copy link

abieri commented Aug 7, 2017

Thanks @jmarantz I believe i've read that and in my other property I had the accept setting but i was missing the "Save-Data". I've added it and will come back to you tomorrow

@jmarantz

This comment has been minimized.

Copy link
Contributor

jmarantz commented Aug 7, 2017

Please see http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-headers-behavior

Notice that CloudFront removes the header by default, and you need to override that. It's probably worth taking a deep look at that doc, as well as the PageSpeed companion:

https://modpagespeed.com/doc/reference-image-optimize#AllowVaryOn

The reason this is related to private/300 caching headers is that if mod_pagespeed sees a request for a .pagespeed. resource generated for (say) a webp-enabled browser (Chrome, Opera etc) but the request does not include Accept:image/webp, then it will send a jpeg in response to the webp request. It assumes that this was due to a missing request-header and to avoid poisoning a proxy cache, sends the conservative header.

To avoid this, you must make sure PageSpeed sees all relevant headers.

@abieri

This comment has been minimized.

Copy link

abieri commented Aug 9, 2017

thanks a lot @jmarantz , I had a another read through those and I've double checked my configs and here is what I have :

  • AWS cloudfront whitelist headers Accept and Save-Data

  • I've added the directive : pagespeed AllowVaryOn "Accept, Save-Data";

I've still experiencing the issue though. Do you see anything else that maybe prevents from the accept header to be taken in account.

Would that be worth that I send you my config in PM?

All the best

@jmarantz

This comment has been minimized.

Copy link
Contributor

jmarantz commented Aug 9, 2017

I can see the problem in PageSpeed Insights (PSI) on cigars.com. I'm not sure I fully understand what's going on, but I have a couple of theories.

First of all, I see two versions of the same image, depending on how it is fetched. From PSI, www.cigars.com references:

https://d18md549cxuah8.cloudfront.net/images/banner/1480293162163-1.png/640/x130.pagespeed.ic.2vy6u77e0w.webp

This is served with 5-minute private TTL, which I confirmed with 'wget -S'. In general the reason that MPS serves a 5-minute private TTL is because the content-hash (2vy6u77e0w in this case) does not match the content being served. This makes MPS assume that the user has changed the content of the origin image and wants to allow that change to propagate. In reality, most of the time it is being optimized differently.

However, when I fetch www.cigars.com from Chrome directly, MPS rewrites the URL slightly differently:

https://d18md549cxuah8.cloudfront.net/images/banner/1480293162163-1.png/640/x130.pagespeed.ic.PePNL2uupd.webp

The only difference is the content hash, which is PePNL2uupd rather than 2vy6u77e0w. This image is served with public caching for one year. What's puzzling to me is that both versions of the image are exactly the same size (10388 bytes), and are in fact byte-for-byte identical.

One theory is that somehow your HTML is being cached unexpectedly, possibly by Cloudfront, and despite the fact that your HTML is served with Cache-control: max-age=0,no-cache.

Another theory is that MPS does some special-casing around serving requests from PSI, which doesn't send Accept:image/webp headers but does in fact process webp, and that this is interacting poorly with the header propagation rules in Cloudfront.

I'm tempted to remove those rules so that PSI does not see webp-transcoding at all.

Lastly I want to point out that this is really an issue of PSI seeing a different result from normal browser. If you view your site in Chrome the caching behavior is good.

@jmarantz

This comment has been minimized.

Copy link
Contributor

jmarantz commented Aug 9, 2017

One other quick question: do you have a setting for ModPagespeedServeRewrittenWebpUrlsToAnyAgent in your config? Default is 'on' and I want to know if you've turned that off.

@abieri

This comment has been minimized.

Copy link

abieri commented Aug 10, 2017

@jmarantz Thanks for your previous explanation

  • The html is definitely not cached by cloudfront, but i wonder if I should run a pagespeed cache cleaning on my different servers?
  • the ModPagespeedServeRewrittenWebpUrlsToAnyAgent flag isn't in my config
  • Concerning removing those rules how do you wanna test that?
  • What I'm mostly worried is that it might be a PSI issue, but would that be picked as slow pages when being crawled?

All the best

@jmarantz

This comment has been minimized.

Copy link
Contributor

jmarantz commented Aug 10, 2017

I don't think you should need to clean cache manually due to config changes. This is because the config signature is used as part of our cache key. However, if you change content you may need to clean cache manually, and that could indeed cause this issue, so it's worth giving it a try. Be sure to use one of the mechanisms described in https://modpagespeed.com/doc/system#flush_cache and don't just try to delete the cache directories.

Also can you make sure your pagespeed configuration & versions are consistent between the servers?

I think the short-cache problems have a lot less weight than other PSI issues. But I am leaning toward removing mod_pagespeed's special-casing of PSI, which it has to do with user-agent, and doesn't come through in Accept headers. I was not able to repro the issue you saw with PSI on mod_pagespeed's demo page: https://developers.google.com/speed/pagespeed/insights/?url=http%3A%2F%2Fmodpagespeed.com%2Frewrite_images.html%3FPageSpeed%3Don%26PageSpeedFilters%3Drewrite_images%2Cinline_images%2Cresize_images%2Cinsert_image_dimensions . There is a cache-expiry warning but it's for an in-place resource, so it's correct.

@abieri

This comment has been minimized.

Copy link

abieri commented Aug 15, 2017

Thanks @jmarantz, I did an attempt to clear the cache but the issue is still there.

The nginx configs are same on the FE webservers but might differ slightly due to reverse proxy names on the CDN server. Should I attempt to make them absolutely identical?

@abieri

This comment has been minimized.

Copy link

abieri commented Aug 15, 2017

@jmarantz I've updated my configs to ensure these are absolutely identical, i'm still having the same issue with the expiry.
The pagespeed + nginx versions are all identical, we updated them lately.

@jmarantz

This comment has been minimized.

Copy link
Contributor

jmarantz commented Aug 15, 2017

I have to admit I've seen private/300 caching with CloudFront before and I'm running out of ideas on how to solve it. But I think it might be related to mod_pagespeed's hacks to optimize for PSI, and not really affect real users. I just visited www.cigars.com again manually and the headers looked fine in Chrome DevConsole.

I understand that optimizing PSI score may be a goal, but I think we may have to leave this as an open issue, which would be resolved by removing the special-casing mod_pagespeed does when serving responses to PSI. And this specific issue (TTL of images) is not (IIUC) a major factor in the PSI score, even though it does list it as something to improve.

@abieri

This comment has been minimized.

Copy link

abieri commented Sep 28, 2017

@jmarantz @oschaaf apologise for the delayed answer. Is there anyway to actually deactivate that PSI optimization via an mod pagespeed option? I think in a way it'd be better for us to have a real insight on the user's experience if there wasn't any PSI special case within mod pagespeed if that makes sense?

@oschaaf

This comment has been minimized.

Copy link
Member

oschaaf commented Sep 28, 2017

@abieri as of today, I don't think such an option exists today. I think that earlier, @jmarantz proposed axing out the PSI-specific stuff completely.

@abieri

This comment has been minimized.

Copy link

abieri commented Oct 4, 2017

@oschaaf @jmarantz let me know if you end up doing so, I'll then have to upgrade my pagespeed version i imagine.

@jmarantz

This comment has been minimized.

Copy link
Contributor

jmarantz commented Oct 4, 2017

@jmarantz

This comment has been minimized.

Copy link
Contributor

jmarantz commented Oct 12, 2017

I have started working on this.

jmarantz added a commit that referenced this issue Oct 13, 2017

Remove special-case serving of webp to PSI (which lacks accept:image/…
…webp) due to difficulties transmitting user-agent through cloudfront. See #1585

jmarantz added a commit that referenced this issue Oct 14, 2017

Remove special-case serving of webp to PSI (which lacks accept:image/…
…webp) due to difficulties transmitting user-agent through cloudfront. See #1585 (#1646)
@abieri

This comment has been minimized.

Copy link

abieri commented Oct 18, 2017

Thanks a lot @jmarantz when is this going to be released?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment