Cloudfront cache is deleted every 1 second

[thedgbrt]

Describe the bug
All assets return x-cache: Miss from Cloudfront, even after multiple refreshes. It only returns a cached version when a request is made less than one second after the previous one.

Environment
Default Gatsby app. The amplify console sets the infrastructure to Infrastructure: Gatsby-Amplify

Proof
Request the same asset with 2000ms interval, we always get a MISS:

while true; do curl -X HEAD -i https://master.dsvk1bi06ng1b.amplifyapp.com/ -s | grep -Fi x-cache; sleep 2; done
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront
x-cache: Miss from cloudfront

Request the same asset with 100ms interval, we only get a MISS every ~1second:

while true; do curl -X HEAD -i https://master.dsvk1bi06ng1b.amplifyapp.com/ -s | grep -Fi x-cache; sleep 0.1; done
x-cache: Miss from cloudfront
x-cache: Hit from cloudfront
x-cache: Hit from cloudfront
x-cache: Hit from cloudfront
x-cache: Hit from cloudfront
x-cache: Miss from cloudfront
x-cache: Hit from cloudfront
x-cache: Hit from cloudfront
x-cache: Hit from cloudfront
x-cache: Hit from cloudfront
x-cache: Hit from cloudfront
x-cache: Hit from cloudfront
x-cache: Hit from cloudfront
x-cache: Hit from cloudfront
x-cache: Miss from cloudfront
x-cache: Hit from cloudfront
x-cache: Hit from cloudfront

To Reproduce
use this command from the terminal:
while true; do curl -X HEAD -i https://master.dsvk1bi06ng1b.amplifyapp.com/ -s | grep -Fi x-cache; sleep 2; done

Alternatively, you can go to the deploy url and refresh the page with the network inspector open.

Expected behavior
Cloudfront should only invalidate cache when a new deployment is made. For every request after the first one, it should return the cached version from the CDN, as mentioned here.

Sample code
I've created a demo repo to view the issue.

• repo url
• deployed link

[thedgbrt]

I could also reproduce with a standard create-react-app.

• deploy link
• repo url

[swaminator]

@thedgbrt The content is indeed being cached, we just have an issue with reporting the information correctly in the header responses which has led to confusion from other customers before as well. The diagram in the blog post doesn't fully explain how our CDN works - we have dual layer CDN distributions so what happens is that you're seeing the miss from the CDN Edge vs the CDN Origin. We will prioritize a fix for this soon. Hope that helps - thanks for the detailed explanation.

[thedgbrt]

@swaminator thanks for the quick answer.

What I'm concerned about is response time. When I manage to hit the CloudFront cache, I get a response in 10-30ms, which is what I'm used to when using a CDN.

However when I don't hit it (that is, 99% of the time right now), it's in the 50-200ms range, and can go up to 500ms. For an SPA with code split between 10-20 files each page, this has a dramatic impact on user experience. I routinely have to wait 1-2 seconds for a simple page change that could be almost instant.

I don't know how much that has to do with proper cache headers being set, or with the way amplify console fundamentally manages CDN responses. Is the fix that is being prioritized going to lower response times to expected CloudFront levels?

[cslogan-red]

Trying to provide a little more clarity until we've got a further solution. As @swaminator mentioned, your content is being cached and returned by our origin CDN, however you're seeing the response header from the closest edge CDN as a miss (which is by design), which is understandably confusing. The design is a bit more complex than alluded to, but in order to facilitate instant deployment changes (including invalidation of cached resources) and have as performant hosting as possible for all of our customers, we've implemented a dual layer CDN structure with an edge layer in-between. The response times you're seeing come from the brief edge layer invocation, before the content is then retrieved from the origin CDN (if it exists there, otherwise it is retrieved all the way from actual origin, which takes much longer). This round-trip time is several times faster than if we went straight to origin, and is still giving you the benefit of cached content (read; not re-serving from hosting origin if the file hasn't changed). We understand that certain customers (like in the example provided, with heavy code splitting) may want to optimize their edge CDN performance even further, and we are looking at a number of ways to provide this.

That said, your unmodified content is being cached and served from our origin CDN, the edge CDN response header just makes this confusing.

[mrcoles]

@cslogan-red what about being able to respond with 304 “not modified” for assets (for example my site.webmanifest file gets requested with a 200 status every time, despite the request header and response header etags matching) and also what if the instant deployment optimizations were bypassed if the cache headers are aggressive enough (e.g., "Cache-Control: public,max-age=31536000,immutable")?

[cslogan-red]

@mrcoles we're considering a number of solutions for both the confusing initial response header from the edge CDN as well as tuning the edge CDN's performance that just need prioritization, we'll respond to this thread once we've implemented an update!

[arelaxend]

Is it expected that Chrome or any browser can not cache to disk files provided by cloudfront such as JS or CSS files ? For example, https://app.fays.io/7ce58fce-d67e-47c5-8ad7-7163b45d2b87, after each tab refresh, Chrome downloads a 1.2 megabytes js file.

[isi-gach]

have you tried something like this?
amplify.yml

version: 0.1
frontend:
  phases:
    preBuild:
      commands:
        - yarn
    build:
      commands:
        - yarn build
  customHeaders:
    - pattern: '/*.chunk.js'
      headers:
      - key: 'Cache-Control'
        value: 'public, max-age=31536000, immutable'

[arelaxend]

have you tried something like this?
amplify.yml

It works. 🥇 I always forget this amplify.yml file ! I have used pattern: '**/*' instead for more generality.

[supersetapp]

@arelaxend setting those Cache-Control headers can be super useful, but be careful you’re not caching content that you want to update at some later time. Doing pattern: '**/*' would set that Cache-Control header on your HTML files and other things that usually don’t want such extreme caching (usually you only want aggressive caching on just images, JS, CSS—especially if they have a content hash in their filename).

This doesn’t address the issue of how the CDN is not returning 304 not modified when request and response etag headers match (#92 (comment))

[masterofkfit]

Any update on this?

This should be fixed ASAP as cloudfront is considered broken

[litwicki]

@masterofkfit @thedgbrt can you please let us know if you're still experiencing this as it should be resolved.

[robbie-thompson]

@swaminator why was this closed? I can still replicate this on a standard setup.

[marco910]

@robbie-thompson I had the same issue a few weeks ago and found a solution that works for me. I don't know if this is the "right" way, but it works. To cache the assets in the CloudFront cache I did the following:

1. Added these custom header to my customHttp.yml:

customHeaders:
- pattern: '**/*.html'
    headers:
      - key: Cache-Control
        value: 'public, max-age=0, s-maxage=60'
      - key: x-frame-options
        value: DENY
      - key: x-xss-protection
        value: 1; mode=block
  - pattern: '**/*[!.html]'
    headers:
      - key: Cache-Control
        value: 'public,max-age=31536000,immutable'
      - key: x-frame-options
        value: DENY
      - key: x-xss-protection
        value: 1; mode=block

2. Enable performance mode in the app settings in the Amplify console.
3. Redeploy the last version to create a new build with the custom headers and performance mode.

[cslogan-red]

@robbie-thompson see performance mode docs for more info. We launched our redesigned hosting and deployment modules around the time this issue was closed, which removes the CDN sandwich I described earlier in this thread for a more standard and fine tuneable setup. By default, in "development mode", we expire the CDN cache every second to facilitate instantly showing deployed changes in origin, once you're in steady state, on say your Production branch that doesn't see deployed changes as regularly, enabling performance mode for that branch reconfigures the CDN timeout to be biased towards app performance at the sake of a delay between deployed changes appearing. Further, you can fine tune the behavior in addition by providing your own custom header overrides for specific paths, as @marco910 example shows.

[mathan-xflowpay]

@cslogan-red I have enabled performance enabled mode and have used customHttp.yml to set custom header for only specific paths as mentioned by @marco910. But I see the same cache header is getting applied to everything including the index.html also and making it indefinitely cached.

[marco910]

@mathan-xflowpay I had some issues with caching in the last time too. For Gatsby I'm using this config for custom headers right now:

customHeaders:

  # Static files
  - pattern: static/**/*
    headers:
      - key: Cache-Control
        value: 'public, max-age=31536000, immutable'
      - key: x-frame-options
        value: DENY
      - key: x-xss-protection
        value: 1; mode=block

  # Static assets
  - pattern: '**/*.+(js|css|jpg|jpeg|png|webp|svg|tiff|bmp|woff|ttf|ico|gif)'
    headers:
      - key: Cache-Control
        value: 'public, max-age=31536000, immutable'
      - key: x-frame-options
        value: DENY
      - key: x-xss-protection
        value: 1; mode=block

  # HTML and JSON
  - pattern: '**/*.+(html|json)'
    headers:
      - key: Cache-Control
        value: 'public, max-age=0, s-maxage=60, must-revalidate'
      - key: x-frame-options
        value: DENY
      - key: x-xss-protection
        value: 1; mode=block

I'm not using performance mode, so I can't tell you what effect it has on the caching, but normally, you don't need it. Try to redeploy the site with the new headers, clear your browser cache or use incognito mode and what a few hours (sometimes it takes sometime until old files are removed from the cache).
Please never cache HTML files. That can cause big issues when updating.

[cslogan-red]

@marco910 config is a good custom header example for a static site generator like Gatsby or if you have a customized, heavily code split SPA, in either case you may have several thousand assets of varying types in your production bundle - Amplify Console's development mode (standard for all apps) and optional performance mode are catch-alls that work well for most simple SPA bootstraps like create react app or vue CLI, but we provide the custom header (customHttp.yml) functionality so that you can fine tune for your individual application's framework.

If you're going to use your own custom headers, I would avoid using performance mode altogether and simply define Cache-Control headers for all asset types, as per Marco's example above, as this guarantees Amplify Console will only apply a CDN cache duration of 2 seconds for all asset types, and you can provide overrides for this duration per asset type as necessary for your app.

When using performance mode, know that Amplify Console applies a blanket CDN cache duration of 10 minutes to all asset types - this is a straight-forward way to ensure CDN edge performance for the duration of a common user interaction at the expense of the 10 minute delay in new assets appearing post-deployment. Additionally, regardless of your custom header configuration and whether or not performance mode is enabled, Amplify Console sets the TTL max-age for all assets to 24 hours as a fallback to ensure your CDN assets sync with your most recent deployment at least once per day. Behind the scenes via HTTP/2, if your content has not changed in origin, both your client (browser) as well as the CDN are smart enough to re-serve "stale" cached assets via 304's so long as the Etag headers match from origin. In other words, even with development mode enabled, if you're not deploying changes, your assets are being re-served (without being re-downloaded from origin) from the closest CDN edge location so long as the Etag headers continue to match origin.

[mathan-xflowpay]

Hi @marco910 and @cslogan-red ,

Thanks for the explanation. My application is a nextjs based one and I am using the default nextjs deployment which amplify provides. I used the following custom headers and my performance mode is disabled.

customHeaders:
  # Static files
  - pattern: _next/static/**/*
    headers:
      - key: Cache-Control
        value: 'public, max-age=31536000, immutable'
      - key: x-frame-options
        value: DENY
      - key: x-xss-protection
        value: 1; mode=block
      - key: x-content-type-options
        value: nosniff
      - key: content-security-policy
        value: "script-src 'self' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:;font-src 'self' data: https://fonts.gstatic.com https://fonts.google.com"
      - key: strict-transport-security
        value: 'max-age=63072000; includeSubDomains; preload'
      - key: X-DNS-Prefetch-Control
        value: 'on'

  # HTML and JSON
  - pattern: '**/*.+(html|json)'
    headers:
      - key: Cache-Control
        value: 'public, max-age=0, s-maxage=60, must-revalidate'
      - key: x-frame-options
        value: DENY
      - key: x-xss-protection
        value: 1; mode=block
      - key: x-xss-protection
        value: 1; mode=block
      - key: x-content-type-options
        value: nosniff
      - key: content-security-policy
        value: "script-src 'self' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:;font-src 'self' data: https://fonts.gstatic.com"
      - key: strict-transport-security
        value: 'max-age=63072000; includeSubDomains; preload'
      - key: X-DNS-Prefetch-Control
        value: 'on'

But every asset is getting the same cache-control header as the one which I gave for html glob pattern which is public, max-age=0, s-maxage=60, must-revalidate. But expectation is only the index html file should have that header and my static css and js assets should have public, max-age=31536000, immutable. I think amplify is applying the last set of headers in the customHttp.yml file always irrespective of the pattern match. You can check my live site here

[marco910]

@mathan-xflowpay Maybe you can also have a look at that custom header config (but yours seems to be ok):

customHeaders:
  - pattern: /_next/static/**/*
    headers:
      - key: Cache-Control
        value: 'public, max-age=31536000, immutable'
      - key: x-frame-options
        value: DENY
      - key: x-xss-protection
        value: 1; mode=block
  - pattern: /_next/data/**/*
    headers:
      - key: Cache-Control
        value: 'public, max-age=0, s-maxage=0, must-revalidate'
      - key: x-frame-options
        value: DENY
      - key: x-xss-protection
        value: 1; mode=block
  - pattern: '**/*[.html]'
    headers:
      - key: Cache-Control
        value: 'public, max-age=0, s-maxage=0, must-revalidate'
      - key: x-frame-options
        value: DENY
      - key: x-xss-protection
        value: 1; mode=block

It can take a while until all "old stuff" is removed from the edge caches. That can take up to 48 hours. You can also try to redeploy the current version from the Amplify console. This may trigger a cache invalidation in CloudFront.

[mathan-xflowpay]

@marco910 Thanks for this. But I have tried this and multiple other variations also. But irrespective of the pattern always the last cache-control gets applied to all the css, js and index. In the example which you provided its always applying the cache-control of the pattern '**/*[.html]' for everything. I am thinking that definitely something is wrong with amplify which is causing this. My live website is here

[marco910]

@mathan-xflowpay I'm facing currently a very similar issue that the cache is not being cleared after redeploying a new version of the site. If you change the content on a page and redeploy the site, do you see the changes on the page?

UPDATE: I figured out, that my output directory in the build settings was wrong. After changing that, it worked.
Applying new headers can take time. I experienced that in the past already.

[Reselence]

@marco910 I've tried adding your suggested customHttp.yml but I'm also not seeing any differences. When you say that your output directory was incorrect, what was it and what did you change it to? Maybe I'm doing something wrong too?

My files are as follows:

amplify.yml

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - nvm install 16.10.0
        - npm ci
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
      - .next/cache/**/*

customHttp.yml

customHeaders:
  - pattern: /_next/static/**/*
    headers:
      - key: Cache-Control
        value: 'public, max-age=31536000, immutable'
      - key: x-frame-options
        value: DENY
      - key: x-xss-protection
        value: 1; mode=block
  - pattern: /_next/data/**/*
    headers:
      - key: Cache-Control
        value: 'public, max-age=0, s-maxage=0, must-revalidate'
      - key: x-frame-options
        value: DENY
      - key: x-xss-protection
        value: 1; mode=block
  - pattern: '**/*[.html]'
    headers:
      - key: Cache-Control
        value: 'public, max-age=0, s-maxage=0, must-revalidate'
      - key: x-frame-options
        value: DENY
      - key: x-xss-protection
        value: 1; mode=block

[marco910]

@Reselence When deploying a static generated (SSG) site to Amplify you need to apply a different output directory than for sites with server-rendering (SSR) or Incremental Site Regeneration (ISR). For SSG you need to use out and for sites with SSR and ISR .next.

These are my build settings for an SSG site:

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - yarn install
    build:
      commands:
        - npm run export
  artifacts:
    baseDirectory: out
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

Please note that you have to change the build command to the command you specified in your package.json file. In my case, npm run export will run next build && next export. This will only work for SSG sites. For SSR and ISR sites you have to use next build so that Amplify detects this as sites with SSR/ISR.

[Reselence]

@marco910 Ah that's an issue. I use an API route for a contact form, but API routes do not work with export. When building my site however, all pages except the API route are marked as static so I would have assumed that I could still control the cache properties.

Performance mode seems to work without a customHttp file, but I want to cache images etc for as long as possible since I will only update this site a couple times a year.

Is there no way to have control of the cache properties this way?

[marco910]

@Reselence You can try to add the headers already in Next.js. Also can try using this in customHttp.yml:

customHeaders:
  - pattern: '**/*.{jpg|jpeg|png|svg|bmp|tiff|webp|avif|ico}'
    headers:
      - key: Cache-Control
        value: 'public, max-age=31536000, immutable'

Maybe you have to edit the pattern depending on the file types you're using on your site.

[Reselence]

@marco910 Setting the headers for those file types actually worked, thank you. Although I'm a little concerned now as even my html and scripts are cached for that length even though the pattern only includes images.

Edit: It seems that the pattern is being applied to everything, not just those image types. I've added the pattern for html from previously to override it to 0 max-age and it seems to be working. For the time being this seems fine.

Maybe I'm just not understanding how the pattern paths work. Is there documentation on how it works?

Either way, thanks for all the help Marco!

[marco910]

@Reselence Sorry, my mistake. This patter should work:

**/*.{jpeg,jpg,svg,webp,avif,bmp,tiff,gif,png}

[cslogan-red]

@Reselence (and for anyone's future reference) as noted in my earlier explanation; if you're not regularly deploying changes, real world there isn't much of a difference between the default and performance modes (we've designed it that way on purpose)

Performance mode guarantees closest edge node cache hits for the blanket 10 minute interval, however due to how all modern browsers and CloudFront CDN's handle 304 refresh hits on identical Etag headers, if your content hasn't changed in origin (no deployments), you'll see about the same performance in the standard mode for the most part.

The standard mode expires CDN cache every 2 seconds (with no caching on client), if content in origin hasn't changed, a request from client will look like; GET index.html, browser sends (via HTTP/2) Etag to CDN, CDN says, index.html expired, forwards Etag to origin, origin responds 304 - not modified, CDN forwards 304 to client - if copy doesn't exist on client, it's downloaded from edge CDN (not all the way from origin, and even though it's expired), but in most cases it will already exist on the client, Etag on client is identical, so it's reserved from client. That entire loop happens automatically in modern browsers behind the scenes via HTTP/2 - while the content on the client is rendered immediately, if that whole trip returns unmodified, then the client does not refresh resources in the browser, and performance is identical to performance mode.

The caveat is that occasionally that 304 loop will drop at a CDN edge node that fails to propagate the cached resources, and so a download from origin will be required (primary benefit of performance mode), but more or less you're getting the same result, which will be sufficient for most Amplify apps. (We host Amplify Admin on Amplify Console to prove its effectiveness, we do not use performance mode) Performance mode is encouraged only if you're regularly seeing high TTFB, and further you'll likely have to customize the Cache-Control header entirely for complex static site generator apps with thousands of cached resources.

[Reselence]

@cslogan-red Thanks for the explanation! Very interesting to see how it all works.

My main objective right now is getting all files to have appropriate cache headers, which is something I'm struggling with. I'm having the same issue as @mathan-xflowpay where only the last pattern is taken into account for all content types, regardless of the pattern specified.

I've tried everything @marco910 suggested (thank you by the way) but I'm still left with the same issue.

I'm still new to server side things and AWS in general so I wanted to avoid creating an issue for this in case it was just me doing something incorrectly.

[cslogan-red]

@Reselence like I said - the default mode without using performance mode or any cache control overrides via custom headers may be enough for your application, it's not a requirement to configure them, and performance is generally about the same. Many production apps run in this mode.

Most folks run into trouble when testing very long max-age (client) and s-maxage (CDN) values and assume a deployment will bust those caches - as noted in our docs, this is not the case, your individual app CDN's are not invalidated post-deployment for many reasons, it's an expensive operation, and it has global propagation ramifications. Instead, post-deployment, files are invalidated in cache based on the default Cache-Control headers we configure for you; once every 2 seconds (10 minutes in performance mode) As a fallback, we set the max-TTL on all files to 24 hours - this supercedes any value you provide.

This means if you set very long values for max-age or s-maxage via custom headers, make requests to your app (which applies the headers), and then realize that's not the correct configuration and deploy updates/change the headers - any files covered by your previous custom headers config must wait for the cache you set to expire (up to the 24 hour max-TTL limit) before new custom header or deployment changes will be reflected on those files. This is why we recommend to simply stick with the default configuration for most apps.

Additionally, custom headers are applied in top-down order from your custom header file; least specific selectors should be up top, however, if you have an overlapping less specific selector path with a more specific selector, CloudFront may choose to use the least specific selector and we cannot control this.

In short, recommendations for using Cache-Control custom headers:

1. Use the default standard mode that Amplify Console gives your app, which applies a global 2 second CDN cache invalidation Cache-Control header (with no caching on client). Due to my explanation above, performance will be similar to performance mode in most cases.
2. Only stop using standard mode once you've benchmarked a problem area.
3. We recommend enabling either performance mode or your own custom headers on a separate branch connected to Amplify Console that is heavily tested before deploying to your production environment.
4. If using custom headers, it's best to not use performance mode; accept the default 2 second CDN invalidation configuration, and only add the most specific custom header path overrides that you can, as opposed to a global path that may overlap with more fine-grained paths.

[mathan-xflowpay]

@cslogan-red

Thanks for the detailed explanation.

But the problem @Reselence and I are facing is specific to image caching and I have my performance mode disabled. I am targeting only the images via this pattern ‘ **/*.{jpeg,jpg,svg,webp,avif,bmp,tiff,gif,png}’ in my customHttp.yml and this is the only pattern I have in my customHttp.yml file but still the cache control header given for this pattern is getting applied to every other js and index.html files also which is not expected.
Can you let us know why that is happening when the pattern is for only the images?

Also I have connected my another test branch to amplify to test the changes before going to production. Thanks for the suggestion :)

[cslogan-red]

@mathan-xflowpay I believe the response header behavior is a long-standing issue that's been documented in prior GH issues and is actually a limitation of CloudFront itself, it ends up reflecting the last applied custom header value for all files, but that's just the header response value, not what's actually set on the file itself in the CDN and client. You should be able to set different file types to different (short) values to manually verify behavior.

I've looped in @Athena96 from the Console team to take a look this week and make sure this isn't a new regression.

[mathan-xflowpay]

@cslogan-red Thank you. Just to be clear this issue is happening only in the nextjs project and custom headers in my other projects work fine.

@Athena96 Happy to help if you need any details related to this issue.

[ferdingler]

We are aware of an issue with custom headers and SSR applications. The glob pattern is currently not honored and headers are applied to all assets regardless of your specified pattern. We have a fix on the pipeline, apologies for the inconvenience.

[mathan-xflowpay]

@ferdingler Thanks for the update. Can you also pls share the ETA on this?

[marco910]

I updated my Next.js app to SSR and ISR and facing now the same issue with custom headers where the last setting of the customHTTP.yml is applied to all files (CSS, HTML, JS etc.).

Does anyone have a solution for this, or is this an open long-term issue?

[dottodot]

I'm finding this a bit confusing as to what the correct setup is for Nextjs ISR.

With no custom headers and performance mode off there seems to be no caching and load times are slower 1.2s, ISR works though.
With no custom headers and performance mode on caching seems to work and load times are down to 100ms but the ISR never regenerates the page.

[marco910]

Are there any updates on the issue with custom headers?
Just tried to deploy a site with custom header and the last header of the customHTTP.yml is applied to all files.

@dottodot That's true, it is confusing.
Static assets should have a Cache-Control of public, max-age=31536000, immutable and all other public, max-age=0, s-maxage=60, must-revalidate (or depending on your needs). When deploying a static generated Next.js site it works fine, but when using ISR it doesn't work anymore.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.