Website links are sometimes broken after a new deployment

[billyjanitsch]

It seems that #403 has returned (even after a hard refresh with cache disabled, this time).

I get a console error when I load the page:

bootstrap 9cd85a3096a8a403f60a:52 Uncaught TypeError: Cannot read property 'call' of undefined
    at t (bootstrap 9cd85a3096a8a403f60a:52)
    at Object.<anonymous> (production-app.js:6)
    at Object.<anonymous> (app-4840c27285bf1ea8bd6d.js:228)
    at t (bootstrap 9cd85a3096a8a403f60a:52)
    at window.webpackJsonp (bootstrap 9cd85a3096a8a403f60a:23)
    at app-4840c27285bf1ea8bd6d.js:1

And another when I click on any link:

index.js:211 Uncaught TypeError: window.___navigateTo is not a function
    at t.navigateTo (index.js:211)
    at gatsby-browser.js:12
    at catch-links.js:56

[bvaughn]

To the best of my knowledge, this is a transient issue with Gatsby (similar to gatsbyjs/gatsby/issues/1838). Links aren't dead for everyone. (They're working fine for me, for example.)

I have seen them occasionally break after a new deployment. In my experience, shift+reloading resolves the issue. It sounds like that isn't the case for you? Can you try in an incognito tab and repro?

I'm also curious which page you're landing on initially? (Just www.reactjs.org or a deeper link?)

FYI: I'm going to edit this issue title to be a bit more accurate. 😄

[bvaughn]

cc @KyleAMathews (just an FYI, no response expected ❤️ )

[billyjanitsch]

Interesting. I happened to be landing on this page, and the issue was present in an incognito tab when I was first playing around with it, but now a hard refresh has fixed it.

I'm not sure how Gatsby is bundled, but it looks like several JS files are loaded, along with some inline JS in the HTML. I wonder if there's a race condition between them that tends to balance out once the server has cached the files in the new deploy.

[KyleAMathews]

@billyjanitsch it's almost certainly something like this — the biggest part of the problem afaik is that webpack 1 doesn't make it possible to create unique URLs for unique JS bundles so browsers try using older versions when they should be redownloading an updated version.

Once Gatsby upgrades to webpack 3, we'll be able to fix this https://medium.com/webpack/predictable-long-term-caching-with-webpack-d3eee1d3fa31

[gaearon]

I reproduced this and grabbed the HTML+JS before and after.
Also ran it through Prettier so it's easier to debug.
To run, unpack and npx serve . in the folder.

This is the version right after deployment, with JS files from my disk cache:

before_pretty.zip

It crashes with:

As you can see there's no module with ID 843:

Now, this is the version after force-refreshing the page:

after-pretty.zip

This doesn't error. It has more files, but let's look at the diff for one of the existing files:

diff --git a/path---index-a1623bfeb06af287794f.js b/path---index-a1623bfeb06af287794f.js
index 9b16d49..e10e7b4 100644
--- a/path---index-a1623bfeb06af287794f.js
+++ b/path---index-a1623bfeb06af287794f.js
@@ -1,5 +1,5 @@
 webpackJsonp([0x81b8806e4260], {
-  837: function(e, t) {
+  843: function(e, t) {
     e.exports = {
       data: {
         code: {

Note 843 at the very end—the same one that crashed earlier. Now it exists.

Where is that require coming from, anyway? From app-fbe9851cb9cdb5f99bfb.js which has not changed after resetting the cache:

    function(n, o, t) {
      t(1),
        (n.exports = function(n) {
          return t.e(0x81b8806e4260, function(o, e) {
            e
              ? (console.log("bundle loading error", e), n(!0))
              : n(null, function() {
                  return t(843); // <-------- "too new" module ID
                });
          });
        });
    },

So it looks like what happened during deployment was:

• We added new pages to the docs (and thus shifted module IDs). All bundles referencing each other have changed because the IDs are different. The crux of the issue is that Webpack only changed hashes for bundles whose source code has changed, disregarding the fact that changing module IDs in the bundle actually makes it different on the disk.
• The app.js bundle contains paths to the documentation pages. This is why Webpack recognized this as a change of content, and gave it a new hash. But the new bundle also depends on new module IDs in other bundles (whose hash has not changed)!
• For the above reasons, app.js (whose source code has changed) got a new hash, but path--index.js (for whom only the module IDs has changed) kept the old hash despite different content on the disk.
• HTML references the app.js by its new hash, so it loads a new version of it. However, other files such as path--index.js whose source has not changed still have the old hashes (even though technically their content is different).
• This is why path---index-a1623bfeb06af287794f.js that actually included the module 837 (now 843) has not changed its hash, and so the cached version on the disk still declared its content as module 837. Hence the crash.

Can we disable caching between deploys altogether if there’s no easy fix until Gatsby updates webpack? This is a very broken experience considering we deploy multiple times daily. I'd rather have all users download all new chunks on every deploy than have them use force-refresh.

[gaearon]

Can we just do this? https://medium.com/@songawee/long-term-caching-using-webpack-records-9ed9737d96f2

[gaearon]

We could also try:

https://www.npmjs.com/package/webpack-plugin-hash-output

@STRML says it works well. webpack/webpack#4659 (comment)

[STRML]

Worth a shot. Fixed a major pain point for us. We're still using it in production without issue.

[KyleAMathews]

Thanks for the analysis Dan.

I'm surprised this is happening as I'd understood that Netlify's headers were setup to do etag validation of every file. Though looking at Wikipedia I guess that's not necessarily sufficient https://en.m.wikipedia.org/wiki/HTTP_ETag#Strong_and_weak_validation

https://www.npmjs.com/package/webpack-plugin-hash-output is for webpack 2+ I believe.

The easiest fix is to either figure out why etag validation isn't working or to force all js files to download fresh on each visit.

I've been working on the webpack 3 upgrade the past week and it'll be my only focus as well in January when I'm back. One of the things I'll be doing is implement https://medium.com/webpack/predictable-long-term-caching-with-webpack-d3eee1d3fa31 and write a integration test which does many builds making small changes in between to verify bundle paths are changing correctly.

[KyleAMathews]

Ok, figured out what it might be. I forget that Netlify by default puts your files behind Cloudflare — which changes the default caching rules adding cache-control:public, max-age=14400. In the site config you can disable Cloudflare which should also remove the cache-control header.

[KyleAMathews]

gatsbyjs.org by contrast is cache-control:public, max-age=0, must-revalidate

[gaearon]

Any tips on how to disable it? I can't find anything in Netlify's panel :-(

[gaearon]

Oh, I guess this is why:

I don't know enough about how this was set up but maybe we need to update DNS to point to Netlify? I'll defer this to @bvaughn because I don't understand how this works.

[fool]

Just FYI, Netlify never uses CloudFlare ourselves. This is configured by site owners independent of Netlify, and is a suboptimal setup for a few reasons including the one you've found but more importantly, you're using their CDN in front of our CDN which is an antipattern and slows down your webservice.

It generally works well but sometimes fails catastrophically (we've seen this a half dozen times) resulting in site downtime.

I'm talking through how to disable it with minimal downtime with a member of the Netlify project in a support ticket now so we should be in better shape soon :)

[bvaughn]

Wow, great research Dan and Kyle!

I wasn't checking work emails over the holidays so I missed this thread until now. 😄

I don't know enough about how this was set up but maybe we need to update DNS to point to Netlify? I'll defer this to @bvaughn because I don't understand how this works.

The old reactjs.org site used Cloudflare. When we made the new site live, in order to make the switch instant, we continued using Cloudflare so as to avoid a long DNS propagation cycle.

I'm pretty sure that at this point, @sophiebits (who controls the reactjs.org domain name I think?) could update the DNS to point directly at Netlify.

[KyleAMathews]

@bvaughn yeah! Should be pretty easy transition.

[sophiebits]

Seems cumbersome to cut over the SSL without downtime. I updated Cloudflare to not cache.

[nechmads]

We started to have exactly the same issue on our site in production. It seems very random, refreshing the browser sometimes bring up the site without error and sometimes fails with the same error again and again.
Did any of you solved this in the end? Kind of desperate as basically our production site is down :(

[gaearon]

In our case removing CloudFlare in front of Netlify solved this.

[KyleAMathews]

@nechmads file an issue over in the gatsby repo?

Also, added a caching page for Gatsby https://www.gatsbyjs.org/docs/caching/

[nechmads]

Thanks @KyleAMathews, @gaearon. We had to completely disable Cloudflare as well, which our sys admin definitely didn't like, but nothing much we can do right now. We had to get the site back up as soon as possible.