Endless reaload after re-deployment

[limbit]

We discovered a serious problem when using appcache and we didnt found a solution yet:

when we are using appcache, the app gets into a infinite reloading after a re-deployment on our server.

A detailed description can be found here: http://stackoverflow.com/questions/31764488/meteor-reloads-endlessly-after-re-deployment-on-server-with-appcache

packages

meteor-base
mongo
appcache
meteorhacks:npm
less
reactive-var
iron:router
accounts-password
alanning:roles
sergeyt:typeahead
email

matb33:collection-hooks
djedi:sanitize-html
http
standard-minifiers
mobile-experience
blaze-html-templates
session
jquery
tracker
logging
reload
random
ejson
spacebars
check
ground:db
mdg:geolocation


npm-container

This will keep us from publishing it on our server for the customers. Does anyone know how to solve it?

The cache even doesnt reload the new data into the cache. Whats wrong there?

[Svenskunganka]

I've got a similar issue, however I did not do any code or package changes - instead my host had to migrate my virtual instance to another node so I had to take the site down for a brief moment. When I started the Meteor app again it is now stuck in an infinite reload loop.

I do not have the appcache package, I do use Nginx and I haven't made any configuration changes at all. It was running perfectly fine before the migration. All other meteor apps this instance hosts runs perfectly fine except one.

This do not happen locally in development, only in production. There hasn't been any changes to the code or nginx configuration at all - this "just happened".

No errors in the logs on the server, no errors in Chrome's console (just deprecation warnings). No HTTP error status codes either, Meteor DDP Monitor reports that it's able to open a WebSocket connection to the server, subscriptions works as well. Cache is off.

My package list:

accounts-password               1.1.4  Password support for accounts
angular                         1.3.0  Everything you need to use AngularJS in your Meteor app
angularui:angular-ui-bootstrap  0.13.0  Native AngularJS (Angular) directives for Bootstrap.
angularui:angular-ui-router     0.2.15  angular-ui-router (official): Flexible routing with nested views in AngularJS
cfs:gridfs                      0.0.33  GridFS storage adapter for CollectionFS
cfs:standard-packages           0.5.9  Filesystem for Meteor, collectionFS
check                           1.1.0  Check whether a value matches a pattern
civilframe:angular-jade         0.0.3  Jade templating for Meteor-Angular
ejson                           1.0.7  Extended and Extensible JSON library
email                           1.0.8  Send email messages
flemay:less-autoprefixer        1.2.0  The dynamic stylesheet language + Autoprefixer
jquery                          1.11.4  Manipulate the DOM using CSS selectors
logging                         1.0.8  Logging facility.
meteor-base                     1.0.1  Packages that every Meteor app needs
meteorhacks:fast-render         2.11.0  Render your app before the DDP connection even comes alive - magic?
meteorhacks:npm                 1.5.0  Use npm modules with your Meteor App
mobile-experience               1.0.1  Packages for a great mobile user experience
mongo                           1.1.3  Adaptor for using MongoDB and Minimongo over DDP
mquandalle:bower                1.5.2_1  Use Bower packages in your Meteor app
npm-container                   1.2.0+ Contains all your npm dependencies
random                          1.0.5  Random number generator and utilities
reload                          1.1.4  Reload the page while preserving application state.
session                         1.1.1  Session variable
standard-minifiers              1.0.2  Standard minifiers used with Meteor apps by default.
tmeasday:publish-counts         0.7.2  Publish the count of a cursor, in real time
tracker                         1.0.9  Dependency tracker to allow reactive callbacks
twbs:bootstrap                  3.3.6  The most popular front-end framework for developing responsive, mobile first projects on the web.

[limbit]

which version of node.js do you use?

[Svenskunganka]

@limbit v0.10.40, you?

[limbit]

v0.10.41 (current revision). Anyway you should update your node.js because of several security issues. Did you try to clean the cache before re-deployment?

Anyway, it works for us after removing the appcache package. This is why i think it is a meteor issue.

[Svenskunganka]

Yeah cache is off (not using appcache package though). Checking the DDP monitor says that the client has to be updated (although very hard to spot due to the reloads). Looks like an issue with the autoupdate package (webapp). I'll try to grab a screenshot.

Was going to upgrade node after the migration, but ran into this and wanted to fix this issue first though.

EDIT:
Additionally, meteor run --production locally doesn't produce this issue either.

[limbit]

you could try to deploy a debugable version: meteor build ../build --server=xxxx --debug

During debugging we saw that the autoupdate package and the reload package produce the recursion but without any solution yet.

[Svenskunganka]

@limbit do you use PM2?

I managed to fix the issue by not using PM2 at all - atm I've got a temporary fix in place where I simply run node /path/to/main.js in a screen session. I'm preparing CI & Docker container for the project which will be a better solution.

[limbit]

@Svenskunganka we didn't use PM2 but it looks great.

We still get the endless reload and i hope to get some help from the meteor-team soon.

[limbit]

Does anyone else have any idea? Thanks

[limbit]

compared to the working hot-pushes it does not download the current state. Perhaps this is the cause for the endless reloading.

window.applicationCache.update() does not trigger downloading also.

[tmeasday]

Hi @limbit,

It's difficult to debug this issue without a simple reproduction we can play with. Please provide a full reproduction as described in: https://github.com/meteor/meteor/wiki/Contributing-to-Meteor#reporting-a-bug-in-meteor

If you figure out more specifics about what the problem is, please open a new ticket or ping me and we'll reopen this one.

[introrse]

We are seeing the same appcache infinite-reload bug @limbit describes. I don't have a repro to share. If get one I will. Not all re-deploys are affected, and we think the bug may occur or not-occur based on the elapsed time since most recent re-deployment. We're still trying to figure out exactly the circumstances.

In the meanwhile, here is the work-around we are using: We modify the appcache module to briefly return 404s for appcache.manifest for any IP which has requested a manifest more than 3 times in the past minute.

When the bug strikes, an affected client fetches the manifest repeatedly until it hits the 404 error, then quickly falls back to non-cache loading. (Interestingly, this takes 2-3 404s, but it does work.) The next time the client loads it gets a normal manifest again and correctly loads the (latest) cache bundle and the cache is active again until the next time the bug strikes. Not ideal, particularly if you had many users behind a proxy (we don't), but it rescues users from the infinite-reload loop and lets us continue to use appcache for offline operation.

The code added to appcache-server.js looks like this:

  ...
  // Count of manifest requests seen this past minute, by client IP                                                                        
  // We use this to 404 clients that seem to be in an update loop.                                                                         
  var recent_clients = {};
  Meteor.setInterval( function() {
      recent_clients = {};
      console.log("Clearing recent appcache-manifest clients list");
  }, 60*1000 );

  WebApp.connectHandlers.use(function (req, res, next) {
    if (req.url !== ('/app.manifest')) {
      return next();
    }

    // Look in the X-Forwarded-For header if the remote connection is to a (local) proxy
    var ip = (req.connection.remoteAddress === "127.0.0.1") ? req.headers['x-forwarded-for'] : req.connection.remoteAddress;

    if (recent_clients[ip]) {
      recent_clients[ip]++;

      if (recent_clients[ip] > 3) {
        console.log("Denying appcache manifest with a 404 to client "+ip+" after "+recent_clients[ip]+" successive manifest requests");
        res.writeHead(404);
        res.end();
        return;
      }

    } else {
    recent_clients[ip] = 1;
    }
...

[pdiniz13]

I am currently having the same issue with GroundDB

[franciskim]

Same issue here. Only difference is I am using Ubuntu's Upstart but it doesn't seem to be an issue caused by that. Meteor goes into a refresh loop for about 3-5 minutes, refreshing every 2 seconds - which is really bad for my visitors.

[shilman]

My app was infinitely reloading under 1.3 / Ubuntu + Upstart. I was running it behind NGINX and had enabled caching in NGINX. When I turned off caching the problem went away. Posting this in case it helps anybody here debug their situation.

[sebastianconcept]

Thanks for sharing @shilman that was happening in a project and we solved it with add_header Cache-Control no-cache;

It was hurting us really bad.

I'm very surprised, in the worst possible way, that this directive is not recommended in the article about setting up Meteor on nginx http://www.meteorpedia.com/read/nginx so I've just added it.

[justingrayston]

Surely having a no_cache policy on the entire app isn't the best approach here?

I used the following, which fixed the redirect loop because the document page that has the refs to the js etc must not be cached but those static files referenced can be.

#cache static
location ~*  \.(jpg|jpeg|png|gif|ico|css|js|svg)$ {
    expires 30d;
}

[sebastianconcept]

@justingrayston since your regex has |js| there I don't get how you are not-caching or how that config had made you safe from the redirect/reload loop.

[justingrayston]

@sebastianconcept doesn't the html document reload have the references to the new JS?

[sebastianconcept]

@justingrayston I'm not sure but sounds like it is worth trying

[sebastianconcept]

@justingrayston nginx: [emerg] invalid number of arguments in "location" directive in /etc/nginx/sites-enabled/...

[justingrayston]

@sebastianconcept It was working for me in staging and local. Where did you place it exactly? I am in the midst of feeding 4 kids ATM but I will have a look later and paste my working config.

[justingrayston]

@sebastianconcept You are right, you get a redirect loop with JS and mashed styling when you have css in there too. I wonder if there was a 1.3 change to how this works? I am sure this worked on an older meteor app.

When I have created non meteor apps with a client that is deployment aware, the way I model it, is that the JS is notified (by polling, web socket or service worker) that a new version is available and that triggers a reload that has a cache break url that the client will then remove once loading the new document, that then has references to all the new static js and css. This means you can cache the JS for the app for a long time, saving on bandwidth, which adds up at scale. Works a treat, and I made the terrible mistake of seeing a similarity in how meteor seemed to do it. Apologies.

This is my nginx config that I have tested a little this evening and seems to work when running meteor, meteor --production and using a bundle.

location / {

        proxy_pass http://*****.***.***.**.local;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forward-Proto http;
        proxy_set_header X-Forwarded-Proto http;
        proxy_set_header X-Nginx-Proxy true;
        # no cache
        expires -1;

        # cache static
        location ~*  \.(jpg|jpeg|png|gif|ico|svg)$ {
            expires 30d;
            add_header Cache-Control "public";
        }
        proxy_redirect off;
    }

I still stand by what I said, that a blanket no cache rule isn't good for bandwidth costs at scale. I just wish that you could cache JS and CSS for long periods. @tmeasday are we way off here, have we missed something, should we be able to cache those assets?

Please feel free to pick over and amend, just let me know :)

[tmeasday]

Are we still talking about an issue with appcache and endless reload?

@justingrayston I think in general it's not a good idea to change caching headers in a proxy layer. The Meteor server is the one that should decide if the resource can be cached.

In the case of Meteor the URL of JS+CSS assets changes within the boilerplate HTML when you deploy a new version, so meteor serves a long-lived Cache-Control header for those two files, but not caching headers for the boilerplate itself.

Infinite reload loops sound like issues with the old boilerplate being cached (either by the proxy, or perhaps the browser in the appcache case) when it should not be.

[justingrayston]

Yup we are.

In the case of Meteor the URL of JS+CSS assets changes within the boilerplate HTML

That fits with what I was thinking.
You are so right, not sure why cache got into the config because it is nothing but a proxy. Copy paste config and then try to debug I think.

So really we should add neither caching or no_cache rules. Meteor will take care of it.

Testing now, and thanks for the response!

I will confirm all is good.

[justingrayston]

Confirmed. I see good long cache headers on the JS and CSS when removing everything from nginx. Reloads work just fine. This is why it worked on the other project that isn't upgraded to 1.3, I just checked the config and it doesn't do anything to the cache.

Thanks @tmeasday

Phew.

    location / {

        proxy_pass http://*****.***.***.**.local;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forward-Proto http;
        proxy_set_header X-Forwarded-Proto http;
        proxy_set_header X-Nginx-Proxy true;


        proxy_redirect off;
    }

[sebastianconcept]

@tmeasday just to let you know I was having the reloads while not using appcache in a Meteor 1.3 app behind nginx

[justingrayston]

@sebastianconcept I haven't added appcache, so when I said

Yes we are

above, I wasn't exactly correct. The app I was having issue was pretty vanilla. Without any cache declaration in the nginx proxy it now works just fine.

[sebastianconcept]

@justingrayston sure, np. Thanks for sharing

[MichaelJCole]

@sebastianconcept @justingrayston @tmeasday

I'm also having a reloading loop problem. It seems like this thread recommends to remove nginx caching. Somehow my configs had this:

    location / {
        proxy_pass $backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade; # allow websockets
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header X-Forwarded-For $remote_addr; # preserve client IP

        # this setting allows the browser to cache the application in a way compatible with Meteor
        # on every application update the name of CSS and JS file is different, so they can be cache infinitely (here: 30 days)
        # the root path (/) MUST NOT be cached
        if ($uri != '/') {
            expires 30d;
        }
    }

And what I understand is to remove the last commented part because Meteor does this itself.

The above commit also includes this at the bottom of of the location directive:

        add_header Cache-Control no-cache;

My understanding is this would add alot of traffic to the server. Is it necessary to resolve the reloading issue?

For me, redirect issues started around 1.3. Is there an nginx config guide for 1.3?

Thanks!

[justingrayston]

Yes remove both, the expires and the no cache header, just let nginx proxy and leave cache headers to Meteor. Works fine with the config I posted.

[menelike]

Yes remove both, the expires and the no cache header, just let nginx proxy and leave cache headers to Meteor. Works fine with the config I posted.

FYI: This thread deals with the same problem https://forums.meteor.com/t/app-constantly-refreshing-after-an-update/23586/112

Disabling any caching related stuff, especially

        if ($uri != '/') {
            expires 30d;
        }

made it work, thanks a lot!

This is my working location block:

    location / {
        proxy_pass http://app;
        proxy_redirect      off;
        proxy_set_header    Host              $host;
        proxy_set_header    X-Real-IP         $remote_addr;
        proxy_set_header    X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Proto $scheme;

        # WebSocket specific
        proxy_http_version 1.1;
        proxy_set_header    Upgrade           $http_upgrade;
        proxy_set_header    Connection        "upgrade";
    }

[azfx]

If you are using meteor app cluster, make sure you add ip_hash; in the the nginx configuration's upstream block.

Example :

upstream awesome-app-servers {
  ip_hash;
  server 123.45.67.8:3000;
  server 123.45.67.9:3000;
}

[introrse]

Update: We appear to have solved our instance of the endless-reload appcache problem.

Given this has been going on for years with no solution in sight, and given intent to fix even seems waning, I hope it may work for others. Even if appcache is eventually going away, the transition will take years and we want to continue to use it during the transition.

The inspiration was pauldulong's comment about onMigration in the Meteor forums thread [App constantly refreshing after an update](https://forums.meteor.com/t/app-constantly-refreshing-after-an-update/23586/142].

We weren't doing any of the caching that has caused (some) others' instances of this problem. Instead, Meteor appears to be reloading from the moment it determines a code refresh is needed, long before the appcache has had a chance to update.

To prevent that, we hook Meteor's onMigrate interface to control when reloads can happen, and monitor window.applicationCache events/status to get the timing correct for the one reload we do eventually need. I'm including a simplified-but-complete version below. (Note that some have reported success with just returning false from their onMigration callback, but that didn't work for us.)

Although our experience so far is limited, we appear to be seeing reliable single-reload appcache updates now.

if (Meteor.isClient) {
    var migrationstate = 0;  // 0==none, 2==appcache update in progress

    // Monitor appcache downloading state to trigger a reload
    // once the download is complete
    var appcache_downloading = false;
    if (window.applicationCache) {
        window.applicationCache.addEventListener('downloading',function() {
            if (appcache_downloading) return;
            if (window.applicationCache.status !== window.applicationCache.DOWNLOADING) return;
            appcache_downloading = true;
            console.log("mig: appcache downloading started");
            window.applicationCache.addEventListener('updateready', function() {
                console.log("mig: appcache ready for restart");
                location.reload();
            });
        });
    }

    // Hook Meteor's onMigrate to detect the out-of-date condition, trigger an app-cache
    // update, and prevent Meteor from prematurely reloading
    Meteor._reload.onMigrate( "useractivity", function(retry) {
        console.log("mig: got onMigrate event indicating new software is ready", migrationstate);
        if (migrationstate>0) return false;  
        if (window.applicationCache.status !== window.applicationCache.DOWNLOADING) return false;

        if (window.applicationCache && (window.applicationCache.status === 1)) {
            window.applicationCache.update();
            migrationstate = 2;
            console.log("mig: triggered appcache to check for and download update");
            return false;
        } else {
            console.log("mig: allowed hotpatch of update given no appcache");
            return [true];
        }
    });
}

[sethjgore]

FWIW, this piece of code stopped my app from reloading infinitely.

add it to client/main.js

Meteor._reload.onMigrate(function () { return [false]; });

[chrisbutler]

this should be re-opened....

[tonymckendry]

Agree with @chrisbutler - we have been using @introrse 's solution above for a long while now, sometimes we are able to remove it for a short time, but generally on the next deploy the issue will begin happening again. The app is only on screen for about 2 seconds before the reload happens, in an endless loop.

Here is our package list:

meteor-base@1.2.0
mobile-experience@1.0.5
mongo@1.3.0
session@1.1.7
jquery@1.11.10
tracker@1.1.3
es5-shim@4.6.15
ecmascript@0.9.0
aldeed:simple-schema
meteorflux:meteorflux
accounts-password@1.5.0
aldeed:collection2
check@1.2.5
raix:push@3.1.0-pre.1
meteorhacks:picker
react-template-helper
msavin:mongol@2.0.1
browser-policy@1.1.0
peerlibrary:server-autorun
ejson@1.1.0
http@1.3.0
retry@1.0.9
reactive-var@1.0.11
percolate:synced-cron
meteorhacks:aggregate
peerlibrary:reactive-mongo
peerlibrary:reactive-publish
peerlibrary:subscription-data
peerlibrary:computed-field
force-ssl@1.1.0
mantarayar:shortid
react-meteor-data
standard-minifier-css@1.3.5
standard-minifier-js@2.2.0
static-html@1.1.12_1
dburles:factory
dispatch:mocha-browser
tap:i18n
shell-server@0.3.0
fastclick@1.0.13
hwillson:stub-collections
xolvio:cleaner
agoldman:sparkpost-mail
crosswalk@1.7.1
edgee:slingshot
space:tracker-mobx-autorun
service-configuration@1.0.11
mdg:validated-method
dispatch:mocha
johanbrook:publication-collector
kadira:flow-router
email@1.2.3
npvn:heapsave
mdg:meteor-apm-agent
dynamic-import@0.2.0
okgrow:analytics

[wsoul]

I had the same problem.
My app is deployed via mup as two instances on the same server with nginx load balancing.
I have figured out that autoupdateVersion of meteor apps instances are different. That's the cause of the infinite reloading. But instances are the same, at least should be, since autoupdateVersion is a hash of the app code. But they weren't.

Cause: On deploy, mup.js file was dynamically created in the root of the app for instances with different port and it was counted as part of the app. That's why autoupdateVersion was different.

Solution: Do mup deploy from a directory which are not loaded as part of your app code. For example, i've created .deploy directory.

I suppose some people had the same scenario.