Progressive Web App prefers server data over local cache when internet available

[charliewhitfield]

Godot version

Official 3.4.1 and 3.4.2

System information

Windows 10, Brave browser

Issue description

Opening an installed PWA behaves as follows:

• If internet is available: Always downloads all data from server.
• If internet is not available: Uses local cache data.

My expectation is that an installed PWA will use cached data unless the internet is available AND the app on the server has been modified. [edit:] even when internet available, downloading updates if they exist in the background. (Don't ask me how this magic works!)

(Note: I am extremely ignorant on the topic of PWAs and my expectation is based on reading links provided in the Godot News article from March 20. There is no Godot documentation yet for PWAs.)

Steps to reproduce

It's hard to see the network traffic in a small project. So here is our large Planetarium PWA for testing:
https://www.ivoyager.dev/app-debug2/planetarium-debug2.html
(This is an "out of the box" PWA export from 3.4.1.stable. No custom template or html.)

1. Follow link to large PWA app.
2. Install.
3. With internet available, start/stop app several times. This always takes some time for a large project with network traffic evident in Task Manager. (For our Planetarium, >10 sec at ~80 Mbps.)
4. Set wifi to airplane mode, and start/stop app several times. This happens much more quickly and is obviously using cached data. (Our Planetarium takes ~1 sec on my average desktop.)

Minimal reproduction project

Any project exported as PWA and loaded to web server should work, though it may need some large files to see network traffic.
Use ours if you like: https://www.ivoyager.dev/app-debug2/planetarium-debug2.html

[Calinou]

cc @Faless

AND the app on the server has been modified.

How does the PWA know if data on the server has been modified without requesting it beforehand?

[charliewhitfield]

@Calinou, PWA intro doc linked in the Godot news article says "Whenever you go to a website, it's up-to-date, ..." and "They can read and write files from the local file system": https://web.dev/what-are-pwas/

I don't know how this magic happens or if this is "out of the box" Godot PWA behavior. The docs do say that the manifest is checked for certain changes. I guess one could force an update with a changed start_url (by including version number in export name), but I don't know if this is general practice.

[Faless]

This was mostly done by design, assuming your server is properly configured, it should not generate extra traffic (beside the validation request) and the cached version will be used anyway.
This is done mostly to avoid issues when the cache is out of sync with the server so cache is always revalidated when internet is available. I.e., this ensures "Whenever you go to a website, it's up-to-date, ..."

[charliewhitfield]

My apologies @Faless, but I'm not following your comment. Does "validation request" = "redownload all data"? I'm not sure if you didn't understand my post or if you are saying that my server is misconfigured (certainly a possibility).

[Faless]

you are saying that my server is misconfigured (certainly a possibility).

Yeah, that was my guess, running the inspector I see that the other files actually get retrieved from cache as expected (e.g. the wasm binary), so I can't really exclude a bug in our service worker.

I'll have a look, my understanding was that caching capabilities would get enhanced by the service worker (i.e. default cache and cache API are the same, the service worker one with higher priority). But this might be one of the cases where the default browser cache refuses keep the pck because it's too big, while the service worker cache would keep it.

EDIT: btw, I love the app ❤️ , great work!

[charliewhitfield]

Thanks for the compliment! The PWA functionality is insanely valuable for the Planetarium!

[Faless]

@charliewhitfield a bit of an update (since the final fix might require some more discussion).
I've created a branch ( https://github.com/Faless/godot/tree/js/3.x_pwa_prefer_cache ) where the serviceWorker uses an offline first approach (i.e. it always tries the cache first, and only falls back to network when cache is missing).

The main problems are:

• New game versions must be exported fully (not just the PCK) because the service worker needs to be updated too.
• When a new version is released, returning users will still get the old version at first.
  • We can detect it, and do something about it.
  • For now, the branch does an very annoying alert and force-reload of the page.

How should we handle updates is kind of a tricky question. In some cases, you really want to force-reload because the old one will not work completely (for partly-online experience), but in many cases just telling the user is fine. In other cases you might not even care about that.

A potential solution would be to do nothing by default (the user will get the updated version anyway at the next browser restart, maybe even after a simple page reload), and implement a hook for it (i.e. let you specify a JS function that will be called).
We could even add it as a signal in the JavaScript singleton ("pwa_updated"), which could be nice (from there you can now easily call JS).

[charliewhitfield]

@Faless, This is precisely what we need for the Planetarium, and I would think any large project that will take more than several seconds to download. Mozilla docs describe offline-first as the "most popular strategy". But perhaps for many small games this is not true (?).

Any chance I can get my hands on the updated service worker without compiling myself? (No hard feelings if not. I need to learn...) [Never mind. I can figure out the js change from your commit.]

[Faless]

@charliewhitfield it's a bit tricky, you can probably just replace the godot.service.worker.js (in the zip) with the updated one, but you will still need to recompile the engine (windows, linux or mac editor) if you want to detect the updated version (because that code is injected in the HTML by the editor exporter).

[charliewhitfield]

So the exported project will differ in project.html and project.offline.html in addition to project.service.worer.js? Perhaps this is the motivation I need to finally learn to compile Godot myself...

Follow-up question. This doc says that update is triggered by a change in server worker "var cacheName". However, it looks to me that the export "const CACHE_NAME" (assuming this is equivalent) is derived from project name only. Does one need version in the project name? (If so, wouldn't that make it a different application?)

[Faless]

@charliewhitfield as you can see in the diff:
3.x...Faless:js/3.x_pwa_prefer_cache
CACHE_NAME now also depend on CACHE_VERSION, and CACHE_VERSION is set by the exporter based on current time and ticks (so collisions a quite unlikely).

[charliewhitfield]

I built Godot from Faless's 3.x_pwa_prefer_cache branch (commit eeffa36) and then built our Planetarium as a PWA. It works! Here's the link for anyone that wants to try it:

https://www.ivoyager.dev/app-debug/planetarium-debug.html

@Faless, Although offline-first is working, there's some funny business going on with 1st visit and updates after installation. I'll post again when I've worked through all the permutations and can be more specific.

At the risk of repeating myself, let me say again how enthusiastic I am about the PWA addition! I didn't know what a PWA was a few weeks ago. Now I see it as the key feature that:

1. Lets us easily distribute our Planetarium with a simple link! (Distribution via Steam, for example, can be problematic for educational use.)
2. Makes it much more re-engageable for the user with no long download wait after installation.
3. Will make it much less bandwidth intensive on our web server, which may save us as our user base grows.

[charliewhitfield]

@Faless, Here's a detailed report for commit eeffa36 exported PWA behavior (link to web app in post above). I've repeated theses observations >3 times, including app updates on web server, using Brave browser. To be clear, when I say "from cache" below, what I mean is that it happens quickly (~1s) without any evident network traffic as observed from Task Manager (full download for me is ~15s with quite obvious network traffic).

1. On first visit after clearing browser data (see note* below on clearing browser data):
   • The app downloads and then starts. Install option is not available yet.
   • The app is running (clearly with all assets). Install option is not available yet. There is a lot of network traffic. This lasts for maybe 10 or 15 secs. (The time and network traffic is consistent with a full 2nd download, I think.)
   • The "reloading!" prompt occurs interrupting the app, and exactly at this time the Install option appears and network traffic stops.
   • After pressing OK, the app reloads from cache (very quickly without network traffic) and restarts. Install option is available.
2. On repeat visits (without installing) the app loads from browser cache (quickly with no network traffic). There is no prompt interruption or restart or re-download. Perfect!
3. Start app after installing (no update available): Starts from cache with or without internet. Perfect!
4. Start app after installing (update available):
   • The app will start old version (quickly).
   • App runs for 10-15 sec during which time there is a lot of network traffic. (From below I conclude it is downloading update in the background.)
   • The "reloading!" prompt occurs interrupting the app. At this time, network traffic has stopped.
   • After pressing OK, the app restarts from cache (quickly w/out network traffic) but it is running the pre-update version. (Yeah, that's weird! I've verified the restart to old version 4 separate times.)
   • After closing and reopening, the app starts from cache (quickly w/out network traffic) running the updated version.

*Note on clearing browser data: Interestingly, even before installation, the site seems to be immune to Brave clear browsing data/clear cached images and files. Doing so doesn't remove cached data as evidenced by fast revisits without network traffic. (In Brave advanced options there is a separate option to clear Hosted App Data which I did not try. I did not want to kill other web apps.) The only way to clear the browser that I know is to install and then uninstall the app with checkbox checked Also clear data from Brave (www.ivoyager.dev).

Conclusion: Background download of updates is happening. But something isn't going right with the prompt.

My expectation from source below** is that the app will continue to run using the cached version while downloading an available update in the background. Ideally, the user would then have a choice - Restart or Continue - so they aren't forced to exit the current session.

**OK, I'm not sure if I'm interpreting this correctly, but regarding background updating: "A new service worker is installed in the background, and the previous one (v1) works correctly up until there are no pages using it — the new Service Worker is then activated and takes over management of the page from the old one."

[charliewhitfield]

Quick follow-up test using Faless' commit bf61f9c (3.x_pwa_prefer_cache branch before the prompt addition):

This one gives a very smooth user experience. App quietly updates in background. Runs new version on restart. There still seems to be a "double download" on first visit (if I'm interpreting correctly; see point 1 in post above) with the Install option appearing only quite a while after the app starts.

Edit: commit bf61f9c is the build we used for our "official" Planetarium: https://www.ivoyager.dev/app/planetarium.html
The only thing more I could ask for is a prompt after update has downloaded giving an option to restart.

[charliewhitfield]

@Faless, Another follow-up on commit bf61f9c:

I've been using the "silent update" version for a couple weeks now, through several app updates. One strange observation is that the html file isn't updating when the rest of the app is. I have visible version numbers in both our loading page (a custom html) and the running app. The running app version updates perfectly every time (so, the .pck file at least) but the loading page does not ever update once installed. New loading page is only updated if I do full uninstall and then revisit page.

Been looking through .service.worker.js and I can't see anywhere that .pck is treated differently that .html. I guess it could be web server weirdness. I always clear cache for our web server and then cloudflare after each update.

[akien-mga]

Fixed by #57485.