Enable installing Kiwix JS as a Progressive Web App

[Jaifroid]

A PWA is usually an offline version of a Web page/site. The Service Worker will cache content and only refresh it when it has internet access.

While our model is somewhat different (never need access to a server), making Kiwix JS work as a PWA could be an easier way of distributing the app, instead of using proprietary browser extensions. In our case, we would serve Kiwix JS from, say https://m.kiwix.org or https://kiwix.org/kiwix-js . But once downloaded, with or without a packaged ZIM, the app would work offline except for checking for code updates when the user asks it to (or on a schedule).

Google has gone all-in on PWAs. See https://developers.google.com/web/progressive-web-apps/ on the advantages, in particular:

Progressive Web Apps are installable and live on the user's home screen, without the need for an app store.

Microsoft is also heavily supporting the model. See https://www.pwabuilder.com/ (a Microsoft service). MS is also automatically placing major PWAs in the Microsoft Store, though developers can own the code if they prefer. Nevertheless, PWAs don't need a store model, and are intended to do away with that model.

I think, for Kiwix JS, such a system would require an easy way to download ZIM contents, as suggested in #114.

It's unclear to me at the moment whether accessing Kiwix JS as a PWA would overcome the need to pick the ZIM file (if packaged). I believe there is a maximum size for a PWA of approx. 20GB, but it can be smaller depending on free disk space of the user. If sufficient space is available, the ZIM could be "cached" (stored in indexedDB), enabling packaged apps.

EDIT: I think the ZIM would still need to be in the file system, which implies having to download it as an extra step, because we can't have binary access to files in indexedDB without loading the whole file into memory...

[kelson42]

@Jaifroid Sounds to be a good idea to make the overall UX even smoother.

[mossroy]

Well, it might be another way to distribute the app, but I don't see as an enhancement in terms of UX.

We already have an online version of kiwix-js (used for testing) : https://kiwix.github.io/kiwix-js. It's hosted on github, I update it after each release.
It's true that we could probably make this web page a PWA, so that it becomes available when offline. It could be another way to distribute the app (without the need to use browser extensions, and without their CSPs).

But, in terms of UX, I see it more as a regression compared to our already-existing modes.
Kiwix is an offline application : I find it strange to need to connect to an online web page, use it at least 30 seconds (see https://developers.google.com/web/fundamentals/app-install-banners/) in order to be able to install it. I don't think a PWA can be distributed and installed offline (but I might be wrong?), whereas a browser extension or a mobile application can.

The idea of a PWA is to smoothly transition between a web page and a local application. But I don't think we want this transition : we only want to install the app once, and then use it offline. It makes little sense for us to connect (the first time) to a web page in order to read the content of a local ZIM file (because, as you said, it does not solve the need to manually pick a local ZIM file).

I'd be glad to work on that, but the installation process seems counter-intuitive to me in our use-case.

It might make more sense IMHO that online wikipedia would be a PWA, and enable an offline mode. But it does not really fit with the concept of a big ZIM file containing all the content, that would need to be downloaded at some point.

[Jaifroid]

OK, it's just an idea. I read more an more articles about how PWAs will be the future of all apps, and that app stores won't be necessary. PWAs can be packaged in app stores (that's what MS is pushing for) though personally I think the whole point of them is to be able to distribute apps without a proprietary store model. If you think about it, going to a web page to install the app, especially an open-source one, and going to a vendor's store to install it, both involve having an online connection, and the store is going to be a lot slower and require longer online.

PWAs are Service-Worker centric and leverage all the possibilities of that technology. A PWA would also solve our problem with Service Worker requiring to be run and "installed" from https:// (in Web context).

They're supposed to be a native app experience on devices, with push notifications for updates, etc., not just an "offline" web page (at least according to Google and Microsoft's hype....). We'll see if it materializes quite like that!

[Jaifroid]

Just an update on where I'm at with PWA versions of the app (I've been doing this on Kiwix JS Windows, as it is an easier testbed for me to try out experimental things, to see what works and what doesn't before considering porting here). Some main milestones are the following:

• I made it possible to install the app as a PWA in browsers that support this, simply by visiting https://kiwix.github.io/kiwix-js-windows . This works best in browsers that issue
  onbeforeinstallprompt, which are currently only Chromium-based browsers (including new Edge);
• There is no-longer any need to interact with the page for 30 seconds before this prompt is issued: it is issued immediately if the app fulfils the conditions (which it does). The app intercepts this prompt and puts an Install button on the Configuration page (see screenshot below), and if user clicks on this, the install prompt will show. In Firefox Mobile, onbeforeinstallprompt is not yet supported, but the app is recognized as a PWA and the "Add to Homescreen" icon is shown in the address bar. Firefox Desktop appears not to support PWAs as yet.
• The installed app works offline, but the experience is not optimum because the user still must pick the ZIM file each time the app starts, and we have a problem with slow binary search on Android (on PC it is pretty fast);
• One way to overcome the requirement to pick the ZIM file on each app launch is to include native filesystem code: on Windows, if the app is packaged as a UWP app, then it has access to WinRT runtime, and can open files in its package without the need to pick them (also remember picked archives). This has the advantage over the current UWP app that Service Worker is of course fully supported (it is a requirement for PWAs);
• I have released a UWP version of the PWA signed with the Kiwix certificate, which can be side-loaded on any Windows 10 device, whether x86, x64 or ARM, though on Windows Mobile (where Service Worker is not supported) it runs as a Hosted Web App instead of a PWA;
• Another way to provide File System support may be Investigate adding Filesystem access with the File and Directory Entries API #586, which is now supported by all the major browsers except Safari in a read-only manner, though I suspect on some browsers we would run into quota issues in trying to store some of our very large ZIM files in the virtual file system;
• Another way might be to use a successor to Phone Gap/Cordova, e.g. https://capacitor.ionicframework.com/ , which provides native bridges to mobile platforms.

[Jaifroid]

Very useful demo app showing what PWAs are capable of today:

https://whatpwacando.today/

[Jaifroid]

With the optimizations, including peter-x's least-recently-used file cache, binary search is now acceptable on the Android PWA in ZIMs up to a few GB in size.

For wikipedia_en_medicine_maxi_2020_03.zim, in jQuery mode, the landing page opens within ~5s, and 15 search results are returned within ~3s. A search result that redirects to a differently titled page takes ~4s. A page load that also has to load a single uncached stylesheet takes ~6s. Page times can be appreciably longer in Service Worker mode, because JS has to be decompressed form the ZIM for each page (until it is cached).

In full English Wikipedia, however, the search is still ridiculously slow (unusable, effectively).

[Jaifroid]

In full English Wikipedia, however, the search is still ridiculously slow (unusable, effectively).

Just to update this: this sentence is no longer true at least on my Android. It's actually usable with the latest browser updates to Chrome and Edge (Android).

[Jaifroid]

As promised in #727, some updates on PWA functionality. It could be an interesting alternative for Firefox users, and maybe even could be packaged transparently in a Firefox Extension -- this should work because the code would be drawn from a secure (https:) server, not from locally packaged code. There is a version running at https://pwa.kiwix.org/ .

As mentioned above, a PWA that leverages Service Worker technologies, caches itself (using Cache API) on first visit to the PWA site. Thereafter it works offline (if an offline-first strategy is used), but as per the Service Worker specification, the worker will periodically check for updates to its own code when the machine is online. The Cache needs to be numbered according to the app's release version, which keeps caches separate for each new version. The app can check if an update is waiting, and can tell the user that it will be installed on next relaunch of the app. The installation doesn't need any special coding -- it's handled by the Service Worker specification, and the Cache is upgraded as part of this.

There are a couple of downsides for Firefox users -- again, Firefox is dragging its feet on the key technologies, citing security concerns:

• Firefox doesn't support the Window.onbeforeinstallprompt event for PWAs, so the user can't install it as an app, and can only bookmark the page (the page can still be used offline). This is counter-intuitive in offline mode, because the browser has to visit the "site" to run the app, and it indeed does run offline, but the browser appears to be accessing the site. In fact, it is merely accessing its cached copy of the code. The experience is much better on Chromium browsers, because the app is installed and dispenses with most of the browser chrome. The user does not feel they are accessing a web site (and they aren't).
  • It may be possible to overcome this lack of support for installation by wrapping the PWA in a Firefox extension. As the code would appear to be running from https (but in fact from the Cache), Service Worker should work in this context. It could even be a transparent upgrade for current Extension users?
• Firefox doesn't support the File System Access API, whereas both Chrome and Edge do. This is relatively easy to support, and is a natural companion to installation of the PWA. Firefox users have to pick the ZIM file each time, whereas for Chromium browsers, users just have to give permission to read the last-loaded file or folder with a single click. There is upcoming support for dispensing with the permission check once a PWA has been installed.

[mossroy]

Could you provide the URLs of corresponding bugzilla open issues (if you have them)?

[Jaifroid]

Regarding installation of PWAs, there's extensive discussion here: https://bugzilla.mozilla.org/show_bug.cgi?id=1407202
and DEV discussion here: https://bugzilla.mozilla.org/show_bug.cgi?id=1264819
It seems they've decided it's not a priority on Desktop systems and probably "wontfix".

Regarding File System Access API, see mozilla/standards-positions#154 . I can't find a bugzilla specifically, possibly because the spec is still draft, though implemented and being used in Chromium browsers.
You can see browser support on Mozilla's own documentation for part of the API: https://developer.mozilla.org/en-US/docs/Web/API/FileSystemHandle .

[Jaifroid]

PS Chromium devs provide a polyfill for the File System Access API that uses it where available, and falls back to earlier methods where not. See https://web.dev/file-system-access/#polyfilling. Having said that, the API isn't difficult, and easily substitutes for a standard file picker without really needing a polyfill. But it could simplify code not to have different types of file picker showing according to browser API support.

[Jaifroid]

Updated PWA install experience in Windows shown below (this is browser-provided -- in this case by Edge Chromium). I have also tested the install experience on Ubuntu using Chromium browser for Linux. It creates a .desktop file in the user's Home directory, which can be copied (manually) into the menu shortcut directory. I found I had to edit the .desktop as the path erroneously included the snap container by which Chromium had been installed (by the Ubuntu Store). (Perhaps if I had installed Chromium via apt-get it would not have had that issue.) On both OS's, the installed shortcut can be used to open the app in its own window without browser chrome, and the File System Access API is used for remembring the picked file or directory between app loads. The only difference from a native app is that the user is prompted for permission to read the last loaded file or folder (it's a single-click to give permission).

[Jaifroid]

This news may be of interest:

https://blog.pwabuilder.com/posts/microsoft-and-google-team-up-to-make-pwas-better-in-the-play-store/

PWABuilder can make packages for the Microsoft Store and for the Play Store from a simple PWA URL, and allows for a certain amount of customization. I've tested these with pwa.kiwix.org. After a bit of bugfixing at pwa-builder/CloudAPK#78, this now works very well. Obviously it's not in itself a build solution, but their code is available to use.

Note that the Android app generated is not really useable due to the issue noted in #581 - everything works, but slowly. It's pretty bad for the full English Wikipedia, and tolerable for WikiMed. Also, the File System Access API is not supported in Android Chrome/Edge (yet?), which means we are stuck with the plain old filepicker. So this is more a curiosity than anything else.

On Windows, the generated PWA package is pretty fast and it leverages the File System Access API for a near-native experience.

[Jaifroid]

This issue would be closed by #771.