Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a Service Worker for offline support #2917

Closed
1 of 2 tasks
nolanlawson opened this issue May 8, 2017 · 13 comments
Closed
1 of 2 tasks

Add a Service Worker for offline support #2917

nolanlawson opened this issue May 8, 2017 · 13 comments

Comments

@nolanlawson
Copy link
Contributor

nolanlawson commented May 8, 2017

(Opening an issue to allow for some discussion, rather than just submitting a big ambitious PR.)

A Service Worker would give Mastodon offline support for static assets, improve overall performance, and also provide an "install to home screen" prompt in supported browsers (notably Chrome and Opera, soon Firefox and Edge). It could also fall back to Application Cache to support non-SW browsers like Safari and Edge.

It seems to me that the offline-plugin would be the best candidate here, since it uses Webpack's innate understanding of the asset graph (perhaps at the expense of removing more control over the Service Worker itself though).

I noticed there was already some preliminary work on this in #2617 but it got abandoned. I started picking this up again myself (see this commit) but so far I don't have the SW actually working yet (probably due to a misconfiguration in offline-plugin).

Goals:

  • cache static assets indefinitely
  • cache index.html for offline support (at the very least, the page should load while offline)
  • get the install prompt banner

Questions:

  • how often to update the SW?
  • should we optimistically cache extra assets? (e.g. emoji images)
  • what kind of expiry system should we have for assets?
  • do we need to give admins special instructions for Cache-Control on the sw.js file? I believe the spec tells browsers to ignore Cache-Control and update it every ~24 hours, but this is worth verifying.

  • I searched or browsed the repo’s other issues to ensure this is not a duplicate.
  • This bug happens on a tagged release and not on master (If you're a user, don't worry about this).
@Gargron
Copy link
Member

Gargron commented May 8, 2017

The offline-plugin run into some issue with CDN_HOST usage on glitch.social, and I had no idea how to debug this in production, so I gave up until it's in a separate PR by someone who knows more than me.

@nolanlawson
Copy link
Contributor Author

I have some experience with SWs and would like to look into this. (Not trying to prevent anyone else from working on it; just declaring my intent to tackle this next. 😄)

@nolanlawson
Copy link
Contributor Author

@beatrix-bitrot Do you remember what the issue was that you ran into with Service Worker + CDN? I have a local production environment I can test in.

@nolanlawson
Copy link
Contributor Author

I've been looking into this a bit (see this branch). I have a solution that mostly works – it's able to register a ServiceWorker to statically cache all the Webpack assets, including fixing the CDN_HOST problem.

Unfortunately there's not much point yet, because as of right now this solution isn't buying us much unless it can cache the actual /web/timelines/home and/or /web/getting-started HTML pages (without those, it won't actually work offline, and it won't trigger Chrome's install prompt). And that part's proving a bit difficult because those files are server-generated and seem to rely on a cookie to set the csrf-token, which the ServiceWorker doesn't seem to have access to, so we may need to dynamically generate it from the HAML or try to make the page static and get the token some other way. I'll keep digging.

@nolanlawson
Copy link
Contributor Author

OK, in order to solve this problem AFAICT we will need the following:

  1. A precompile step or an API to generate the main HTML of the app without the CSRF token
  2. postMessage() from the UI thread to the SW thread to store the CSRF token whenever it changes
  3. Service Worker dynamically generates the HTML with the most recent CSRF token

I'm not sure I see any other way around this, but then again I'm also extremely ignorant of Rails and HAML.

@ramlmn
Copy link

ramlmn commented May 13, 2017

I want to address some issues that we might face building a SW

  1. emoji

    If we use a plugin like offline-plugin or sw-precache-webpack-plugin, they generate file maps and hashes that are only generated by webpack, where emoji's are not. And caching emoji's individually is impossible. I already suggested @nolanlawson like generating a json file of emojis like

    {
      "2049": "<g fill=\"#ff5a79\"><path d=\"m6 42.4h10l4-40.4h-18z\"/><ellipse cx=\"11\" cy=\"54.4\" rx=\"7.7\" ry=\"7.6\"/><path d=\"m40.2 2.1c-11.6.7-17.7 7.3-18.2 19.2h11.7c.1-4.1 2.5-7.2 6.7-7.7 4.2-.4 8.2.6 9.4 3.4 1.3 3.1-1.6 6.7-3 8.2-2.6 2.8-6.8 4.9-8.9 7.9-2.1 3-2.5 6.9-2.7 11.7h10.3c.1-3.1.3-6 1.7-7.9 2.3-3.1 5.7-4.5 8.5-7 2.7-2.3 5.6-5.1 6-9.5 1.6-12.9-9-19.1-21.5-18.3\"/><ellipse cx=\"40.5\" cy=\"55.6\" rx=\"6.5\" ry=\"6.4\"/></g>",
      "2122": "<g fill=\"#4d5357\"><path d=\"m2 2v7.5h10.3v24.5h7.5v-24.5h10.3v-7.5z\"/><path d=\"m54.5 2l-6.6 13.2-6.5-13.2h-7.5v32h7.5v-13.2l6.5 13.2 6.6-13.2v13.2h7.5v-32z\"/></g>",
      "2139": "<circle cx=\"32\" cy=\"32\" r=\"30\" fill=\"#4fd1d9\"/><g fill=\"#fff\"><path d=\"m27 27.8h10v24h-10z\"/><circle cx=\"32\" cy=\"17.2\" r=\"5\"/></g>",
      "2194": "<circle cx=\"32\" cy=\"32\" r=\"30\" fill=\"#4fd1d9\"/><path fill=\"#fff\" d=\"m12 32l15.5 16v-11.4h9v11.4l15.5-16-15.5-16v11.4h-9v-11.4z\"/>",
      "2195": "<circle cx=\"32\" cy=\"32\" r=\"30\" fill=\"#4fd1d9\"/><path fill=\"#fff\" d=\"m32 52l16-15.5h-11.4v-9h11.4l-16-15.5-16 15.5h11.4v9h-11.4z\"/>",
      ...
    }
    
    Code used to generate emoji.json
    const fs = require('mz/fs');
    const path = require('path');
    const DomParser = require('dom-parser');
    
    const emojiDir = path.resolve(__dirname, 'emoji');
    
    fs.readdir(emojiDir)
      .then(files => {
        let json = {};
        files.forEach(filename => {
          if (filename.endsWith('.svg')) {
    
            let emojoName = filename.substr(0, filename.length - 4);
            let data = fs.readFileSync(path.resolve(emojiDir, filename), 'utf-8');
    
            let parser = new DomParser();
            let dom = parser.parseFromString(data);
    
            let content = dom.getElementsByTagName('svg')[0].innerHTML;
    
            json[emojoName] = content;
          }
        });
    
        fs.writeFile(path.resolve(__dirname, 'emoji.json'), JSON.stringify(json));
      });

    Then, we can use fetch event in SW, open the json file, generate a custom response, and return it to the user.

    Sample SW fetch code for emoji
    self.onfetch = event => {
      let url = new URL(event.request.url);
      
      // ...
      
      if ((/\/emoji\/\w+.svg$/i).test(url)) {
        caches.open(cacheName)
          .then(cache => {
            cache.match(new URL('emoji.json', baseURL))
              .then(response => response.json())
              .then(emojiData => {
                let id = ''; // Extract svg filename from url
                if (emojiData[id]) {
                  let body = `<svg viewbox="0 0 64 64">${emojiData[id]}</svg>`;
                  return new Response(body, { status: 200, headers: clunkyHeaders });
                } else {
                  // Return 404 or try to fetch from server
                }
              });
          })
      }
      
      // ...
    
    };

    The emoji.json generated for testing is like ~3MB, but we can bring that down using svgo.

  2. videos

    Handling videos is one of the hardest parts of SW. Unlike others videos are loaded by the browser through range-request, which are difficult to handle. We may either not cache videos entirely or write a library that handles range requests (sample).

  3. html

    Caching views like / and optionally views under /settings/ (letting user change settings offline or not). If the user logs out, purge all the cache (or not let user logout when in offline mode).

  4. Background sync

    May need to implement this to update user actions (fav, boost, delete, new toot, replies, settings change) when network is available (or not let user do any actions offline).

  5. User data

    <!-- Other peoples opinions go here because I don't have a good strategy for caching data from /api/v1/* links. Images can be cached just normally anyway. -->

All this blathering is about describing all the things (some ideas) that we need to do while using a ServiceWorker. I sincerely think that we should be writing our own SW anyway (even if we wanted to implement push notifications in the future).

It's not totally perfect or complete. Any other suggestions or corrections are highly accepted. 😊

Sample SW code
// sw.js
'use strict';

// May store these in indexedDB
const NAME = require('../package.json').name;
const VERSION = require('../package.json').version;

importScripts('files-to-cache.js'); // To be generated by webpack

self.oninstall = evt => {
  evt.waitUntil(
    caches
      .open(`${NAME}-v${VERSION}`)
      .then(cache => {
        // Add all static assests from the list generated by webpack 
        // which includes 404 error page
      })
  );

  self.skipWaiting();
};

// When new SW is installed
self.onactivate = _ => {
  const currentCacheName = `${NAME}-v${VERSION}`;
  caches.keys().then(cacheNames => {
    return Promise.all(
      cacheNames.map(cacheName => {
        if (cacheName.indexOf(NAME) === -1) {
          return null;
        }

        if (cacheName !== currentCacheName) {
          return caches.delete(cacheName);
        }

        return null;
      })
    );
  });

  self.clients.claim();
};

self.onfetch = evt => {
  const FETCH_TIMEOUT = 10000;
  const request = evt.request;
  evt.respondWith(
    caches.match(request)
      .then(response => {

        // Check if the request is to the API or to assests
        // and act accordingly
        // API requests return json data from cache that are not too old
        // And any other requests to the assests are mostly cached
        // or else thay are fetched and served (but not cached)

        // If the request is for an emoji, the code from above
        // is suitable for managing the requests

        if (response) {
          return response;
        }

        return Promise.race([
          fetch(evt.request),
          new Promise(resolve => {
            setTimeout(resolve, FETCH_TIMEOUT);
          })
        ]).then(response => {
          if (response) {
            return response;
          }

          return caches.match('/404/');
        });
    );
};

self.onmessage = event => {
  // If user logs out, purge cache
};
if ('serviceWorker' in navigator) {
  navigator.serviceWorker.register('/sw.js', {
    scope: './'
  }).then(registration => {
    // SW updates
    registration.onupdatefound = _ => {
      let newWorker = registration.installing;

      newWorker.addEventListener('statechange', _ => {
        if (newWorker.state === 'activated'  && navigator.serviceWorker.controller) {
          // View a toast saying "Update available"
        }
      });
    };
  });
}

@nolanlawson
Copy link
Contributor Author

nolanlawson commented May 13, 2017

@ramlmn Thanks for the feedback! I still think we should go with offline-plugin vs a fully custom service worker, because offline-plugin does still allow for some custom code (via the rewrites) function) on top of automatically generating the "right" code for the Webpack asset graph. There is also the entry option that allows us to inject arbitrary code into the SW. So we can use offline-plugin but then still add whatever capabilities we want.

Right now I am trying to work solely on solving the HTML problem, which is tricky because of the CSRF token. I could also submit what I already have, but it will only do offline caching of JS/CSS/etc resources and wouldn't fully work offline.

@nolanlawson
Copy link
Contributor Author

nolanlawson commented May 13, 2017

OK, I've made more progress this morning. I've solved the HTML issue (by configuring offline-plugin to use credentials) and the emoji issue (by configuring them as optional assets, i.e. assets that are cached on-demand). My latest branch is here.

To respond to @ramlmn:

  1. emoji See above; caching on-demand is fine for now. In my ideal world though I agree it'd be nice to just ship a JSON file in the service worker and respond using that.
  2. videos These probably shouldn't be cached; they're too large.
  3. HTML: we can cache any HTML file we want using offline-plugin's rewrites option. I still need to solve the "clear cache on logout"; that's my next task but it can be accomplished with offline-plugin's events API.
  4. Background sync: I would love this, but it's a more ambitious project. From working on PouchDB I can tell you that offline sync from client to server is a hugely difficult problem, and the naïve solutions almost always lead to more problems than they solve (e.g. conflicts, duplicates, lost data, etc. – this is a distributed computing problem, so you need some kind of conflict resolution strategy).
  5. User data: for now I'm just skipping caching of API data.

So basically the "offline mode" as I envision it is merely the fact that static assets and HTML are cached offline, so the user can read stale toots while offline, but nothing else works without a network connection. This is roughly how the Twitter native app seems to work, so it seems reasonable for a first pass.

Hopefully I'll be able to submit a PR by the end of this weekend. 😃 Next steps:

  • Ensure SW cache is cleared on logout or when the session token becomes invalid
  • Nail down SW update strategy, e.g. figure out if we want skipWaiting() or not and what to do when a SW becomes redundant
  • Ensure the AppCache fallback works (e.g. for iOS Safari).
  • Try to figure out sourcemaps for the sw.js file (currently not working).

@nolanlawson
Copy link
Contributor Author

I've filed an issue on offline-plugin about the lack of sourcemaps: NekR/offline-plugin#254

@NekR
Copy link

NekR commented May 15, 2017

HTML: we can cache any HTML file we want using offline-plugin's rewrites option.

You may just straight add externals: ['/whatever.html'] or other urls. rewrites is kind of made for other things, i.e. making generated index.html be presented as / in SW or to rewrite some assets to CDN (to be fetched from CDN).

Let me know if you need any help with SW/offline-plugin or review something :-)

@rixx
Copy link

rixx commented Jun 4, 2017

Implementing ServiceWorker would be a great feature, a wonderful asset to usability, and would give mastodon a noticable edge. I love that work is being done on this! @nolanlawson – any info on your current progress?

@nolanlawson
Copy link
Contributor Author

@rixx See nolanlawson/blob-util#29, there are only a few tasks remaining. :)

@Gargron
Copy link
Member

Gargron commented Apr 8, 2018

We do have a service worker now...

@Gargron Gargron closed this as completed Apr 8, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants