Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FR: Firestore offline persistence in web workers

Closed
lamplightdev opened this issue Jul 5, 2018 · 61 comments
Closed

FR: Firestore offline persistence in web workers #983

lamplightdev opened this issue Jul 5, 2018 · 61 comments

Comments

@lamplightdev
Copy link

lamplightdev commented Jul 5, 2018

  • Operating System version: n/a
  • Browser version: n/a
  • Firebase SDK version: n/a
  • Firebase Product: firestore

Are there any plans to make offline persistence available in web workers? Inspired by @davideast 's article I dropped 95% of my Firebase bundle size using this one weird trick to use firebase in a worker, I've noticed offline persistence isn't available in that context. Browsing the source, I suspect it's the reliance on (and absence in workers of) localStorage that's the issue - so I was wondering if there's any alternative to using synchronous storage in this case? Thanks.

@joshdifabio
Copy link

joshdifabio commented Jan 14, 2019

Any news on this guys? The lack of offline persistence in service workers & web workers is a bit of a killer for PWA.

@MarvinMiles
Copy link

MarvinMiles commented Jan 16, 2019

Same here

@mikelehen
Copy link
Contributor

mikelehen commented Jan 16, 2019

Thanks for the feedback. Could folks impacted by this tell us a little bit about their use case / architecture (what kind of worker they're trying to use Firestore from and for what purpose exactly)?

@joshdifabio
Copy link

joshdifabio commented Jan 17, 2019

@mikelehen thanks for responding.

In my case, the data I pull from Firestore sometimes needs quite a lot of processing before I present it to users. As a result, I do most of my Firestore interaction inside a Web Worker instead of on the main thread in order to reduce UI jank. I could move this work to the Service Worker instead but I assume that this Firestore SDK limitation applies to Service Workers too since they also do not support the localStorage API.

Unfortunately, this approach means that I lose offline persistence in my app, even though workers support IndexedDB.

As the Service Worker API in particular becomes more popular I imagine that this will become a problem for more people.

@mikelehen
Copy link
Contributor

mikelehen commented Jan 17, 2019

Thanks for your use case. One option would of course be to pull the data from Firestore on the main event loop and then just do the processing on the webworker, but if it's a lot of data, perhaps there are performance wins for moving the Firestore interaction off the main event loop as well. So it's a reasonable request.

Service workers are a bit more complicated. They don't support XHR, which Firestore (and possibly other parts of Firebase) depends on. And service worker lifetime is generally short-lived. Our supposition so far is that it doesn't make that much sense to use Firestore from a service worker. But if folks have use cases, please do share.

@joshdifabio
Copy link

joshdifabio commented Jan 18, 2019

Thanks for your use case. One option would of course be to pull the data from Firestore on the main event loop and then just do the processing on the webworker

Thanks. I did consider this, but it wasn't practical for a few reasons. For example:

  1. This approach would significantly increase the amount of boilerplate code I'd need to write in order to get data in and out of the worker thread, as the worker thread often decides which queries to send to Firestore based on the results of other queries
  2. When processing Firestore data I often need access to Firestore's snapshot objects in order to process the data more efficiently (e.g. using DocumentSnapshot.get()), but class instances cannot be passed between the main thread and workers.

@joshdifabio
Copy link

joshdifabio commented Jan 22, 2019

Something I forgot to add previously: every now and then I see the main thread blocked for 100ms+ by Firestore XHR-related scripts shortly after cancelling snapshot listeners. (Note that this timing is taken on a powerful MacBook Pro and I'd expect the main thread to be blocked for much longer on mobile.) This is another reason I offload most of my Firestore interaction to a worker thread.

@mikelehen
Copy link
Contributor

mikelehen commented Jan 22, 2019

@joshdifabio That sounds pretty strange. Offhand I can't think of anything CPU-intensive we do when unsubscribing (and XHRs themselves are async). How exactly are you measuring the main thread blocking? Thanks!

@joshdifabio
Copy link

joshdifabio commented Jan 23, 2019

@mikelehen interesting. I observed jank and then used the performance profiler in Chrome. I haven't looked at it in detail yet, I only really saw that Firestore XHR was at the top -- perhaps it was GC or something in my own code being triggered. I'll have another look soonish and get back to you.

@dexster
Copy link

dexster commented Feb 19, 2019

My use case is for offline updates to be saved to Firestore without having to open the relevant web site when online ie. like web push where only the browser is required to be open. I assume due to localstorage API unavailability, this will not be possible

@mikelehen
Copy link
Contributor

mikelehen commented Feb 21, 2019

@dexster I don't think any kind of worker will be able to run when the relevant web site isn't open. For that you'd need a browser extension or something.

@joshdifabio
Copy link

joshdifabio commented Feb 22, 2019

I've done some more performance profiling over the past couple of weeks, and this is what I've observed regarding Firestore performance.

Whenever I read data from Firestore, I can see about about 50-150ms spent in Firestore code when the XHR response comes back. Everything happening in that time period appears to be Firebase code. Firestore processes the response synchronously on the main thread, and obviously even the lower bound of this figure (50ms) is way beyond what can realistically be accommodated on the main thread without introducing UI jank. These times were measured on a MacBook Pro; the times are higher on my Pixel 2.

I previously said that this work was happening whenever I cancelled Firestore snapshot listeners, but this was a mistake, it's when the data comes back. I see this all the time, it's not an edge case -- 50ms of work in the main thread is the lower bound when listening to a single record.

Unless I'm reading the figures wrong in Chrome's profiler and including my own application code in the figures, which I don't think I am, I suspect that these figures will be easily reproducible on any web app which uses the Firestore JS SDK.

@mikelehen
Copy link
Contributor

mikelehen commented Mar 1, 2019

@joshdifabio Thanks for the added details. I was able to reproduce ~50ms with a big document in Chrome on my Mac. The time was basically spent parsing the result from the backend.

Note that I believe I made a recent size optimization (4983420) that will also improve performance a little (it removed some logging calls that should have an impact here). That should go out in our next release.

All that said, 50ms for a large document doesn't seem unreasonable to me. At 60fps, you might skip a couple frames, but for a typical web app, this should be imperceivable. A live-action game might be a different story. In any case, if you're sensitive to these sorts of hiccups, moving the processing to a separate worker seems like a reasonable approach. Sorry that persistence doesn't work at the moment. We'll give this some thought in the future.

@SebasG22
Copy link

SebasG22 commented Jun 6, 2019

@mikelehen
This is really exicting based the powerful of web workers. I'm facing this error because said that offline persistence is not present on web workers yet on firebasejs 6.1.0. It's strange because IndexDB is available on web workers.

image

@mikelehen
Copy link
Contributor

mikelehen commented Jun 6, 2019

@SebasG22 Oh, interesting... I guess you get that error because we are checking for existence of the window global (

if (typeof window === 'undefined' || window.indexedDB == null) {
) which doesn't exist in a web worker...

We could check directly for the indexedDB global but then you would just get a different error: 'IndexedDB persistence is only available on platforms that support LocalStorage.'

We may be able to make our dependency on LocalStorage optional, in which case persistence might work to some degree in a web worker... though there may be rough edges. I'll try to take a look at this.

@mikelehen mikelehen self-assigned this Jun 6, 2019
@SebasG22
Copy link

SebasG22 commented Jun 6, 2019

@mikelehen , It will be really helpful if you can take at look at this.
You right, from my perspective there is not need to access window or depends or localStorage. It will be better only ask for IndexedDB.
Thanks.
I'm willing to help in whatever you need 👍

@tsavo-vdb-knott
Copy link

tsavo-vdb-knott commented Sep 11, 2019

Hello All!

Just wondering if there is any update on this @mikelehen ?

Every other Firestore feature is totally fine in a worker except the persistence - really would like to avoid the cost on the main thread of serializing, copying, and sending snapshots into the worker for further processing. Our current architecture allows for our main thread to be entirely reactive to updates from off thread data changes.

Would appreciate any further information on this.

Cheers,
-Tsavo

@mikelehen
Copy link
Contributor

mikelehen commented Sep 11, 2019

@T-Knott-Mesh Sorry! I have not spent time on this. @thebrianchen mentioned he may be able to take a look in the coming weeks though. So stay tuned.

@mikelehen mikelehen assigned thebrianchen and unassigned mikelehen Sep 11, 2019
@tsavo-vdb-knott
Copy link

tsavo-vdb-knott commented Sep 11, 2019

No problem ! Thank you for the response :) we will create a work around for now but we are really looking forward to potential capabilities in a worker.

Appreciate you all ! @mikelehen @thebrianchen

@thebrianchen
Copy link

thebrianchen commented Oct 1, 2019

I've been looking into enabling persistence for web workers by removing the LocalStorage requirement, and it could work with a few caveats.

Firestore implements a leasing mechanism in enabling persistence to ensure that multiple clients do not concurrently write to the underlying IndexedDB that powers it. This leasing mechanism relies on LocalStorage, since it allows Firestore to reliably record lease information even when browser tabs are closing. IndexedDB on its own can't reliably write as a tab is closing. When a client enables persistence, it must first obtain the primary lease, which prevents other clients from becoming the primary. Then, when the client closes, it marks the lease as available in LocalStorage, which allows the next client to become the primary. Consequently, without LocalStorage, once a client holding the lease terminates, no other client can takeover the lease until it expires (after 5 seconds). This creates two limitations:

  • Refreshing the page on a client with persistence enabled causes the client to lose persistence.
  • Multi-tab functionality through synchronizeTabs is not supported.

As a potential workaround, we're considering a firestore.enablePersistence({experimentalForce: true}) option, which will enable persistence for the client even if another client is already running with persistence enabled. We'll also change Firestore to allow operation even if LocalStorage is unavailable. With these changes you would have a few options for implementing web workers with persistence:

  1. If you do not require multiple Firestore clients, you can enable persistence with the experimentalForce flag every time to forcefully transfer the lease to the newest client. However, Firestore clients currently assume that once they are given the lease, it cannot be taken away. This means that if a user opens your app in a new tab, the new tab will have persistence enabled, but Firestore will break in all other tabs with an error that the AsyncQueue has failed. If this use case is popular, we can look into adding additional logic to make it more developer-friendly.

  2. To use web workers, and have page refresh work, you will have to manage multiple tabs yourself, either via LocalStorage in the main browser, or via a shared worker. You must provide additional logic to check when it is appropriate for a tab to takeover the primary lease via experimentalForce.

  3. If you do not want to manage tabs, you can run a single instance of Firestore in a Shared Worker. Since Shared Worker persists across browser tabs, you can enable persistence with experimentalForce: true each time.

Please try the experimental version by using npm install with the downloaded tarball (download UPDATE (2019/12/10): Uploaded new version that uses 1.8.1 instead of 1.7.0).

We are still evaluating if this functionality is useful and exactly how we expect folks to use it. So please provide us feedback based on whether it works for you and how you end up using it (in a web worker, shared worker, etc.) Thanks!

SAMPLE APP that implements Firestore with persistence in web workers

@tsavo-vdb-knott
Copy link

tsavo-vdb-knott commented Oct 1, 2019

@thebrianchen let's goooooo !! Woo! This is great, already do tab management with Service Worker and notification stuff so building out additional handling is no prob. Will be testing this out tomorrow 🎉

@laurentpayot
Copy link

laurentpayot commented Oct 1, 2019

Great news! Too bad Safari dropped support for shared workers 😕 Will try with normal workers anyway.

@laurentpayot
Copy link

laurentpayot commented Oct 1, 2019

Also shared workers don't seem to be implemented on Chrome for Android (36% of global users).

@wilhuff
Copy link
Contributor

wilhuff commented Oct 1, 2019

@laurentpayot We understand that shared workers are of quite limited utility. They happen to work quite well with these changes though, so we've called that out. Anyone writing for the general web will have to avoid them and use one of the other options.

@SebasG22
Copy link

SebasG22 commented Oct 2, 2019

Hey @thebrianchen,
Thanks for your contribution. 🚀
I will try it on the weekend on a demo repo

@thebrianchen
Copy link

thebrianchen commented Nov 24, 2019

@laurentpayot That sounds like a viable option for your use case! However, Firestore doesn't do this because (1) it would add 50ms to the initialization time and (2) we aren't sure what timeout duration would be reliable given the potential throttling of background tabs.

In your case though, you could continue implementing this in your main application in combination with the current logic in the sample app. The onunload cleanup can track the active tab with persistence, and if and only if persistence is marked as unavailable, you can run your tab notification timeout trick. That way, if you only have one tab open, you won't have to wait for the timeout. Also, I'm not sure if 50ms is sufficient to complete all full cycle of read/writes. If you do end up using this approach though, please let us know how it works for you!

@laurentpayot
Copy link

laurentpayot commented Nov 25, 2019

@thebrianchen

The onunload cleanup can track the active tab with persistence, and if and only if persistence is marked as unavailable, you can run your tab notification timeout trick.

Great idea, I will let you know how it works!

@laurentpayot
Copy link

laurentpayot commented Nov 27, 2019

@thebrianchen I ended-up using the following code where tab detection is used only if persistence is taken. Tested after a kill -9 of chrome process to simulate a crash, works great too. Thanks for the idea!

const TAB_DETECTION_TIMEOUT = 100

let isOtherTab = new Promise(resolve => {
  localStorage.tabPing = Date.now()
  window.addEventListener('storage', e => {
    if (e.key == "tabPing") localStorage.tabPong = Date.now()
    if (e.key == "tabPong") resolve(true)
  })
  setTimeout(() => resolve(false), TAB_DETECTION_TIMEOUT)
})

// tab detection too in case of previous crash with unreleased persistence
let usingPersistence = !(!!localStorage.persistenceTaken && await isOtherTab)
if (usingPersistence) {
  localStorage.persistenceTaken = "1"
  dedicatedWorker.postMessage('enablePersistence')
  // releasing persistence when closing tab
  window.addEventListener("beforeunload", () => localStorage.persistenceTaken = "")
}

@laurentpayot
Copy link

laurentpayot commented Dec 10, 2019

@thebrianchen sorry to bother you again but it looks like the firestore experimental build is not compatible with firebase 7.5.1 and higher (see #2420). Would you mind updating the build again?

@thebrianchen
Copy link

thebrianchen commented Dec 10, 2019

@laurentpayot Updated. If you don't want to scroll up here is the link for 1.8.1.

@laurentpayot
Copy link

laurentpayot commented Dec 11, 2019

Thanks @thebrianchen! It did fix #2420 👍

@scottfr
Copy link

scottfr commented Dec 18, 2019

We're very interested in this for use in our Chrome Extension as extensions are moving away from a persistent background page model to service workers.

The constraints outlined in #983 (comment) are completely acceptable for this use case. There should only ever be one active service worker from the extension's origin so conflicts shouldn't happen.

As an aside, have you considered using web locks (https://developer.mozilla.org/en-US/docs/Web/API/Web_Locks_API) as an option where thy are available? It seems like it might solve this issue on Chrome.

@thebrianchen
Copy link

thebrianchen commented Jan 8, 2020

@scottfr That's exciting news! In your extension, does passing in experimentalForce: true allow persistence to work for you without any additional code, or do you have to add logic to release the persistence lock in edge cases?

Thanks for bringing up web locks -- we will definitely consider adding support once we determine the viability of experimentalForce.

@raphaelbs
Copy link

raphaelbs commented Apr 5, 2020

Hi everyone. +1 interested in this feature.

Could folks impacted by this tell us a little bit about their use case / architecture (what kind of worker they're trying to use Firestore from and for what purpose exactly)?

In our case, we have an application that uses only Firestore and Auth, just to read and write user data, nothing fancy.
Even though, I'd just migrated the entirety of Firebase to a worker and the application has shown a significant increase in performance for its entry pages:

  • avg -2.8s for First CPU idle
  • avg -1.6s for Speed Index
  • avg -1.1s for Minimize main-thread work
  • avg -0.7s for Reduce JavaScript execution time

I got those numbers running synthetic tests with Lighthouse server for the branch I'm currently working on.

Is there a way to ship the experimentalForce flag into the SDK so we can validate the data in production?
We also use the offline persistence (the app is a PWA) and would be ideal not to lose that.

@thebrianchen
Copy link

thebrianchen commented Apr 9, 2020

I will look into adding this as an experimental feature in the coming weeks. Thanks for sharing the feedback!

@laurentpayot
Copy link

laurentpayot commented Apr 16, 2020

+1 for this as an experimental feature of the official Firestore releases. It's getting harder and harder to make custom builds work along with always changing Firebase products (modules location, types location, testing, etc.)

@tsavo-vdb-knott
Copy link

tsavo-vdb-knott commented Apr 18, 2020

Case for this:

Worker Driven Design ~ 3:41 ~

Screen Shot 2020-04-18 at 5 16 39 AM

Appreciate all the work Firebase is doing and I - as a development platform author of Runtime.dev am making it a first class citizen - Web worker support is paramount to data driven apps :)

Cheers,
-Tsavo

@marianopaulin02
Copy link

marianopaulin02 commented Apr 19, 2020

Im also using the trick of moving firebase-related logic to a service worker in order to have just 1 instance and 1 single connection to Firebase Realtime DB even when multiple app tabs are open, and Im keeping the worker alive via sending 10-seconds periodical empty messages. Now I was trying to implement Firestore and found a limitation in the service worker because this one doestn't support XMLHttpRequest (used for Firestore as mentioned before in the thread). There is a way to implement Fetch instead, or maybe a wrapper?

image
I got this error running a simple db.collection(..).doc(..).get() operation
Now i'm using Firebase 7.14.1 via importScripts

@laurentpayot
Copy link

laurentpayot commented Jun 20, 2020

So far the experimentalForceOwningTab option is working perfectly, thanks @thebrianchen for all the work! 👍

@thebrianchen
Copy link

thebrianchen commented Aug 12, 2020

Hey everyone! This experimental feature has been out for a few months now, and we're hoping to collect some usage feedback to see how this feature has been working for you to see if/how we should support this in the long term. In particular, how are you using experimentalForceOwningTab? Which workers are you using this with, and how much additional code are you writing to make Firestore requests? Thanks!

@laurentpayot
Copy link

laurentpayot commented Aug 14, 2020

@thebrianchen It's been working great so far. I use dedicated workers, with this code to detect multi tab usage and disable persistence for new tabs.

@jwhiting
Copy link

jwhiting commented Feb 9, 2021

@thebrianchen thank you for the feature. FYI I am using experimentalForceOwningTab for the use case of a personal information management Chrome extension. The architecture uses a web worker spawned by the background page, and popup and full-page tab UIs. The UI pages use message passing to the background context, which then delegates (also via messaging) all storage and processing tasks to the web worker. The main reason to use the web worker is to prevent UI jank, as the storage and processing tasks can be fairly computationally expensive, and are fired often. Consider a rich text editor with a large document: serializing and storing this document, even when done every N keystrokes via a debounced/throttled callback, can still interrupt the natural responsiveness of keystrokes. We also use local javascript to compute a fulltext index on the documents to enable offline fulltext search, and updating this index is expensive (another reason for a web worker, though not specific to Firestore persistence, but it is certainly nice to be able to do all the heavy lifting in one context. In the future we might investigate persisting the fulltext index to Firestore as well.)

@stroobandt
Copy link

stroobandt commented Apr 14, 2021

Would it not be better and fairer to offer persistence for any web worker on Chrome for Android, independently of whether firebase is in use or not?

My use case is recurring PWA notifications, as detailed in this open issue with the Google Chrome for Android PWA team. The lack of web worker persistence on Chrome for Android is currently the only thing standing in the way of an implementation.

It really looks like corporate Google engineers of different Google divisions are waging similar battles against a common problem.

Moreover, according to the spec, I would be inclined to say: we are dealing here with a Chrome for Android bug. The reason being: A web worker with outstanding timers, database transactions, or network connections should be treated as a protected worker.

@ReaperTech
Copy link

ReaperTech commented Jul 25, 2021

Im also using the trick of moving firebase-related logic to a service worker in order to have just 1 instance and 1 single connection to Firebase Realtime DB even when multiple app tabs are open, and Im keeping the worker alive via sending 10-seconds periodical empty messages. Now I was trying to implement Firestore and found a limitation in the service worker because this one doestn't support XMLHttpRequest (used for Firestore as mentioned before in the thread). There is a way to implement Fetch instead, or maybe a wrapper?

image
I got this error running a simple db.collection(..).doc(..).get() operation
Now i'm using Firebase 7.14.1 via importScripts

@marianopaulin02 does this page help? https://stackoverflow.com/questions/63261382/writing-data-in-service-worker-to-cloud-firestore

@thebrianchen
Copy link

thebrianchen commented Jan 20, 2022

Closing this issue as the feature has been launched a while ago. Feel free to open a new issue if there are any other questions or issues!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests