IndexedDb Persister

[jonathanstanley]

Now that we have better persister in react-query v4 alpha 🚀, I was also going to see about sharing my IndexedDb persister. It's super easy:

import { get, set, del } from "idb-keyval";
import { PersistedClient, Persister } from "react-query/persistQueryClient";

/**
 * Creates an Indexed DB persister
 * @see https://react-query.tanstack.com/plugins/persistQueryClient#building-a-persistor
 * @see https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API
 */
export function createIDBPersister(idbValidKey: IDBValidKey = "reactQuery") {
  return {
    persistClient: async (client: PersistedClient) => {
      set(idbValidKey, client);
    },
    restoreClient: async () => {
      return await get<PersistedClient>(idbValidKey);
    },
    removeClient: async () => {
      await del(idbValidKey);
    },
  } as Persister;
}

Benefits

1. In my use, I haven't seen it even come close to needing throttling. Indexed-DB is high performance and async.
2. There is no need to serialize/deserialize since Indexed DB supports the structured clone algorithm. dates, recursive objects, and other native types are all handled natively. see also Cache Persistence in IndexedDB #1638
3. no 5MB limit
4. It possibly can replace WebStoragePersister (other than the dependency mentioned below, I can't think of a reason to use WebStoragePersister over this) (nevermind on this one; see discussion)

Shall I submit as a PR?

I can also add documentation and examples for use.

The only issue is dependencies. This requires idb-keyval and react-query currently only has 3 dependencies. I checked into implementing without a dependency, but idb-keyval handles a couple browser quirks.

[cliffordfajardo]

Nice! Using the vanilla IndexDB API's are not a great experience so glad that IDB keyval exists; it definitely makes the implementation easier.

If idb-keyval is used for the IndexDB Persister, it should be included as a peerDependency like how reactQuery lists react as a [peerDependency here] ?(https://github.com/tannerlinsley/react-query/blob/master/package.json#L72)

It possibly can replace WebStoragePersister (other than the dependency mentioned below, I can't think of a reason to use WebStoragePersister over this)

I think having the option to use LocalStorage still might be good to keep for wider browser support? 🤔

no 5MB limit

This is no limit is nice. I've hit the limit 5MB limit very easily with localStorage, due to 3rd party libraries stuffing data in the localStorage for their own requirements, leaving less space for my app data
For example:

• AWS Amplify stores it's authentication data in LocalStorage

Some follow up questions are:

Are there any downsides/alternatives to including idb-keyval?

• I really couldn't think of any off the top of my head

[jonathanstanley]

Thanks for your thoughts! I generally find libraries with few or no dependencies to be much better (part of why I chose react-query). Although idb-keyval is great, it isn't really to do with react-query... just this plugin. So, for me, this is itself a downside to including it. A peer-dependency wouldn't inflate the dependency count. But a peer-dependency could actually be more annoying for users since they would be prompted to install idb-keyval in their package even if they don't use this IndexedDbPersister. In my mind, it doesn't seem right to set this as a peer dependency.

Your point about AWS is well taken. I could see use cases that need to work with other systems that makes it preferable to provide persisters for each case. I'll edit my OP.

re: browser support
local storage does have a 1.8% advantage:
https://caniuse.com/indexeddb2 (93.4% of users have it)
https://caniuse.com/mdn-api_window_localstorage (95.2% of users have it)

[TkDodo]

isn't this just an async storage persister with a bit of renaming?

import { get as getItem, set as setItem, del as removeItem } from "idb-keyval";

createAyncStoragePersister({
  storage: { getItem, setItem, removeItem }
})

you would also profit from throttling. While we've made the AsyncStorage interface to work best with react-native's AsyncStorage, the interface is still pretty generic so that it should also work with things like idb-keyval or localForage.

this is the interface that an AsyncStorage needs to fulfil:

https://github.com/tannerlinsley/react-query/blob/9a20452f7139de90a5d94021cfe57dbd87cc9256/src/createAsyncStoragePersister/index.ts#L3-L7

[TkDodo]

@jonathanstanley what do you think about the generic proposal?

[jonathanstanley]

Those interfaces look good and I think would help create persisters that don't serialize to strings. 👍

However, thinking about the bigger picture, I'm not sure it is appropriate to use for Indexed DB persister. Since Indexed DB doesn't serialize and is async, I can't think of any purpose of a throttle. Rather, even a throttle 0 may introduce some slight overhead and slow it down. Without a throttle or serialization, the createAsyncStoragePersister doesn't add anything for Indexed DB persister. So, it seems odd to use it. And using it as an example for createAsyncStoragePersister could be confusing for users.

FWIW, when I first looked at persisters i was discouraged from building a persister and thought the two creator functions were the only way. Perhaps the wording "Officially Supported Persisters" is a little confusing because they aren't actually persisters... they are builders for persisters.

In any case, I actually think Indexed DB persister would be a great candidate for a "Build your own persister example" .

1. I'd be happy to submit a PR for this as an example in the alpha docs.
2. I could also propose some changes to help clear up the wording a bit.

[TkDodo]

In any case, I actually think Indexed DB persister would be a great candidate for a "Build your own persister example" .

that would be awesome, thank you. Happy to accept contributions for that, and also 🟢 for clearing up any wordings you deem confusing 👍

Rather, even a throttle 0 may introduce some slight overhead and slow it down.

My suggestion would be to not call throttleAsync at all in the AsyncPersister if a value of 0 is passed. Or maybe a value of false to turn it off, widening the type to number | false. Thoughts?

[jonathanstanley]

A throttle 0 skipping the throttle code may help others? Or maybe it would just add complexity? I'm really not sure what the use case would be, so I don't know that I have helpful feedback on that.

I'll send a PR soon.

[jonathanstanley]

submitted PR: #3356

[jonathanstanley]

merged in PR: #3356

[TkDodo]

@jonathanstanley do we still need to adapt the AsyncPersister to accept generics as proposed here?

[jonathanstanley]

It could be helpful for others developing persisters, however it isn't needed for my contributions.

[onekiloparsec]

@TkDodo does your proposition still holds? Like @CavalcanteLeo below, I am very much interested having such persister for query-vue. I am currently trying to write one using the dispatched information here and there for IndexedDB.

The little factory I've written (see below) uses the experimental function experimental_createPersister. It seems to work, but indeed goes through the (standard JSON) serialization. Should I open an issue? I am not yet familiar enough with the codebase for opening a PR.

[schachfeld]

Thank you for the insight! This is a way around the 5MB limit for localStorage.
I have one problem though: IndexedDB is really slow. I am writing about 300 MB of data which of course is pretty massive. The problem however is that the AsyncPersister saves the Cache on every url change (react router). I tried to use throttle already which had a positive impact on render time but when I reloaded the page some of the already cached requests where being fetched again (I have staleTime and cacheTime on Infinity right now).
Is there any solution to this? I really like this library but I need a way to cache this massive amount of data...

[nhd2106]

@schachfeld Hi, did you resolve your problem, sir?

[greatstudentkaze]

@nhd2106 hi! did you manage to solve this problem?

[nhd2106]

Hi, just update your all tanstack-query dependencies to the same version.

[greatstudentkaze]

@nhd2106 all deps already have the same 4.20.4 version.. or you mean to v5?

[dberardo-com]

is this still an issue ? i could not understand well the problem here so i dont know whether that affects me, but would like to be on the safe side

[milahu]

feature request:
all queries should wait for persister.restoreClient

in my case, restoreClient takes 0.5 seconds,
but my interface is faster, so queries are fetched, not loaded from cache

related: XY problem issue in #4368

this would help: a global setting queryClient.enabled so we can disable all queries
or queryClient.paused as a "default undefined" option

draft ...

node_modules/@tanstack/query-core/build/lib/queryObserver.esm.js

  onSubscribe() {
    if (this.listeners.length === 1) {
      this.currentQuery.addObserver(this);

      if (shouldFetchOnMount(this.currentQuery, this.options)) {

        //this.executeFetch();

        const queryClient = this.client
        if (!queryClient.paused) {
          this.executeFetch();
        }
        else {
          // queryClient is paused -> fetch later
          if (!queryClient.afterPersistQueryClientRestore) {
            queryClient.afterPersistQueryClientRestore = []
          }
          queryClient.afterPersistQueryClientRestore.push(() => {
            if (shouldFetchOnMount(this.currentQuery, this.options)) {
              this.executeFetch();
            }
          })
        }

      }

      this.updateTimers();
    }
  }

node_modules/@tanstack/query-persist-client-core/build/lib/persist.esm.js

async function persistQueryClientRestore({
  queryClient,
  persister,
  maxAge = 1000 * 60 * 60 * 24,
  buster = '',
  hydrateOptions
}) {
  queryClient.paused = true

  try {
    // ...
  } catch (err) {
    // ...
  }

  queryClient.paused = false
  if (queryClient.afterPersistQueryClientRestore) {
    for (const fn of queryClient.afterPersistQueryClientRestore) {
      fn()
    }
  }
}

[TkDodo]

feature request:
all queries should wait for persister.restoreClient

again, that is what the PersistQueryClientProvider is doing. It's documented:

https://tanstack.com/query/v4/docs/plugins/persistQueryClient#usage-with-react