Cache Persistence in IndexedDB

[usman087]

React query have experiemental cache persistence using local storage. However in our application we have large data cached and local storage has max storage limit of 5mb. In order to store large amount of data , i have modified persist-localstorage-experimental to use IndexedDB instead of localstorage. I have used idb-keyval library to intreact with IndexedDB, This is very tiny library just 300 bytes.
Here is modified Code to persist in IndexedDB instead of localstorage.

• Please let me know if you have any feedback .
• Are there any known issues of using Indexed DB as persistence layer ?
  On cache busting we also added logic to check the application version. This covers the scenario if our models have changed with a new release and cached data can no longer map to the new model. App_Version is just a global variable on windows.

persistWithIndexedDB.ts

import { QueryClient } from "react-query";
import { dehydrate, hydrate } from "react-query/hydration";
import { get, set, del } from "idb-keyval";

interface IndexedDBCache {
  timestamp: number;
  buster: string;
  cacheState: any;
}

interface Options {
  /** The key to use when storing the cache to IndexDB */
  IndexedDBKey?: string;
  /** To avoid indexedDB spamming,
   * pass a time in ms to throttle saving the cache to disk */
  throttleTime?: number;
  /** The max-allowed age of the cache.
   * If a persisted cache is found that is older than this
   * time, it will be discarded */
  maxAge?: number;
  /** A unique string that can be used to forcefully
   * invalidate existing caches if they do not share the same buster string */
  buster?: string;
}

export async function persistWithIndexedDB(
  queryClient: QueryClient,
  {
    IndexedDBKey: indexedDBKey = `REACT_QUERY_OFFLINE_CACHE`,
    throttleTime = 1000,
    maxAge = 1000 * 60 * 60 * 24 * 7, // 7 days
    buster = window?.App_Version || ""
  }: Options = {}
) {
  if (typeof window !== "undefined") {
    // Subscribe to changes
    const saveCache = throttle(() => {
      const storageCache: IndexedDBCache = {
        buster,
        timestamp: Date.now(),
        cacheState: dehydrate(queryClient)
      };
      set(indexedDBKey, JSON.stringify(storageCache)); // set in Indexed DB
    }, throttleTime);

    queryClient.getQueryCache().subscribe(saveCache);

    // Attempt restore
    const cacheStorage = await get(indexedDBKey); // get from Indexed DB

    if (!cacheStorage) {
      return;
    }

    const cache: IndexedDBCache = JSON.parse(cacheStorage);

    if (cache.timestamp) {
      const expired = Date.now() - cache.timestamp > maxAge;
      const busted = cache.buster !== buster;
      if (expired || busted) {
        del(indexedDBKey); // Delete from Indexed DB
      } else {
        hydrate(queryClient, cache.cacheState);
      }
    } else {
      del(indexedDBKey);
    }
  }
}

function throttle(func: (...args: any[]) => any, wait = 100) {
  let timer: number | null = null;

  return function (...args: any[]) {
    if (timer === null) {
      timer = setTimeout(() => {
        func(...args);
        timer = null;
      }, wait);
    }
  };
}

Now you can use it with your QueryClient Initialization

queryClient.ts

import { QueryCache, QueryClient, QueryClientProvider } from "react-query";
import { persistWithIndexedDB } from "./persistWithIndexedDB";
const queryCache = new QueryCache();
const queryClient = new QueryClient({
  queryCache,
  defaultOptions: {
    // options here
  }
});
persistWithIndexedDB(queryClient);
export { queryClient, queryCache, QueryClientProvider };

[JCBsystem]

Very interesting will give you feedback when i have tested it
Good job!

[cliffordfajardo]

I also hit the 5MB mark pretty quickly when I first used it. One option I'm exploring is compressing the JSON data when I store it for persistence and then deserialize it for example like this:

• https://stackoverflow.com/a/20774589/2971795

FYI @usman087

• Tanner updated the experimental localStorage persistence API
• Docs: https://react-query.tanstack.com/plugins/createLocalStoragePersistor#_top

Are there any known issues of using Indexed DB as persistence layer

It use to be that browser support was lacking, but its supported in all major browsers now (caniuse indexDB)

Are you still using the indexDB solution you posted above?

[usman087]

Yes we have been using it in production for more than a month. No issues so far.

[PabloRomanH]

I tried this and it's working quite nicely.

The problem that I've encountered is that one of my queries needs to wait for the data to be hydrated before it can contact the backend (so I can retrieve only the new information). But since this code is asynchronous, the query is executed before this finishes. I tried disabling the query by default, but then I found that I have no way of reenabling that query that was created in a specific component from the hydration code that's running at the highest level. How would you solve this?

[PabloRomanH]

Okay, I managed to solve it. It turns out that enabling it in the "default settings" of the query also enables all the current instances of the query.

[dev-code-davis]

Thanks for the effort @usman087 . This could be really useful in offline usage scenario where the localstorage can't be access from serviceWorker. That's currently what my app requires.

Wondering what would be the most correct way to import this IndexedDB persistence module? As I see from officially supported localstorage implementation, it is imported as extra module. Have you organized it in similar way? Would like to know some details as to importing this.

[usman087]

I updated above code snippet and added how to us it with query client.

[jonathanstanley]

Thank you @usman087 I was just about to write the same modification using idb-keyval; glad I checked and found it here already.

To others who may not be aware of the potential benefits of IndexedDB API and its use of structured clone algorithm:

• Preserves references within the cloned data, allowing it to support cyclical and recursive structures that would cause errors for JSON.
• Date objects are restored correctly (rather than an ISO string)
• Ability to store/restore CryptoKeys
• In other words: stores more native types, while Web Storage which only stores strings.
• Can store more than 5MB

It doesn't look like browser compatibility will be an issue. Although IndexedDB doesn't solve everything, it seems like it would be better persister than Web Storage persist for react-query.

[jonathanstanley]

I posted a createIDBPersister proposal: #3198

[zikphil]

@jonathanstanley This is not working at all for me. If I swap this as-is instead of the localStorage implementation all my queries still get fetched on page reload as opposed to being read from the cache. Could this have something to do with the async nature of the library?

[TkDodo]

This sounds more like you get a background refetch because of the default staleTime: 0

[jonathanstanley]

I'm not sure; I updated to v4alpha and not using this version. v4alpha allows a different implementation which will be in the upcoming docs. You can see it: #3356