Seeking Guidance for Serverless Bi-directional Sync Between Two Dexie.js Instances

[petr24]

Hello,

I am developing a Chrome extension that operates in a cross-origin setup due to the way extension work. The content script ("client") runs under the host origin (e.g., example.com), while the background page ("server") operates under chrome-extension://{extensionId} origin. Communication between these environments is managed via Chrome's messaging API.

My challenge is to implement a shared database between these environments, however due to cross-origin policies It's not possible to share a single Indexed DB. Although Chrome provides a simple key/value storage API accessible to both environments, it does not meet my needs for more complex data handling.

I am exploring local synchronization options for two instances of Dexie.js. The standard approach, using a custom ISyncProtocol as detailed in the Dexie.Syncable documentation, and the sync_client library, both seem to require a server-based backend, utilizing connections to hosted URL endpoints (see source code).

Is there a straightforward method or existing workaround to achieve bi-directional synchronization between two local Dexie.js databases within this Chrome extension setup? Any suggestions or advice would be highly appreciated.

Thank you.

[dfahlander]

Would it be an option to encapsulate the Dexie calls via a service that uses chrome messaging api instead? It could either be built around your needs or it could mirror Dexie 's api (or parts of it) and be implemented by sending async messages and responding to them on the "server" post. That way you would not be reliant on sync and your extension would be less intrusive since it would not be using up storage quota from visited sites. And if having sync, you might experience situations when one of the parts are being cleaned up and the other is still there. I suppose troubleshooting would be harder.

But if you want to try, you can actually implement sync using chrome's messaging api without the need of a network server. Use runtime.sendMessage on the client and runtime.onMessage on the "server". https://developer.chrome.com/docs/extensions/develop/concepts/messaging but there's a lot to implement on the server part. I don't have an easy example implementation but maybe you could take the old concept from https://github.com/dexie/Dexie.js/blob/master/samples/remote-sync/websocket/WebSocketSyncServer.js and translate it to using runtime.onMessage and to querying a Dexie instance instead of its dummy tables.

[petr24]

I had the same initial thought of effectively doing a REST design. Client sends everything related to Dexie db to background via messaging. However a few downsides emerge related to reusable components, reactive queries, reusable code. Take liveQuery in a svelte component as an example. The component works anywhere with a chrome://{extensionId} host origin, but not in context script "client". I'd then have to run liveQuery in background context for any "client" component, but a ton of boiler plate for managing the liveQuery subscription based on "client" component lifecycle.

It's doable, a ton of code, but it doesn't feel elegant.

As far as the WebSocket code examples, I actually tried to recreate the solution using messaging passing, however the issue I found was I still needed to call Dexie.syncable.connect() to get things started, and that of course assumes a hosted url. If I'm not mistake I could reimplement WebSocketSyncProtocol.js & WebSocketSyncServer.js but that wouldn't be enough. I believe i'd also have to change syncable-connect.js (Line 6) for this to work.

[dfahlander]

This is an interesting use case. I still think propagating calls and liveQueries over messaging would be the most optimal solution (conceptually). If there were some 3rd party library capable of exposing service methods returning promises or observables and translating them to messages and providing a corresponding client proxy - then queries could be performed on the server in the service and the client could use the proxy. Not saying it would be trivial to implement though unless someone has already done it.

[glundgren93]

I have a similar use case. I'm trying to create an extension that will store data on a Indexed DB instance on another origin. On my case the client is not a content script, but an actual webpage. Not sure if its feasible though.

[petr24]

After a handful of experiments, I went with encapsulation and message passing. This turned out to be much simpler than any alternative. There was some additional overhead of handling client disconnects (ie tab/window closed), and component life cycles but overall straightforward. @glundgren93 the logic for webpage or content script will be the same. ie If the URL is different than chrome-extension://{extensionId} message passing is the way to go. I'll post my solution tomorrow so you can work off that if needed.

[glundgren93]

Nice. It would really help a lot to see how you managed to do it!

[petr24]

Here it is, in my particular use case I wanted a seamless interface with svelte stores. For any future readers, feel free to replace svelte specifics for some other framework.

Start with Dexie db Init

// db.js
import Dexie from 'dexie';

const dexieDb = new Dexie("myDatabase");

dexieDb.version(1).stores({
    friends: '++id, name, age', // Primary key and indexed props
});

dexieDb.open().catch (function (err) {
    console.error('Failed to open db: ' + (err.stack || err));
});

export {dexieDb};

Then I init a svelte store interface for "friends" data.

//friendsStore.js
import {sendMessage} from "../utilities.js";
import {dexieStorageAdapter} from "./dexieStorageAdapter.js";

function createStore(filters) {

    const store = dexieStorageAdapter({
        collection: "friends",
        filters: filters,
        initialValue: []
    });

    return {
        subscribe: store.subscribe,
        isLoading: store.isLoading,
        addFriend: (dataObject) => sendMessage({type: "friends-addFriend", dataObject}),
        updateName: (id, newName) => sendMessage({type: "friends-updateName", id, newName})
    };

}


const friendsNew = createStore;
const friendsStore = friendsNew();

export {friendsStore, friendsNew};

Here is dexieStorageAdapter which is what makes the Svelte store interface reactive & handles creating the active subscription between the content script and other pages to background context. I promisfied the implementation so I can have a loading state, not necessary for latency time detection but nice to know if result was loaded when it's empty. The code sends a message to the background page and registers a listener to receive state updates from the background page.

//dexieStorageAdapter.js
const subscriptions = {};

// Define default settings
const defaultSettings = {
    collection: undefined,
    initialValue: undefined,
    filters: [],
    serialize: noop,
    deserialize: noop
};

function noop(data) { return data; }

// Adapter to manage Chrome storage areas
function createStorageAdapter(options = {}) {

    const {collection, initialValue, filters} = {...defaultSettings, ...options};
    if (!collection) throw new Error("No collection provided for storage");

    const promises = new Map();

    function subscribe(callback) {

        //Run callbacks right away, this way stores are initialized to initialValue and not undefined
        callback(initialValue); //Callback for subscription

        // Manage subscribers specific to each collection
        const subscribers = subscriptions[collection] || (subscriptions[collection] = []);

        const chromeStorageCallback = function(result) {
            callback(result); //Callback for subscription
        };

        subscribers.push(chromeStorageCallback);
        getDataPromise().then(callback).catch(console.error);  // Fetch data and then notify the subscriber

        //Unsubscribed
        return function() {
            const index = subscribers.indexOf(chromeStorageCallback);
            if (index !== -1) { subscribers.splice(index, 1); }
            if (subscribers.length) return;
            chrome.runtime.sendMessage({type: "removeSub", collectionName: collection}, () => {});
        };
    }

    function getDataPromise() {

        if (promises.has(collection)) return promises.get(collection);

        const promise = new Promise((resolve, reject) => {
            chrome.runtime.sendMessage({type: "createOrGetSubscriptionData", collectionName: collection, filters}, function(response) {
                if (chrome.runtime.lastError) return reject(chrome.runtime.lastError.message);
                if (response.error) return reject(response.error);
                resolve(response.data);
            });
        }).finally(() => {
            promises.delete(collection);
        });

        promises.set(collection, promise);
        return promise;
    }

    return {
        subscribe,
        isLoading: getDataPromise
    };
}

chrome.runtime.onMessage.addListener(function(message) {
    if (message.type !== "tab-db-sub") return;
    if (!message.collectionName || !message.data) return;

    const subscribers = subscriptions[message.collectionName];
   //Handles situation on page reload, where subscription is active even though no listeners, so it closes it
    if (!subscribers || !subscribers.length) {
        chrome.runtime.sendMessage({type: "removeSub", collectionName: message.collectionName}, () => {});
        return;
    }

    subscribers.forEach(callback => callback(message.data));
});

export function dexieStorageAdapter(options = {}) {
    return createStorageAdapter(options);
}

Here is the code that handles receiving the above messages, ignore "registerEvents" that's just a small wrapper on my end so I don't have to have all messages listeners in one big file for background page messages. All this does is listen for events to add/delete subscriptions.

//subscription-events.js
import {registerEvents} from "./eventRegistration.js";
import {SubscriptionManager} from "./subscription-manager.js";

const subscriptionManager = new SubscriptionManager();

registerEvents({
    "createOrGetSubscriptionData": async function({message, sender, sendResponse}) {
        const data = await subscriptionManager.getDataAndSubscribe(message.collectionName, sender.tab.id, message.filters);
        sendResponse({error: null, data: data});
    },
    "removeSub": async function({message, sender, sendResponse}) {
        subscriptionManager.unsubscribe(message.collectionName, sender.tab.id);
        sendResponse();
    }
});

//Called when window/tab is closed
chrome.tabs.onRemoved.addListener(function(tabId) {
    // Remove this tabId from all subscriptions
    subscriptionManager.handleClientDisconnect(tabId);
});

And finally the actually Dexie implementation for liveQuery. I'm not sold on my filters implementation of using an object literal, but it's simple enough for my use case that I left it in here.

//subscription-manager.js
import {liveQuery} from "dexie";
import {dexieDb} from "../../Shared/Scripts/db.js";

const FILTERS_REF = {
    reverse: (query) => query.reverse(),
    sortByCreatedDate: (query) => query.sortBy('createdDate'),
    whereAll: (query) => query.where({"isDeleted": 0, "starred": 0})
};

class SubscriptionManager {
    constructor() {
        this.subscriptions = new Map();
    }

    #query(collectionName, filters = []) {
        let query = dexieDb[collectionName];
        if (!filters.length) return query.toArray();

        filters.forEach(filterId => {
            if (FILTERS_REF[filterId]) {
                query = FILTERS_REF[filterId](query);
            }
        });

        return query;
    }

    #removeClient(subscription, id) {
        subscription.clients = subscription.clients.filter(i => i !== id);
    }

    #closeSubscription(subscription, collectionName) {
        //Don't close the subscription if there are any clients left
        if (subscription.clients.length) return;
        subscription.dexieSubscription.unsubscribe();
        this.subscriptions.delete(collectionName);
    }

    #messageSubscribers(collectionName, message) {
        if (!this.subscriptions.has(collectionName)) return;
        const clients = this.subscriptions.get(collectionName).clients || [];
        if (!clients.length) return;
        clients.forEach(id => chrome.tabs.sendMessage(id, message));
    }

    #subscribe(collectionName, id, filters) {
        //If sub exists, just add the subscriber, no need to sub twice on the same data
        if (this.subscriptions.has(collectionName)) {
            const subscription = this.subscriptions.get(collectionName);
            if (!subscription.clients.includes(id)) subscription.clients.push(id);
            return;
        }

        const observable = liveQuery(() => this.#query(collectionName, filters));
        const dexieSubscription = observable.subscribe({
            next: (result) => this.#messageSubscribers(collectionName, {
                collectionName,
                type: "tab-db-sub",
                error: null,
                data: result
            }),
            error: (error) => this.#messageSubscribers(collectionName, {
                collectionName,
                type: "tab-db-sub",
                error: "Error getting subscription",
                originalError: error,
                data: null
            })
        });

        this.subscriptions.set(collectionName, {dexieSubscription, clients: [id]});
    }

    getDataAndSubscribe(collectionName, id, filters) {
        //subscribe is for continuous updates
        this.#subscribe(collectionName, id, filters);
        //Return data for initial state to client
        return this.#query(collectionName, filters);
    }

    unsubscribe(collectionName, id) {
        if (!this.subscriptions.has(collectionName)) return;
        const subscription = this.subscriptions.get(collectionName);
        this.#removeClient(subscription, id);
        this.#closeSubscription(subscription, collectionName);
    }

    handleClientDisconnect(id) {
        if (!this.subscriptions.size) return;
        this.subscriptions.forEach((subscription, collectionName) => {
            this.#removeClient(subscription, id);
            this.#closeSubscription(subscription, collectionName);
        });
    }
}

export {SubscriptionManager};

And the background page event listeners for svelte store methods above ("addFriend", "updateName")

//friends.js
import {dexieDb} from "../../../Shared/Scripts/db.js";
import {registerEvents} from "../eventRegistration.js";
import {asyncHandle} from "../../../Shared/Scripts/utilities.js";

registerEvents({
    "friends-addFriend": async function({message, sendResponse}) {
        const result = await asyncHandle(dexieDb.friends.add(message.dataObject));
        sendResponse(result);
    },
    "friends-updateName": async function({message, sendResponse}) {
        const {id, newName} = message;
        const result = await asyncHandle(dexieDb.friends.update(id, {name: newName}));
        sendResponse(result);
    }
});

Small note, asyncHandle and sendMessage are just small wrapper convenience functions. asyncHandle returns error as a value since the response get serialized with sendResponse, throwing errors from background to content script isn't possible.

At the end here is how implementation works.
Once you have this store interface you can use it in components like any other store.

import {friendsStore} from "../../../../Shared/Scripts/Stores/friendsStore.js";    
$friendsStore //Fully reactive

And to have store with filters, just instantiate a new one with key names of the filters in FILTERS_REF.

import {friendsStore, friendsNew} from "../../../../Shared/Scripts/Stores/friendsStore.js";
const friendsFiltered = friendsNew(['reverse', 'sortByCreatedDate']); //filters map to an object literal for modifications

$friendsFiltered

//These are promises
friendsFiltered.addFriend({name: "Jack", age: 25});
friendsFiltered.updateName(1, "Bill");

//Use with asyncHandle to get err, res from promise
const [err, res] = await asyncHandle(friendsFiltered.addFriend({name: "Jack", age: 25}))