ShareDB and React Redux

[eMarek]

I am wondering how people use ShareDB in combination with React Redux?

I made a proof of concept which works, but it has few disadvantages. To be honest, it is kind of nasty. :D For start let me tell you that big portion of my frontend was already done before starting to think about using ShareDB. This means that all my actions, reducers and state were designed and completed already in the past. I didn't want to change this, because it would mean rewriting almost entire app. So I came across solution described below.

Part 1: Opening connection and creating local Doc.
There is nothing really special about this.

import ReconnectingWebSocket from 'reconnecting-websocket'
import sharedb from 'sharedb/lib/client'

var socket = new ReconnectingWebSocket('ws://localhost:8080');
var connection = new sharedb.Connection(socket);
var doc = connection.get('myCollection', 'myDocument');
doc.subscribe(() => console.log("SUBSCRIBED!"))

Part 2: Looking for a diff and submitting OP.
I want to realtime sync only one part of Redux state (state.myReducer.myData). I will probably use watcher in the future, but for proof of concept I made a simple React component.

import jsondiff from 'json0-ot-diff'
import { useSelector } from 'react-redux'

export const Watcher = () => {
  const myData = useSelector(state => state.myReducer.myData)
  const diff = jsondiff(doc.data, myData)
  if (diff.length > 0) {
    doc.submitOp(diff)
  }
  return null
}

Part 3: Dispatching doc.data on op event.
Since my local changes are already in the store I need to update Redux's state only on changes from server.

import Store from './redux/Store'
import { updateMyData } from './redux/actions/MyDataActions'

doc.on('op', (op, localChange) => {
  if (!localChange) {
    Store.dispatch(updateMyData(doc.data))
  }
})

Under the hood updateMyData action sends doc.data into the reducer which replaces entire JSON with the one from doc.data. This is the part which I hate the most. Also this happens on every op event. This means that anytime some other client makes a change my redux state updates. It is not even near perfect.

What do you think about this solution? Which approach do you use?

[curran]

I would recommend embracing ShareDB as a state container (source of truth), rather than duplicating the state in Redux.

[eMarek]

I agree this would be better. Has anyone tried to achieve this and is willing to share it with us?

[alecgibson]

We're doing something like this with vuex rather than Redux, but I believe the principles should be the same. The flow we use is different to yours, though. It goes something like:

1. Subscribe to the Doc
2. Store a mirror of the fetched data in Vuex
3. Receive op event
4. Apply the op to the Vuex data using our own poor-man's re-implementation of json0 (this is where the "magic" happens, and we tell Vuex how to interpret eg oi, od, li, etc. in a way that takes advantage of Vuex's reactivity system — I can't say I've used Redux, but I'd imagine something similar exists)
5. Change is cascaded through Vuex's reactivity, and we don't need to do anything particularly special from this point

When changing data, all ops are submitted via sharedb as the ultimate source of truth. The changes cascade through the reactivity system through the op events as outlined above, keeping ShareDB and our DOM in sync.

[curran]

Here's an example of what I'm using:

import { useEffect, useState, useContext, useCallback } from 'react';
import { ConnectionContext } from '../ConnectionContext';
import { WarningContext } from '../WarningContext';

export const useShareDBDoc = (collection, id) => {
  const connection = useContext(ConnectionContext);
  const { setWarning } = useContext(WarningContext);

  const [shareDBDoc, setShareDBDoc] = useState(null);

  const onError = useCallback(
    (error) => {
      setWarning(error.message);
    },
    [setWarning]
  );

  // Manage doc subscription.
  useEffect(() => {
    if (!connection) return;

    // Clear out old doc in case id changed.
    // (don't want the user to see stale stuff).
    setShareDBDoc(null);

    const doc = connection.get(collection, id);

    doc.on('error', onError);

    // Wait until subscribe finishes before passing this out,
    // so that downstream code can assume data is present
    // and that submitOp will always work.
    doc.subscribe((error) => {
      if (error) {
        onError(error);
      } else {
        setShareDBDoc(doc);
      }
    });

    return () => {
      doc.off('error', onError);
      doc.unsubscribe();
    };
  }, [connection, collection, id, onError]);

  return shareDBDoc;
};

I'm using RxJS so that changes in state high up in the component tree don't re-render the entire app (which is a problem with the Redux approach).

import { useEffect, useMemo, useCallback } from 'react';
import { BehaviorSubject } from 'rxjs';
import { DOCUMENT_INFO } from 'vizhub-database';
import { useShareDBDoc } from './useShareDBDoc';
import { useOpStream } from './useOpStream';
import { useSubmitOp } from './useSubmitOp';

export const useVizInfo = (initialVizInfo) => {
  const vizInfoDoc = useShareDBDoc(DOCUMENT_INFO, initialVizInfo.id);
  
  // Display initial viz until realtime connection has been established.
  const vizInfo$ = useMemo(() => new BehaviorSubject(initialVizInfo), [
    initialVizInfo,
  ]);
  
  const getPreviousInfo = useCallback(() => vizInfo$.getValue().info, [
    vizInfo$,
  ]);
  const vizInfoOp$ = useOpStream(vizInfoDoc, getPreviousInfo);
  
  // Update viz$ info.         
  useEffect(() => {
    const subscription = vizInfoOp$.subscribe(({ previous, next }) => {
      if (previous !== next) { 
        vizInfo$.next(next);   
      }
    });
    return () => subscription.unsubscribe();
  }, [vizInfo$, vizInfoOp$, initialVizInfo.id]);
      
  const submitVizInfoOp = useSubmitOp(vizInfoDoc);
      
  return {
    vizInfo$,
    submitVizInfoOp,
  };  
};    

Then, to use a value from there or submit ops:

import { useMemo } from 'react';
export const useSubmitOp = (shareDBDoc) =>
  useMemo(() => {              
    if (!shareDBDoc) {
      return undefined;        
    }

    return (op, propsToFindDoc) => {
      if (op.length > 0) {     
        const doc =            
          shareDBDoc instanceof Function
            ? shareDBDoc(propsToFindDoc)
            : shareDBDoc;      
        doc.submitOp(op);      
      }
    };
  }, [shareDBDoc]);   

import { useEffect, useMemo, useCallback } from 'react';
import { BehaviorSubject } from 'rxjs';
import { DOCUMENT_INFO } from 'vizhub-database';
import { useShareDBDoc } from './useShareDBDoc';
import { useOpStream } from './useOpStream';
import { useSubmitOp } from './useSubmitOp';

export const useVizInfo = (initialVizInfo) => {
  const vizInfoDoc = useShareDBDoc(DOCUMENT_INFO, initialVizInfo.id);
  
  // Display initial viz until realtime connection has been established.
  const vizInfo$ = useMemo(() => new BehaviorSubject(initialVizInfo), [
    initialVizInfo,
  ]);
  
  const getPreviousInfo = useCallback(() => vizInfo$.getValue().info, [
    vizInfo$,
  ]);
  const vizInfoOp$ = useOpStream(vizInfoDoc, getPreviousInfo);
  
  // Update viz$ info.         
  useEffect(() => {
    const subscription = vizInfoOp$.subscribe(({ previous, next }) => {
      if (previous !== next) { 
        vizInfo$.next(next);   
      }
    });
    return () => subscription.unsubscribe();
  }, [vizInfo$, vizInfoOp$, initialVizInfo.id]);
      
  const submitVizInfoOp = useSubmitOp(vizInfoDoc);
      
  return {
    vizInfo$,
    submitVizInfoOp,
  };  
};    

For example:

  const { viz$, submitVizInfoOp } = useContext(VizContext);
  const vizInfo = useValue(viz$, (viz) => viz.info);
  const vizHeight = useValue(viz$, getVizHeight); 
  const upvotes = useValue(viz$, getVizUpvotes);
...

Then you can use those in rendering logic.

These are bits and pieces from the proprietary https://vizhub.com/ codebase. I wish I had a complete Open Source example project to point you to. Maybe I could put something together... But hopefully this helps a bit. Good luck!

[eMarek]

I've figured it out that my solution is buggy. Calculating op with jsondiff and submitting it does lead to errors in some cases. It causes that some changes get re-applied into data which is wrong. Anyone looking at it rather not use my idea.

@alecgibson I agree that sharedb should be the ultimate source of truth. It is just difficult to achieve this right now since Redux is highly involved into all parts of my app. I am thinking about rewriting part of my app. At least for proof of concept.

@curran May I ask what is inside useOpStream? You pasted useVizInfo function twice. I must say that your idea about using custom hooks is totally brilliant. I love useShareDBDoc and useSubmitOp. I need to decode useVizInfo and example though.

[josephg]

I agree that sharedb should be the ultimate source of truth. It is just difficult to achieve this right now since Redux is highly involved into all parts of my app.

Make your redux actions just call submit in sharedb with an appropriate op, and then subscribe to sharedb's state changes from redux.

[eMarek]

Yes, this exactly is my plan. I have approx 15 actions with the same number of utils inside reducers. It might take me some time, but I think it is worth it. I will let you know about the result.

[eMarek]

@josephg What is the easiest or smartest way to be notified when certain document's data has changes. Let's say there is newer document version and I want to update redux state with it. Should I listen to doc.on('op', function(op, source) {...})or something else?

[curran]

@eMarek Here's useOpStream:

import { useMemo, useEffect } from 'react';
import { Subject } from 'rxjs';

// Subscribes to a ShareDB document for ops,
// returns an RxJS Subject that emits values for each op.
export const useOpStream = (shareDBDoc, getPrevious) => {
  const op$ = useMemo(() => new Subject(), []);

  // Connect to ShareDB doc for realtime connection.
  useEffect(() => {
    if (!shareDBDoc) {
      return;
    }

    // Update on each change.
    const handleOp = (op, originatedLocally) => {
      const previous = getPrevious();
      const next = shareDBDoc.data;
      op$.next({ previous, next, op, originatedLocally });
    };

    shareDBDoc.on('op', handleOp);

    return () => {
      shareDBDoc.off('op', handleOp);
    };
  }, [shareDBDoc, op$, getPrevious]);

  return op$;
};

[eMarek]

@curran, thank for sharing. I ended up doing something like example below.

First, a subscription hook:

import { Doc, Error, Op } from 'sharedb/lib/client'
import { useState, useEffect } from 'react'
import Share, ShareCollections from '../Share'
import Store from '../redux/Store'
import { myCollectionSubscribed, myCollectionOperation} from '../redux/actions/MyCollectionActions'

export const useMyCollection = (docId: string): Doc | null => {
  const [shareDocument, setShareDocument] = useState<Doc | null>(null)
  useEffect(() => {
    setShareDocument(null)
    if (!Share) {
      window.console.error('Could not subscribe to my collection. Missing database!')
      return
    }
    if (!docId) {
      window.console.error('Could not subscribe to my collection. Missing document id!')
      return
    }

    const myCollectionDocument = Share.get(ShareCollections.MY_COLLECTION, docId)
    setShareDocument(myCollectionDocument)

    if (myCollectionDocument.subscribed) {
      window.console.error('Could not subscribe to my collection. Already subscribed to this document id!', docId)
      return
    }

    myCollectionDocument.subscribe((error: Error) => {
      if (error) {
        window.console.error('Subscription to my collection failed!', docId, error)
        return
      }
      if (myCollectionDocument.type === null) {
        // If document type is null, then document has not been created yet.
        myCollectionDocument.create({})
      }
      Store.dispatch(myCollectionSubscribed(docId, myCollectionDocument.data))
    })

    myCollectionDocument.on('op', (operations: Array<Op>, localChange: boolean) => {
      Store.dispatch(myCollectionOperation(docId, myCollectionDocument.data, operations, localChange))
    })

    return () => {
      myCollectionDocument.unsubscribe()
    }
  }, [docId])
  return shareDocument
}

Those myCollectionSubscribed and myCollectionOperation are redux actions which gets dispatched. Funny thing is that you can actually store myCollectionDocument.data into the redux store. It updates automatically when new operation arrives. True ultimate source of truth. I had some troubles with react components not being re-rendered so I took advantage of myCollectionDocument.on('op', () => {} in order to manually correct object references in redux store.

And then I had to rewrite my previously used redux actions as @josephg suggested. They now calls shareDocument.submitOp(shareOperations) and here is an example:

import jsondiff from 'json0-ot-diff'
import { getShareDocument } from '../Share'
import { TMyItem, TReduxCompleteState } from '../redux/Store'
import { addMyItemToMyCollection } from '../redux/utils/MyCollectionUtils'

export const myCollectionAddItem = (myItem: TMyItem) => (
  dispatch: Dispatch,
  getState: () => TReduxCompleteState
): void => {
  const {
    various: { docId },
    share: { myCollection }
  } = getState()
  const myNewCollection = addMyItemToMyCollection({ ...myCollection }, myItem)
  const shareOperations = jsondiff(myCollection, myNewCollection)
  const shareDocument = getShareDocument(ShareCollections.MY_COLLECTION, docId)
  shareDocument.submitOp(shareOperations)
}

I am using jsondiff because I already have bunch of utils how to updated my collection. But in the future I will manually compose operations.

I hope my solution will help someone.

[willmtl]

@eMarek can you should me your myCollectionOperation?

[eMarek]

Nothing special:

export const myCollectionOperation = (
  shareDocumentId: string,
  data: TMyCollectionData,
  operations: Array<Op>,
  localChange: boolean
): TMyCollectionOperation => ({
  type: ActionTypes.MY_COLLECTION_OPERATION,
  shareDocumentId,
  data,
  operations,
  localChange
})

This is just an action creator which returns action which is dispatched. When action comes to reducer, I just replace current redux state data with new data from collection. That's it.

[willmtl]

@eMarek thank you I did something similar.

[curran]

A fresh take that also supports ingesting snapshots:

import React, { useState, useEffect } from 'react';
import ShareDB from 'sharedb/lib/client';
import { isClient, isProd } from '../environment';

// This hook sets up and exposes a ShareDB connection via WebSocket.
// This is how the client gets real time updates from the server.
// Only applicable to the client, does nothing during SSR.
// See https://share.github.io/sharedb/api/connection
export const useShareDBConnection = () => {
  const [shareDBConnection, setShareDBConnection] = useState(null);

  useEffect(() => {
    if (isClient) {
      setShareDBConnection(
        new ShareDB.Connection(
          new WebSocket(`${isProd ? 'wss' : 'ws'}://${window.location.host}`)
        )
      );
    }
  }, []);

  return shareDBConnection;
};

// This hook sets up a live updating representation of a viz.
export const useViz = ({ vizInfoSnapshot, vizContentSnapshot }) => {
  // Initialize the viz from server rendered snapshot data.
  const [viz, setViz] = useState({
    vizInfo: vizInfoSnapshot.data,
    vizContent: vizContentSnapshot.data,
  });

  // Initialize the ShareDB connection via WebSocket.
  const shareDBConnection = useShareDBConnection();

  // In the client, connect the viz to real time updates via ShareDB.
  useEffect(() => {
    if (shareDBConnection) {
      const { id } = vizInfoSnapshot.data;
      const vizInfoDoc = shareDBConnection.get(VIZ_INFO_COLLECTION, id);
      const vizContentDoc = shareDBConnection.get(VIZ_CONTENT_COLLECTION, id);

      const logError = (error) => {
        if (error) console.log(error);
      };

      // Verified manually that ingestSnapshot is working 3/17/22.
      // To verify, comment out the following ingestSnapshot line,
      // refresh, oepn DevTools, look in WebSocket (WS) section,
      // observe that a message is sent back with the entire vizInfo doc.
      // The message looks like `{"data":{"v":4,"data":{"id":"viz1", ... `.
      // Uncomment the following ingestSnapshot line and observe that the
      // message with the entire vizInfo doc is never sent, only the updates are.
      vizInfoDoc.ingestSnapshot(vizInfoSnapshot, logError);
      vizContentDoc.ingestSnapshot(vizContentSnapshot, logError);

      // Subscribe to real time updates.
      vizInfoDoc.subscribe(logError);
      vizContentDoc.subscribe(logError);

      // Process real time updates by updating state.
      // Note: we are using JSON1, which uses immutable patterns, so we can
      // assume that `doc.data` is a fresh new object after each change.
      // Unfortunately if you are using JSON0, this approach would not work.
      // See https://github.com/ottypes/json0/issues/26 for a workaround.
      const updateViz = () => {
        setViz({
          vizInfo: vizInfoDoc.data,
          vizContent: vizContentDoc.data,
        });
      };
      vizContentDoc.on('op batch', updateViz);
      vizInfoDoc.on('op batch', updateViz);
  
      return () => {
        vizContentDoc.off('op batch', updateViz);
        vizInfoDoc.off('op batch', updateViz);
      };  
    }       
  }, [shareDBConnection]);
        
  return viz;
};

export const VizPagePresenter = ({ pageData }) => {
  const { vizInfo, vizContent } = useViz(pageData.vizSnapshot);
  
  // Use vizInfo and vizContent as if they were set up via useState
...

Full code here: https://github.com/vizhub-core/vizhub/tree/32074471c1a03a0fd2a1479478131f105fdd7d6c/vizhub-v3/vizhub-app/src/presenters

Related to #307