Sync ephemeral presence data #207

gkubisa · 2018-04-26T16:23:04Z

This PR adds OT-aware ephemeral data sync as described by @nateps here.

How It Works

Server

The server-side changes are trivial because I reuse all the existing infrastructure for subscribing to docs and publishing ops. So, when you're subscribed to a document, you can publish and receive presence data. The server is essentially a PubSub system for presence data. Apart from that, its other important function is to broadcast null presence data on behalf of the client when it unsubscribes or disconnects, so that its peers know the client is gone.

The only issue I see with this solution is that if a server crashes or PubSub (eg Redis) fails, then the null presence would not be delivered and the peers would not know the client is gone. Considering that this is going to be rare in practice and user impact is likely minimal, I'm not sure, if it is worth the effort. Anyway, I think the solution is to simply expire peer presence data after some time, if they don't send regular heartbeat messages.

Client

Handling Current Presence Data

The current local (this client) and remote (peers) presence data is in doc.presence, which is also a part of the public API. Whenever any operation is applied to the local document, doc.presence is automatically transformed, so that for example a cursor would move forward by one character, if a user typed one character. In case of a hard rollback this data is wiped and syncing must start from scratch.

Publishing Local Presence Data

Publishing is delayed until the document is subscribed to and all pending writes have completed. When we're finally ready to publish, we can simply send the local data stored in doc.presence.

When publishing for the first time, or after subscribe, or reconnect, or hard rollback, a special flag (r - requestReplyPresence) is set on the message to ask all peers to publish their own presence data as soon as possible. This allows new clients to initialize. This system could be optimized a bit by sending the "reply" presence only to the client who requested it, instead of broadcasting. I left it out for now because I'm not sure if it is worth the effort.

Receiving Peer Presence Data

There are a few scenario to handle here...

If we get an out-of-sequence message, we ignore it.
If we get a null presence data from the server (version == null - peer disconnected or unsubscribed), we clear the relevant data in doc.presence.
If we get presence data for a future version of the document, we try to fetch the missing ops first.
If we get presence data for the current version of the document, we transform it against any inflight and pending ops and then store in doc.presence.
If we get presence data for an older version of the document, we transform it against the old ops, inflight op and pending ops, and then store the result in doc.presence.

A few things to note:

transforming presence against create, del or no-op makes presence data null
full presence data is sent every time. It greatly simplifies the implementation, and should have minimal impact on performance, considering that presence data is expected to be small in terms of bytes.
I cache past ops for 1 minute to enable transformation of presence data, if necessary. Considering that presence data is supposed to be real-time, I think we can safely discard that which arrives for versions more than 1 minute old.
I cache received presence data for 1 minute, so that I can compare the sequence numbers of newly received presence data and reject out-of sequence messages. In practice this should eliminate the risk of race conditions without introducing a risk of a memory leak (caused by peers constantly sending presence data and disconnecting).

The problem was that unsubscribe re-added the doc to the connection. Now the doc is removed from the connection after unsubscribe. Additionally, we're no longer waiting for the unsubscribe response before executing the callback. It is consistent with Query, unsubscribe can't fail anyway and the subscribed state is updated synchronously on the client side.

See See https://github.com/nodejs/Release

I had to add the --exit flag workaround to mocha.opts to make it exit when tests are done. A better long-term solution would be to ensure that nothing keeps node running after all tests are done, see https://boneskull.com/mocha-v4-nears-release/#mochawontforceexit.

The problem was that unsubscribe re-added the doc to the connection. Now the doc is removed from the connection after unsubscribe. Additionally, we're no longer waiting for the unsubscribe response before executing the callback. It is consistent with Query, unsubscribe can't fail anyway and the subscribed state is updated synchronously on the client side.

coveralls · 2018-04-26T16:25:01Z

Coverage increased (+0.1%) to 96.629% when pulling 428c46a on Teamwork:sync-presence into 68bde00 on share:master.

dcharbonnier · 2018-04-26T16:50:36Z

lib/agent.js

+
+Agent.prototype._presence = function(presence, callback) {
+  if (presence.seq <= this.maxPresenceSeq) {
+    return callback(new ShareDBError(4026, 'Presence data superseded'));


good practice to make the callbacks async no ?

Fixed, thanks 👍

gkubisa · 2018-04-27T11:41:57Z

lib/agent.js

+  return {
+    a: 'p',
+    src: this.clientId,
+    seq: seq != null ? seq : this.maxPresenceSeq,


!= null checks for null AND undefined. !== null would check only for null.

right sorry I never use equality operator

dcharbonnier · 2018-04-26T17:04:05Z

test/mocha.opts

@@ -1,3 +1,4 @@
 --reporter spec
 --check-leaks
 --recursive
+--exit


Better close correctly everything, this could hide a problem. I felt into the trap already ;-)

Yes, I completely agree. The existing tests did not close everything properly, so I just added this flag after upgrading mocha to get the old behaviour and keep the tests passing. In this PR I focused on adding presence data sync. Fixing tests to close everything properly is a separate issue to me.

gkubisa · 2018-04-27T11:41:57Z

lib/agent.js

+  return {
+    a: 'p',
+    src: this.clientId,
+    seq: seq != null ? seq : this.maxPresenceSeq,


!= null checks for null AND undefined. !== null would check only for null.

rawkode · 2018-06-02T09:40:49Z

I'd love to see this merged to the project. Has there been any discussion regarding this functionality?

curran · 2018-06-02T14:15:16Z

This looks good to me!

The only remaining issue IMO is the merge conflicts. @gkubisa Would you be able to resolve these merge conflicts? Thanks!

rawkode · 2018-06-03T11:19:05Z

@gkubisa If you don't have the time to rebase this, I'd be happy to pick it up

gkubisa · 2018-06-05T12:20:00Z

I resolved those merge conflicts now and the branch is ready to merge IMHO.

curran · 2018-06-10T01:31:28Z

@nateps I'm curious what your initial thoughts are on this PR? Thank you for your time.

gkubisa · 2018-06-11T13:44:01Z

I've just open-sourced Teamwork/ot-rich-text to provide a concrete example of an OT type supporting presence data. We use it at Teamwork.com for real-time rich-text collaboration.

@nateps , would you be able to review/merge this PR some time soon, please?

curran · 2018-06-12T13:51:30Z

@gkubisa It's awesome to see the great work!

I'm considering switching our dependency to @teamwork/sharedb, as this project is clearly suffering. This PR introduces such a useful feature, I'm surprised not a single maintainer has commented on it in 8 weeks.

gkubisa · 2018-06-12T15:17:01Z

I'm surprised not a single maintainer has commented on it in 8 weeks.

Yes, this is really disappointing. :-(

I'm considering switching our dependency to @teamwork/sharedb.

If you do switch, then you might also want to use the other related forks: https://www.npmjs.com/search?q=%40teamwork%2Fsharedb and https://www.npmjs.com/package/@teamwork/websocket-json-stream. The updated docs for OT types are here: https://github.com/teamwork/ot-docs.

Re the future of my fork, I'm going to add local undo/redo very soon. Afterwards I'll most likely just fix bugs and merge PRs.

dcharbonnier · 2018-06-12T15:20:33Z

I'm in @gkubisa

brainkim · 2018-06-12T18:11:33Z

😭 why can’t we all just get along 😤

alecgibson · 2018-10-31T16:17:55Z

NB: We have a repo that ties presence into Quill here: https://github.com/reedsy/quill-cursors

Could be interesting to compare implementations.

ericyhwang · 2018-10-31T16:32:29Z

Re presence for json0, it'd be nice to have full support eventually. That is a non-trivial chunk of work, as @curran points out.

I wonder if it's feasible to do a partial implementation initially, like only support presence for top-level text0 fields first and error on anything else. I haven't dove too deeply yet into this PR to know if that's advisable.

gkubisa · 2018-11-01T13:34:44Z

I wonder if it's feasible to do a partial implementation initially, like only support presence for top-level text0 fields first and error on anything else.

Given that json0 is so generic, perhaps presence on content embedded in json0 is all we could reasonably do? In that case json0 would just delegate presence handling to the appropriate embedded types. Probably the only presence transformations in json0 itself would be related to list operations, so that the correct embedded type would receive the presence later. Anyway, it seems very doable with some effort.

curran · 2018-11-02T07:38:13Z

That makes sense. I'm interested in adding presence for text0 fields within nested json0 objects.

curran · 2019-04-09T01:49:03Z

Hey check this out - @houshuang is working on JSON0 presence!

Here's the PR ottypes/json0#25

houshuang · 2019-04-09T12:43:46Z

In fact, that PR was submitted in error - I was trying to submit against a different upstream :) I thought I closed it, but I guess I didn't. Anyway we are absolutely happy to share this work, just not quite done yet (but feedback welcome!). Here's a quick demo video with Quill and presence/shared cursors. https://www.youtube.com/watch?v=CVjH56Q18cU

curran · 2019-04-10T10:50:07Z

I started working on json0 presence here datavis-tech/json0#2

curran · 2019-04-11T09:56:10Z

I asked @nateps et. al. "What's your take on presence?" during the last video call PR review meeting, and the ensuing conversation was completely new information to me. In summary, Nate expressed concern for scalability, and explained the problem as follows:

The server does not cache presence data.
Every time a new user joins and requests up-to-date presence, the server asks all other clients for their presence, resulting in a client -> server -> remote client -> server -> client round trip, for all remote clients.
This is fine for a few users, but if thousands of users join, performance would be problematic.
Because of this, namespacing the implementation internally would leave room for future implementations that have better performance characteristics.

I haven't dug into this PR enough to verify that all of this is true, but I just wanted to bump the conversation here with the latest thoughts from the Lever side.

My gut feeling is that the server could/should be made to cache presence data for all documents, probably in memory for a bounded period of time.

gkubisa · 2019-04-11T11:18:22Z

Nate's explanation is 100% valid. I implemented it the way it is because:

I assumed that the number of users editing the same document concurrently is not going to be huge in practice - max a few dozen. It is a valid assumption for the documents we manage in my company.
Caching and transforming presence on the server and reconciling it with the clients seemed very complex.
I had concerns about the server-side scalability of caching and transforming presence on the server.
I wanted to avoid stale presence data persisting on servers in case one server crashed and did not clean up the presence it managed.
I did not want to store presence as part of operations due to performance reasons.

houshuang · 2019-04-11T14:25:24Z

I started a write-up to better understand how this works here: https://docs.google.com/document/d/1lxBa74_sfkhaLg0s7S_8yIcboaNZvE6TNHtUYnhAlNc/edit# - note that this is intermingling a bit the concerns between the ShareDB presence mechanisms, how we are proposing to implement ot-json0 with subtypes etc. It already benefited from some clarifications from @gkubisa (thanks!)

curran

As I really would love to see this merged, so the larger community can develop presence work in downstream projects with confidence (e.g. OT types, editor bindings), I put in some time to do a thorough review. The thrust of my review is to narrow down the specific changes required in order to satisfy the original two requests from @nateps and @ericyhwang, which are as follows:

Requests for the current PR:

Add a ShareDB constructor option/flag for enabling presence. It looks like currently, some things like op caching could affect performance, even if you never call doc.submitPresence. It'd be good to not run that type of code if you're not using presence, like if none of your doc types don't support it.

Nate would like all the presence-related info to be nested underneath a top-level property, to make it easier to inspect what's related to presence. The current presence map might be doc.presence.currentDataByClient = {'src1': presenceData1, ...}.

An acknowledgement of the review's correctness/completeness from @ericyhwang and @nateps would be helpfup, and may inspire me to submit a follow-on PR that implements the changes, so that we can see this merged (I'm not sure if @gkubisa has the time at the moment to devote to this). Thanks everyone!

curran · 2019-04-15T02:45:37Z

.editorconfig

@@ -0,0 +1,9 @@
+root = true


This .editorconfig file seems extranious and IMO should be removed.

curran · 2019-04-15T02:46:23Z

.travis.yml

  - "8"
  - "6"
-  - "4"
-script: "npm run jshint && npm run test-cover"
+script: "ln -s .. node_modules/sharedb; npm run jshint && npm run test-cover"


Updates to .travis.yml seem out of scope for this PR. The changes are good, but probably should be kept in a separate PR.

curran · 2019-04-15T02:51:26Z

README.md

@@ -223,6 +228,9 @@ Unique document ID
 `doc.data` _(Object)_
 Document contents. Available after document is fetched or subscribed to.

+`doc.presence` _(Object)_


I think this is where the following comment from @nateps and @ericyhwang apply:

Nate would like all the presence-related info to be nested underneath a top-level property, to make it easier to inspect what's related to presence. The current presence map might be doc.presence.currentDataByClient = {'src1': presenceData1, ...}.

More concretely, the docs here could change from doc.presence to doc.presence.currentDataByClient, or whatever name is preferable.

The name doc.presence.current would feel familiar for React developers, as it's similar to ref.current.

curran · 2019-04-15T02:56:39Z

lib/client/doc.js

+  // The current presence data
+  // Map of src -> presence data
+  // Local src === ''
+  this.presence = Object.create(null);


Re: nesting, suggested change: doc.presence --> doc.presence.currentDataByClient

curran · 2019-04-15T02:57:00Z

lib/client/doc.js

+  this.presence = Object.create(null);
+  // The presence objects received from the server
+  // Map of src -> presence
+  this.receivedPresence = Object.create(null);


Re: nesting, suggested change: doc.receivedPresence --> doc.presence.received

curran · 2019-04-15T03:08:08Z

lib/client/doc.js

    this._sendOp();
  }
+
+  if (this.subscribed && !this.inflightPresence && this.pendingPresence && !this.hasWritePending()) {


if(ShareDB constructor presence flag)

curran · 2019-04-15T03:09:13Z

lib/client/doc.js

-  var op = this.inflightOp;
-  var pending = this.pendingOps;
+  var callbacks = [];
+  if (this.inflightPresence) {


Could be better grouped under if(ShareDB constructor presence flag)

curran · 2019-04-15T03:09:30Z

lib/client/doc.js

  this._setType(null);
  this.version = null;
  this.inflightOp = null;
  this.pendingOps = [];
+  this.cachedOps.length = 0;


if(ShareDB constructor presence flag)

curran · 2019-04-15T03:10:20Z

lib/client/doc.js

+
+// *** Presence
+
+Doc.prototype.submitPresence = function (data, callback) {


All of these methods could be definde only if(ShareDB constructor presence flag)

curran · 2019-04-15T03:11:12Z

package.json

@@ -16,7 +16,7 @@
    "expect.js": "^0.3.1",
    "istanbul": "^0.4.2",
    "jshint": "^2.9.2",
-    "mocha": "^3.2.0",
+    "mocha": "^5.1.1",


Mocha upgrade should be separate PR.

gkubisa · 2019-04-15T07:10:15Z

Thanks for the review @curran . Unfortunately, I currently have no time to work on it. Feel free to continue the work on this feature in your own fork. :-)

curran · 2019-04-16T07:10:46Z

Thanks for letting me know @gkubisa . I may be able to continue this work.

I did get presence working in a fork of json0, continuing the work started by @houshuang .

Here's a demo app: https://github.com/datavis-tech/json0-presence-demo

curran · 2019-04-17T03:33:00Z

@gkubisa Are there any presence-related changes in the Teamwork fork of ShareDB that are not in this PR? I'm starting in on the work of a new PR from this one, and wanted to make sure it's not missing anything. Thanks!

gkubisa · 2019-04-17T05:12:33Z

Great that you're taking it on. This branch should contain all presence-related changes, @curran .

curran · 2019-04-17T06:03:12Z

Excellent. Thanks for confirming! Work has commenced in #286 .

curran · 2020-07-22T17:02:38Z

Suggest to close as stale. Other presence PR has landed #322

gkubisa added 9 commits April 18, 2018 16:31

Update tested nodejs versions in .travis.yml

af84be6

See See https://github.com/nodejs/Release

Add a test

09edf92

Update tested nodejs versions in .travis.yml

9121baf

See See https://github.com/nodejs/Release

Add .editorconfig

4dbefd1

Update mocha

2ef8181

I had to add the --exit flag workaround to mocha.opts to make it exit when tests are done. A better long-term solution would be to ensure that nothing keeps node running after all tests are done, see https://boneskull.com/mocha-v4-nears-release/#mochawontforceexit.

Fix hasWritePending in op's callback

1489e36

Implement ephemeral "presence" data sync

a4499a5

gkubisa mentioned this pull request Apr 26, 2018

Add presence-related properties ottypes/docs#27

Open

dcharbonnier reviewed Apr 27, 2018

View reviewed changes

gkubisa commented Apr 27, 2018

View reviewed changes

gkubisa added 5 commits April 27, 2018 12:53

Execute some callbacks asynchronously

33c7264

Don't send presence unnecessarily

8ff4b33

Re-sync presence after re-subscribe and re-connect

0ff380d

Emit presence asynchronously

d67dd6a

Add submitted param to presence event

e8ec215

Merge branch 'share/master' into sync-presence

9c291b2

Merge 'upstream/master' into fix-doc-destroy

c15448f

jasonm mentioned this pull request Mar 30, 2019

POC: OT type rich-text support for teamwork/sharedb presence mechanics jasonm/rich-text#1

Closed

This was referenced Apr 9, 2019

Interesting Cursor Transform Work by Robert Lord ottypes/json0#27

Open

Presence Fail datavis-tech/json0#1

Closed

Presence 2 ottypes/json0#29

Closed

WIP Presence datavis-tech/json0#2

Closed

curran mentioned this pull request Apr 12, 2019

Presence ottypes/json0#30

Open

curran suggested changes Apr 15, 2019

View reviewed changes

curran mentioned this pull request Apr 15, 2019

Presence for OT Types ottypes/docs#29

Open

curran mentioned this pull request Apr 16, 2019

Alternative Presence Implementation from Propeller Factory #285

Closed

curran mentioned this pull request Apr 17, 2019

WIP Presence #286

Closed

This was referenced Apr 19, 2019

WIP Presence Continuation Teamwork/sharedb#2

Closed

WIP Presence continuation Teamwork/sharedb#3

Open

curran mentioned this pull request Oct 21, 2020

Close Stale PRs #398

Open

curran mentioned this pull request Jan 2, 2021

transformPresence is not a function #423

Open


		// *** Presence

		Doc.prototype.submitPresence = function (data, callback) {

Sync ephemeral presence data #207

Are you sure you want to change the base?

Sync ephemeral presence data #207

Conversation

gkubisa commented Apr 26, 2018

How It Works

Server

Client

Handling Current Presence Data

Publishing Local Presence Data

Receiving Peer Presence Data

coveralls commented Apr 26, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rawkode commented Jun 2, 2018

curran commented Jun 2, 2018

rawkode commented Jun 3, 2018

gkubisa commented Jun 5, 2018

curran commented Jun 10, 2018

gkubisa commented Jun 11, 2018

curran commented Jun 12, 2018

gkubisa commented Jun 12, 2018 • edited

dcharbonnier commented Jun 12, 2018

brainkim commented Jun 12, 2018

alecgibson commented Oct 31, 2018

ericyhwang commented Oct 31, 2018

gkubisa commented Nov 1, 2018

curran commented Nov 2, 2018

curran commented Apr 9, 2019 • edited

houshuang commented Apr 9, 2019

curran commented Apr 10, 2019

curran commented Apr 11, 2019

gkubisa commented Apr 11, 2019

houshuang commented Apr 11, 2019

curran left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

curran Apr 15, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gkubisa commented Apr 15, 2019

curran commented Apr 16, 2019

curran commented Apr 17, 2019

gkubisa commented Apr 17, 2019

curran commented Apr 17, 2019

curran commented Jul 22, 2020

coveralls commented Apr 26, 2018 •

edited

gkubisa commented Jun 12, 2018 •

edited

curran commented Apr 9, 2019 •

edited

curran left a comment •

edited

curran Apr 15, 2019 •

edited