Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.
Sign upMulti-device editing/syncing with CRDTs #250
Comments
This comment has been minimized.
This comment has been minimized.
I discussed this a bunch with @raphlinus today and now have a much better understanding of the whole CRDT model and how to proceed. I've included some (very high-res page-bloaty) pictures of the whiteboard from our discussion along with some quick notes. I've edited the main issue body based on the discussion with new items. cc @cmyr @rkusa since both of you expressed interest in CRDT stuff Example of a revision history, right column is the deletion set at every revision, identical concurrent deletions (last two) necessitate multi-set (each deleted index also has a count) for reversibility so that undoing/reversing one doesn't un-delete a multi-deleted character. Things along the bottom are state in the engine for the head revision: |
This comment has been minimized.
This comment has been minimized.
Thanks for the update! I have a question to the first image. Why is the deletion set at line three ( Another question regarding the tombstones. Since they are not part of the revision, we only have the most reason tomstone string, right? I've added the tombstone state to each line of your first example (and also added an additional starting line for the sake of my following question).
Is this the correct way of how tombstones are going to work? If yes, when I undo all of these revision ( |
This comment has been minimized.
This comment has been minimized.
The leftmost deletion set, also known as Let me rewrite your example to be correct. Below the first line is in the revision, and the next line is the contents of
|
This comment has been minimized.
This comment has been minimized.
Now, it makes sense. Thanks a lot for the detailed answer! |
added a commit
that referenced
this issue
May 8, 2017
added a commit
that referenced
this issue
May 8, 2017
trishume
added
the
planning
label
May 9, 2017
trishume
referenced this issue
May 24, 2017
Merged
Implement serde traits for Engine, Subset, Rope, Revision #316
trishume
self-assigned this
Jul 12, 2017
This comment has been minimized.
This comment has been minimized.
erlend-sh
commented
Nov 2, 2018
Might this become compatible with the CRDT version control system Memo that @as-cii and @nathansobo are working on in xray? |
This comment has been minimized.
This comment has been minimized.
@erlend-sh These two projects aren't particularly related, so I don't think it makes too much sense to think of compatibility; however my understanding is that Memo is designed to be a standalone system that exposes an API that can be used by any consumer, so supporting it in xi at some point in the future should definitely be possibly. It's certainly an interesting project, and I've been following along with interest. :) |
trishume commentedMay 4, 2017
•
edited
This is a tracking issue/roadmap/plan for how I'm going to proceed with adding the ability to synchronize documents. The tasks below correspond approximately to PRs I plan on writing and the order I plan on doing them.
Edit 2017/08/04:📝 👁 I wrote a detailed document describing the CRDT
Edit 2017/05/04: Add more detail.
Edit 2017/05/04: Add ideas for optimizing
Edit 2017/07/12: Heavily refactored list to reflect what actually happened, include links to more PRs, and update plan
cc @raphlinus
xi-rope
refactoringPrior to starting this project, the rope and mini-CRDT data structures were mostly undocumented, lightly tested and used some representations that didn't have good time complexity or didn't fit well with turning the mini-CRDT into a full CRDT. The first step of the project was to refactor and replace some of the existing data structures and algorithms and comment and test them.
from_union
in a multiset so that it can be computed both forward and backwards in the revision history. #304 #312Revision
, possiblyEngine
and transitive dependencies. #316Vec<Revision>
, I think I may also need aRope
likeEngine
but I haven't readengine.rs
yet so I don't quite know how things fit together.Rope
, might need a custom serializer to make it serialize as a string or something, serializing the tree structure like a derive would do seems unnecessary.Fuchsia last-write-wins prototype
As an initial use case of this functionality, implement syncing using ledger for fuchsia/xi.
fidl
interface for ledger. Gerrit CL #339CRDT Merge operation
Implement the
merge
function onEngine
for histories not involving undo.Integrate CRDT into Fuchsia
You can now open Xi on two Fuchsia devices and type into both of them concurrently and they will synchronize and reach a consistent state shortly afterward.
Documentation
Improve CRDT robustness and functionality
Make it fast
The above will have terrible awful scaling properties and will slow to a crawl in many use cases. The next step is to figure out ways to improve the CRDT representation and operations to have better time and space complexity. Some of these ideas are well thought out and are clear wins, others are more speculative half-baked ideas (marked by a "🤔 ").
PageWatcher
update case.Subset
s in a rope with an aggregation on number of inserted and deleted characters so that calculations can do things in the middle of the text without being O(n).Engine
being put in, that will guarantee things are only appended.n
is the size of the history after the undo/common prefix. This would definitely be useful for undo because it would allow an arbitrarily long undo history without significant performance degradation, and would effectively allow us to remove the garbage collector.Subset
so that there are lessSubset
s to rebase and transform repeatedly inmerge
. Like #351 but more extreme.