Interest

[eukreign]

I've been following this project since last year and am really excited about the possibilities this brings to general purpose OT.

It seems that there hasn't been much activity on this and I was wondering if you are still working on it in private or if you have lost interest and moved on to other things?

[josephg]

Hey! Great question.

I still want to go forward with it at some point. The primary blocker when I stopped working on it is that I realised there are some operations that simply can't be transformed, and so I'll need to add conflict markers.

For example:

• Doc: {a:{}, b:{}}
• User 1: Move a inside b
• User 2: Move b inside a

.. In this case, the only sensible result from transforming those operations would be to delete both a and b, but thats super surprising from the user's point of view. So I think the operations should conflict instead. If they can do that I'm going to have to rethink the semantics of all the other operations to see if it makes sense to allow them to conflict as well. I think the API I want is for applications to specify a spec of which operations (/ paths?) should generate conflicts, and which should simply transform automatically.

When I realised all that I put it down until I have more time to come at it fresh. If someone paid me to work on it it might be a different story.

[laughinghan]

In this case, the only sensible result from transforming those operations would be to delete both a and b, but thats super surprising from the user's point of view.

Why doesn't one of them win and the other get discarded, like I assume happens with object insertion conflicts? Meaning two people trying to insert the same key in an object:

• Doc: {}
• User 1: Transform to {a:1}
• User 2: Transform to {a:2}

Result: both users end up with {a:1}.

(Or {a:2}, doesn't matter as long as they end up with the same thing, right?)

I may be missing something but that makes way more sense to me. By the way this project is super cool and you're super cool!

[josephg]

Aw thanks!

Um, its a worse than that because changes can back up.

• Doc: {a:123, x:{}, y:{}}
• User 1: Move x inside y, add 1 to doc.a and move a->x
• User 2: Move y inside x

If we make user 2's change win, what do we transform user 1's operation to? doc.x won't have moved away. So what happens to doc.a? (It could be much worse, too - they could have moved a -> b, b -> c, c ->d, .... and finally w -> x and set a = "hi". Do we have to undo all of those changes? Do we discard user 1's insert as well?

Its a red hot mess.

[laughinghan]

Woah you responded really fast! I'm gonna submit a PR with a bunch of typo fixes to the Spec then.

If we make user 2's change win, what do we transform user 1's operation to? doc.x won't have moved away. So what happens to doc.a?

Hmmmmmm. My intuition is that the move a->x conflict should resolve similarly to an insert; that is, this is a similar problem to:

• Doc: {x:{}, y:{}}
• User 1: Move x inside y, insert doc.x = 1
• User 1: Move y inside x

The problem here is that user 1 got rid of doc.x and expects to be able to insert there. In JSON0, was it possible for deletion of a key to be discarded? If so, how did we transform subsequent inserts of that key? Or was deletion in JSON0 impossible to conflict, only insertions, and now the problem is that a deletion is being atomically coupled to an insertion (i.e. a move)?

[laughinghan]

Hmmmmmm it's always been possible (that is, it's also possible in JSON0) to lose huge insertions, right? (User 1 inserts huge thing into object; User 2 replaces the object with a number.) Maybe losing mass moves isn't worse?

[josephg]

In JSON0, is it possible for deletion of a key to be discarded? If so, how do we transform subsequent inserts of that key? Or was deletion in JSON0 impossible to conflict, only insertions, and now the problem is that a deletion is being atomically coupled to an insertion (i.e. a move)?

In JSON0 the only operations on objects are insert and delete. You can't move something inside an object (or rename a key or anything like that). So delete vs delete always just deletes it. Insert vs insert will pick one winner, but from the point of view of the loser - well, they know the content they just tried to insert. Worst case they can re-insert it or something. And you can't have an insert operation vs a delete operation.

But moves are different in two ways:

• The content you're moving might change between when you create the op and when the move gets executed on the server.
• Your changes are sort of internally conflicting if the move gets punted. Its different from two inserts to the same location, where you're conflicting with another person.

Oh and we can run into a similar problem with moves to the same location:

• Doc: {a:1, b:2}
• Client 1: Move a -> x, insert a:10
• Client 2: Move b -> x, insert b:10

I'm tempted to support something like an op saying move a -> x, but if that would conflict, move a -> end of doc.lostandfound list. Its impossible for a list push to conflict, and then we can guarantee that the following insert of a:10 will succeed.

[josephg]

(But yeah you can have big overwrites like that. And they are actually super rare in practice)

[green-coder]

The doc.lostandfound list is a viable solution (the expected solution, to be precise) for my use case (a collaborative Scratch-like) where it costs a lot to the users (typically young students) to create tree elements while it is easy and fast for them to move them around, parent/deparent them.

[josephg]

Great. Well maybe we should do that then. The other problem is simply that we'll probably want to bundle the lost and found entry with some application-specific metadata (which users conflicted, author, timestamp). That information will need to be passed through somehow - but maybe I can just have transform get passed an optional options object with a wrapLostEntry(op) method that does the work if it's needed. And I'm also tempted to make the behaviour configurable - so you can make it conflict instead if you want.

[josephg]

From memory I think figuring out this behaviour is pretty much all that's left btw. The code is tight though - it's probably going to be a small change but not a simple change.

[jhurliman]

Allowing for a conflict instead of the lost and found behavior would help this generalize out to my use case and presumably others.

[green-coder]

@jhurliman Please let us know about your use case and why you think it would be better to have a conflict. Also, please describe what means a conflict for you, to make sure that we are all talking about the same thing.

Let's all try to make a verbosity effort to make it simpler for @josephg to make his design decisions.

[jhurliman]

This is for a multivariate testing framework where you have one base JSON document and lots of diffs to that base document organized into separate "experiments". The goal is to be able to independently create experiments but apply multiple experiments at runtime. For many of the edits this just requires a way to cleanly merge the changes, but what if one experiment modifies a property of object a but another experiment deletes object a entirely? The best outcome here would be to generate a "conflict" which is logged somewhere and choose not to apply one or both of the experiments entirely.

EDIT: In this non-interactive context, the idea of a lost and found list doesn't make sense because 1) the changes are stored outside of the base document so nothing would be "lost" by a failed merge, and 2) there is no human in the process that would be able to pluck things out of lost and found and manually reapply them.

[laughinghan]

The doc.lostandfound list is a viable solution (the expected solution, to be precise) for my use case

huh, really? My biggest problem with that is doc.lostandfound becoming like, a reserved path. What if someone sets doc.lostandfound to 1 or anything other than a list? If we're gonna do this, the "lost and found" list should be out-of-band. What if instead of a literal list, we emit an event that can optionally be ignored, kinda like how you can trap arithmetic overflow in assembly?

You're welcome to assemble such emitted/trapped events into a list of lost-and-found, moved-but-failed values, but you're also welcome to just ignore them. (Contrast if a list was pushed to, if you ignore that list you get a memory leak.)

[laughinghan]

... and so I'll need to add conflict markers.
... So I think the operations should conflict instead. If they can do that I'm going to have to rethink the semantics of all the other operations to see if it makes sense to allow them to conflict as well. I think the API I want is for applications to specify a spec of which operations (/ paths?) should generate conflicts, and which should simply transform automatically.

Allowing for a conflict instead of the lost and found behavior would help this generalize out to my use case and presumably others.

Super against this. My fundamental understanding of OT is based on my understanding of distributed version control systems like git. In the version control world, there's a distinction drawn between "textual" merge conflicts, and "semantic" merge conflicts: textual conflicts are when there's a conflict in the literal changes to the text, such as two people modifying the same line in the same file; whereas semantic conflicts are when two valid changes combine into an invalid merge, such as if one person renames a function and all its call sites, but another person adds a new call to the function in a new file, which is not a conflict at the changes-to-text-files-level, but is an important kind of conflict because it won't compile/run since the new function call uses the old name from before the other person renamed it. git is oblivious to semantic conflicts, of course, but with textual conflicts it will stop and demand you fix the conflict before proceeding, ideally so that your manually merged result will be a valid working state.

My fundamental understanding of OT is that it guarantees there are never textual conflicts, or equivalently, that everything that would be a textual conflict is automatically resolved identically by all parties; while also promising that changes that clearly, unambiguously don't semantically conflict will definitely be resolved as expected, but this promise is "best effort": it's meant for like, two changes to totally separate parts of a document; not, two changes to the same part of a document that a human could tell what the "expected" merge of is, but the algorithm isn't sophisticated enough to. The whole reason that textual conflicts need to be resolved automatically is that lots of tiny changes are being exchanged in real-time, but conversely this means that semantic conflicts that happen can be resolved interactively in real-time; for example if two people start typing in the same place at the same time, and are dissatisfied with whose insertion was automatically chosen to be first, they can switch it around.

Which is why I'm a little befuddled why this is that big of a problem in the first place. Okay, I understand now, conflicts involving a move can't be resolved by ignoring a move because other stuff may have happened in the place that the move vacated from; but then just resolve such conflicts as drops and move on, and if that's not what the user wanted, that's what Ctrl-Z is for, right?

(Ctrl-Z would add back the value to the place that the move vacated from; if something else had been added there, it gets dropped, and if you want to keep both values, you gotta copy the tried-to-move-but-instead-dropped value and then Ctrl-Y to add back the something else, and past the dropped value somewhere else. Which is maybe kinda annoying but strictly better than stuff lost to insertion conflicts, which you can't even recover via Ctrl-Z because that's for your own actions not counterparties.)

[josephg]

@laughinghan:

huh, really? My biggest problem with that is doc.lostandfound becoming like, a reserved path. What if someone sets doc.lostandfound to 1 or anything other than a list? If we're gonna do this, the "lost and found" list should be out-of-band. What if instead of a literal list, we emit an event that can optionally be ignored, kinda like how you can trap arithmetic overflow in assembly?

I'm imagining it being explicitly named inside each operation:

[
  [descend 'a', {pick up item 0}],
  [descend 'b', {drop item 0}],
  [descend 'lostandfound', {push to this list on conflict, marked with this metadata}]
]

... Which is a bit wordy, but it means the OT code doesn't have to make any assumptions about how your data is structured. Of course, if you always put your lost and found data in the same place, you could just insert the lost and found rules into every operation on the client, before calling transform. Another option is to store the lost and found path with the document snapsnot - though that'd make the whole thing less pure in a sense. Or I could just add another argument to transform?

The nice thing about that is that If there's no lost&found path specified, the conflicting item gets discarded.

You're welcome to assemble such emitted/trapped events into a list of lost-and-found, moved-but-failed values, but you're also welcome to just ignore them. (Contrast if a list was pushed to, if you ignore that list you get a memory leak.)

Nooo.... I don't like that at all.

The nice thing about a lostandfound list is that the changes will resolve in the same way on every peer. So, every peer will end up with the same item in the same place in the lost and found list. The stream would have the same property - it would emit the same sequence of items. But I'm not really sure what clients should do with a stream like that - maybe if the system did first-writer-wins, the server could ignore the stream entirely and then on clients the only objects that would pop out of the stream would be items that you displaced. Then the client could re-insert the newly displaced orphan somewhere in the document. The problem is that by the time that happens the server has already accepted the operation. If the cilent sends a conflicting change then gets disconnected / crashes / closes the app, the orphaned object will be unexpectedly lost forever.

---

So how about something like this:

• Do the thing I just said above with the capacity to elect a lost and found list
• Add two functions to the OT code:
  • withLostAndFound(op, path): Sets the lost and found path in op, which is a helper method so if all operations in your app put the lost and found path in the same place you don't have to store that path in every operation over the wire / on disk.
  • getConflicts(op1, op2): Returns a list of all potential (detectable) conflicts between concurrent ops op1 and op2. Each conflict will specify a conflict type, location, and conflicting data. We can put anything we want in here, and its entirely up to the application whether it calls this function at all, and which conflicting edits it cares about. Would that keep you happy @jhurliman ?

[jhurliman]

Yes :)

[laughinghan]

The nice thing about a lostandfound list is that the changes will resolve in the same way on every peer. So, every peer will end up with the same item in the same place in the lost and found list.

Ohhhhh I forgot that such a lost-and-found list would be shared, which is pretty different from my event emitting/trapping idea.

on clients the only objects that would pop out of the stream would be items that you displaced

Yes that's what I was imagining. The client/application would be welcome to reserve (at the application layer, not the OT layer), root.lostandfound as a list, and push emitted/trapped lost-and-found values into that (shared) list. How well/poorly would that work for you use case, @green-coder?

Another option is to store the lost and found path with the document snapsnot - though that'd make the whole thing less pure in a sense.

Agreed 👎

Or I could just add another argument to transform?

The nice thing about that is that If there's no lost&found path specified, the conflicting item gets discarded.

👍 Seems fine to me. The use case I have in mind avoids these conflicts at the application layer (moves are only between arrays), so no reserved path and the path being optional are my main concerns here, though the emitting/trapping still feels more pure to me.

Question: Will the value be discarded if undoing the move doesn't conflict? I.e.:

• doc: {a:1, b:2}
• client 1: move a->x
• client 2: move b->x

If client 1 wins, do they end up with {x:1, b:2} or just {x:1}? I'm worried that like, in order to keep b:2 we'd have to somehow see into the future that there's no forthcoming insert at b, but maybe such a future insert would simply override b:2 so it's no problem?

[green-coder]

I like the approach chosen by @josephg as it is flexible and generic data structure.

I think that "first-writer-wins" used in addition with the "lost-and-found" for the loser of the conflict would work fine for my use case.

[josephg]

@laughinghan:

the emitting/trapping still feels more pure to me.

Yeah I like the emitting / trapping for that reason, but relying on the client to do something in response to a mutation confirmation is unreliable. If the system works by:

1. Client1 sends op1, client2 sends op2
2. Server receives conflicting ops. Transform removes conflicting part of op2.
3. Server sends acknowledgements to client1, client2.
4. Client2 notices that part of its op was removed, resubmits

... Then the question is, what happens if client2 disconnects between steps 2 and 4? Is the document in a consistent state? Have we lost data?

In any case, you can always re-create that behaviour if you want it using the conflict detection function. If you don't configure the lostandfound list, concurrent edits will be deleted. Then when the client finds out about the concurrent edits, run the conflict check in the client and emit an event stream.

As for your question, those operations do conflict. By default, the resulting document would be {x:1}. If you want to keep b under the system I proposed, you'll need to run the conflict checker on the server and reject / trim op2, or set a lostandfound list (so the result would be {x:1, lnf:[2]})

Stuff that would conflict:

• Mutually blackholed moves (a->b.x vs b->a.x)
• Move-into-deleted item (a->b.x vs delete b)
• Drop into the same location (a->x vs b->x)

And we could also detect and report on:

• Moving the same item to different places (a->x vs a->y)
• A drop inside a destination that has moved (a->b.x vs b->c)
• Attempt to move an item that was deleted by op2 (a->x vs delete a)

---

Also in other news, I've gotten back to the codebase. I haven't added the blackhole detection yet, but I've fixed a bunch of bugs. The fuzzer makes it through 280 iterations before crashing now (up from 20 iterations a few days ago).

[green-coder]

@josephg Please let us know if we can help you with the testing or code review.

I personally would like to assist, although I am not familiar with coffee-script so I may need some time to get started.

[josephg]

@green-coder thanks... its all javascript now (I finished converting it a couple of days ago). But ... I'm not sure if there's room for more people to plug away at it. Its the sort of thing that takes me days to ramp up on, and I wrote it.

Do you mind updating the spec based on what we've been talking about above?

[green-coder]

@josephg That seems a little difficult for me at the moment but I can give it a try, that seems like a good start.

I suggest you to create a 'ot-json1' chatroom using https://gitter.im for discussing with people willing to contribute, and put a link to it on your readme file.

[laughinghan]

... Then the question is, what happens if client2 disconnects between steps 2 and 4? Is the document in a consistent state? Have we lost data?

Ohhhh right we can't rely on any one client to do anything, every client/server needs to independently arrive at the same result; but if we emitted the event on every client/server and the application pushed it into the lost-and-found list, there'd be tons of duplicates.

As for your question, those operations do conflict.

I know, what I meant was, undoing the move (which leaves b:2) doesn't conflict with anything. If it later turns out to conflict, it can still be overwritten, right? By which I mean:

• doc: {a:1, b:2}
• client 1: move a->x
• client 2: move b->x
• server resolves conflict with client 1 winning, doc is now {x:1, b:2}
• client 2 hasn't gotten the memo yet, now does set b:3 (doc: {a:1, x:2, b:3})

When client 2 and the server now sync up, they'll resolve to {x:1, b:3}; the b:3 must win over undoing the move (b:2) to avoid the undoing-a-whole-musical-chairs-chain-of-moves scenario. (Remind me why undoing a whole musical chairs chain of moves is bad, again?)

Maybe that's still bad because it won't be added to the lost-and-found list consistently by all clients?

By the way, thanks for taking the time to answer all my questions. I'm excited that you've gotten back into the codebase!

[josephg]

Wait, slow down.

• doc: {a:1, b:2}, version 1 (tracking versions is the responsibility of the caller)
• client 1: move a->x, version 1
• client 2: move b->x, version 1
• Server receives client 1's op. Doc now {b:2, x:1}, version 2
• Server receives client 2's op. transform(op2, op1, 'left', {lostandfound:'lnf'}) -> move b->lostandfound.
• doc now {x:1, lnf:[2]}, version: 3

client 2 hasn't gotten the memo yet, now does set b:3 (doc: {a:1, x:2, b:3})

• client sends insert b:3
• Doc is now {b:3, x:1, lnf:[2]}

... Easy.

[josephg]

I don't have a good place to update the status of this project - I keep a project journal, but its not online anywhere.

Anyway, good news: A week or so ago the tests passed, but the fuzzer would get through about 10 iterations before finding bad input. I've been busy fixing behaviour, and its running right now, having passed more than 400 000 iterations successfully. There might still be a bug or two, but the transform function is basically finished now. (Whew, what a ride.)

The ot type itself isn't finished yet though - compose still hasn't been written, and once its written I expect the randomizer will start finding some new bugs. But compose is way easier to write than transform, so its all downhill from here.

I also haven't implemented the conflict checker or lost and found list, but both of those should be reasonably straightforward. (The transform function has probably taken about 1-2 months of time to write over the last 3 years. I expect each of those to take about 1-2 days each. The checker itself will just internally call transform with different arguments, creating & returning a list of conflicts instead of returning the transformed object itself.)

... > 1 million iterations and still going strong ...

[green-coder]

Hi @josephg, I tried hard to find a way to contribute to the project and modify the spec.md file, but I found it difficult to impersonate your ideas, experience and writing style without fearing to be somehow wrong.

Also, a few details are bothering me as I would personally do things in a different way, developing first a reference version where algorithm complexity would not matter at all, make it work in the whole system, and then optimize the algorithm. One of the difficulties you have with json1 is that you made yourself facing all the problems at the same time : usability issues, conflict resolution policies, implementation, validation, optimizations, Coffeescript<->JS waltz.

I am sure that there are people like me who would love to contribute, but they also have to face all of that at the same time, that's a pretty steep learning curve. I believe that it is the reason why much of the activity happen inside the issue tracking conversation rather than within PRs. I hope you can finish the project, I am sure people will be ready when it will be the time to test it.

[josephg]

Also, a few details are bothering me as I would personally do things in a different way, developing first a reference version where algorithm complexity would not matter at all, make it work in the whole system, and then optimize the algorithm.

Well thats what I'm doing. The code at the moment is a bit of a mess - there's a lot of code duplication and code written out explicitly that would be better off tidied away behind some simple abstractions. But the fuzzer is a great teacher - it keeps showing me new test cases I didn't consider. Many of the abstractions I have come up with have turned out to be wrong, or badly designed. I've also had to adjust some of my test cases. Once the code is working I want to do a cleanup pass - which is much easier once I know the complete set of behaviour. (As it is each new failing test case found by the fuzzer gets simplified and added to the standard set of tests)

Progress update: I got compose working - it took about a day to write. Up until this point the fuzzer was only generating simple operations (usually just one edit). Now that compose is working the fuzzer is finding some new bugs in transform that I hadn't seen before.d

My current sticking point is bugs of this form:

• op1: Move a->x, insert x[0] = 5
• op2: Move a->y

Expected result of transform(op1, op2, 'left' / 'right')

• left: Move y->x, insert x[0] = 5
• right: Insert y[0] = 5

... Which require adding some more tracking code. But I've been pulled off this project for now to do things which result in my rent being paid. I hope to get back and finish it soon. Its so close!

[logixworx]

What is the status of this project as of today? Last update was 5 months ago

[josephg]

Awesome :)

Since you're all keen.. I want some feedback on something. Sometimes two operations try to mutually destroy one another's contents. Eg, op1 moves doc.a -> doc.b.x and op2 moves doc.b -> doc.a.x.

There's two ways transform could respond:

• Option 1: Detect this and make the operations delete the contents of both doc.a and doc.b
• Option 2: Detect this and throw an exception when it happens

Note that its a pretty rare thing to happen by accident. If we throw, lots of applications won't expect an exception and that could cause problems. But silently deleting user data is generally a Really Bad Thing and in my own applications I'll probably want this behaviour. (I'm considering making transform optionally throw on all instances of deleted user data.)

My question is this: I'm going to implement option 2. But is it worth also implementing option 1? Recovering here is tricky, and I'm not sure if its worth doing the work.

[joeleaver]

I think Option 2 is correct. Nothing should delete that isn't explicitly a delete.

…

On Sat, Dec 1, 2018, 4:07 PM Seph Gentle ***@***.*** wrote: Awesome :) Since you're all keen.. I want some feedback on something. Sometimes two operations try to mutually destroy one another's contents. Eg, op1 moves doc.a -> doc.b.x and op2 moves doc.b -> doc.a.x. There's two reasonable responses that transform could have: - *Option 1*: Detect this and make the operations delete the contents of both doc.a and doc.b - *Option 2*: Detect this and throw an exception when it happens Note that its a pretty rare thing to happen by accident. If we throw, lots of applications won't expect an exception and that could cause problems. But silently deleting user data is generally a Really Bad Thing and in my own applications I'll probably want this behaviour. (I'm considering making transform optionally throw on *all* instances of deleted user data.) My question is this: I'm going to implement option 2. But is it worth also implementing option 1? Recovering here is tricky, and I'm not sure if its worth doing the work. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEkz4C_J94rDu97yzVyXGtzkdoKD6HIks5u0xmzgaJpZM4JKnB_> .

[michael-brade]

Yeah, I agree, Option 2 seems to be correct. Actually, I don't quite understand why this scenario should result in a complete delete (technically, yes, but not intentionally). Apparently the authors of those operations just wanted to move their stuff to another node and created a conflict. Since with trees actual unresolvable conflicts are possible (which is not the case for OT on a string/list), an exception seems ok to me.

[josephg]

OT and CRDT traditionally are conflict-free. Thats part of the point of them. But yeah; I think thats probably the wrong baggage to carry here. Mostly conflict free seems more correct.

Deleting arguably makes sense in some situations - like, if I move a file into a directory at the same time as you delete the directory, its reasonable that my file gets deleted too. Although I can also see the argument that this should generate a conflict instead. And if the operations are live user interactions, deleting is not a big deal because you'll see the delete immediately. In a 3d modelling program, I add a primitive to a car and you delete the car. The primitive gets deleted too. So long as its clear to me that you deleted the car I was working on, thats not a big correctness problem.

I've been thinking about the APIs internally. I've been trying to avoid transform having different modes because of the complexity burden. I think I know how I want to do it now. I think I'm going to change how the transform function itself works so it either returns {ok:true, result:<transformed output>} or {ok:false, conflicts:[... {path:[...], conflictType:'insertInRemovedTree' / 'blackhole' / ...}]}.

Then the standard transform function will call that and throw if there are conflicts. And I can also add a no-conflict wrapper which calls transformRaw, and if there are errors it would delete everything at all the paths which conflict and try again.

[michael-brade]

I agree with your examples, but in those cases that you mention, one operation is a true "delete", so there is no surprise and I wouldn't want a conflict or an exception here. In case of two conflicting moves, I'm not so sure I'd call that "delete"...

I've been thinking about the APIs internally. I've been trying to avoid transform having different modes because of the complexity burden. I think I know how I want to do it now. I think I'm going to change how the transform function itself works so it either returns {ok:true, result:<transformed output>} or {ok:false, conflicts:[... {path:[...], conflictType:'insertInRemovedTree' / 'blackhole' / ...}]}.

Then the standard transform function will call that and throw if there are conflicts. And I can also add a no-conflict wrapper which calls transformRaw, and if there are errors it would delete everything at all the paths which conflict and try again.

YES!! This is awesome. I really like this idea :)

[josephg]

Yes true! The different times we get conflicts are:

• op1 moves or inserts into a location that has been deleted by op2
• op1 and op2 both try to insert different values at the same place in an object
• blackhole - op1 and op2 try to move objects inside one another
• An embedded edit throws / generates a conflict in the same way

I'm probably forgetting a few cases - I'll make a proper list as I write the code.

There's also some things that I think shouldn't conflict. (What do you think?)

• op1 moves something out of a location deleted by op2. The object you're moving out has already been deleted, and I think op2's intent is clearly to delete all of op2.
• op1 moves or edits something entirely within an object that was deleted by op2. Again, op2 deleted it so your edits turn into noops.

[michael-brade]

There's also some things that I think shouldn't conflict. (What do you think?)

• op1 moves something out of a location deleted by op2. The object you're moving out has already been deleted, and I think op2's intent is clearly to delete all of op2.

Hm. This one is not so obvious...

What if op1 (moving b out of x.a) happens 10 minutes before another user submits op2 (deleting x.a)? Then we should be able to move out of the (not yet deleted) x.a, right? But in OT we don't have timestamps, only versions, right? So we don't know how long the delete happened before the move.

But: even if both operations happen at the same time, or the other way around, don't you think that the intention of op2 is fulfilled by deleting the child a of x, irregardless of someone else taking something out of a and moving it elsewhere?

I'd say it this way: op2's intention is to not have a be a child of x. op2's intention is not to destroy a. (Well, it might be, but we don't know, do we? And I would interpret ops in a way that is least destructive)

So yes, this case shouldn't conflict, but neither should op1 become a noop.

• op1 moves or edits something entirely within an object that was deleted by op2. Again, op2 deleted it so your edits turn into noops.

This one is easy: yes.

[josephg]

I'm most of the way through implementing this now - I'm just finishing up fixing some tests.

At the moment the API is looking like this:

• type.tryTransform(op1, op2, side) -> {ok:true, result} or {ok:false, conflict:{...}
• type.transform(op1, op2, side) -> result. Just a wrapper for tryTransform that throws if a conflict happens
• type.transformNoConflict(op1, op2, side) -> result. Always recovers from conflicts, just like the old behaviour. May delete user data in the process.

Its a bit all or nothing like this. Given that I think most of the time you'll want to auto-recover from some conflicts but not others I might add a recoverFromConflicts:... option or something.

I've decided to make an invariant with conflicts that if transform(op1, op2, 'left') generates an error, transform(op2, op1, 'right') must generate the same error.

The current implementation has 3 different conflicts:

• RM_UNEXPECTED_CONTENT: One operation removed an object and the second operation inserted or moved something into the object that was removed, or edited something inside the removed content. Eg rm x vs ins x.a = 5 or rm x vs some embedded string op
• DROP_COLLISION: Two drops (or two inserts, or a drop and an insert) tried to insert into the same location. Eg: ins x = 5 vs ins x = 6. We don't generate a conflict if the inserts are identical.
• BLACKHOLE: The operations made an object get unparented, and instead contain itself. Eg: mv x -> y.a vs mv y -> x.a

These things don't currently conflict:

1. op1 moves an object that was removed by op2

Eg: mv x -> y vs rm x

I'm reasonably happy with 1. @michael-brade:

op2's intention is not to destroy a. (Well, it might be, but we don't know, do we? And I would interpret ops in a way that is least destructive)

We sort of do know. From op2's point of view, the document contained the x.a property when they submitted their rm operation. So I think the intent is to remove x.a, and I feel comfortable safely preserving that.

2. op1 and op2 both move the same object to different locations

Eg: mv x -> y vs mv x -> z

This is an interesting case. This is really easy for transform to detect, and there are probably cases where users will want to disallow this. Given we have all the rest of the conflict infrastructure, I'm tempted to add a conflict for this.

The API I'm expecting people to use will be something like transform(op1, op2, 'left', {autoRecoverFrom: [RM_UNEXPECTED_CONTENT, DROP_COLLISION]}). So I'd rather add this now if we decide its important. But I don't know! Thoughts? Can anyone think of any use cases where you'd explicitly want this to conflict?

3. op2 does an embedded edit on an object that op1 removed

EDIT: I've implemented this and rolled it into RM_UNEXPECTED_CONTENT.

Eg: at x, make embedded rich text edit {...} vs rm x

This one is killing me. This will come up if, for example, Alice is typing into a field and Bob deletes the entire row. The standard way to handle this is to just delete the embedded operation too. But the problem with that is that if the embedded operation inserts a whole bunch of user content, we'll be deleting their new content too. I mean, this is exactly what the RM_UNEXPECTED_CONTENT error is for.

I think I might make a conflict for this too - although you'll generally want to auto recover from this.

[dcworldwide]

This is such an important body of work thankyou. When I review the code...its mind boggling tbh to wrap your jead around. This is a clear case where typescript would add much needed self documentation, rigor / compiler checks and refactoring support

[josephg]

^_^

I've thought about porting the code to typescript. I'm working on another project in typescript and I'm quite fond of typescript in general. But surprisingly, I don't think typescript would add much here. Most correctness bugs in this library are logic bugs rather than typing bugs. And I can't think of any bugs that the typescript compiler would find that won't get picked up by the now 200+ tests + fuzzer.

I also don't expect to do much refactoring at this point. There'll be a cleanup pass to trim out all the debugging information and rename some variables but hopefully that should be about it.

Types would definitely be useful for consumers to help create & interact with operations via cursors and other utility methods. But I'm really not sure what else they'd get us.

[michael-brade]

These things don't currently conflict:

1. op1 moves an object that was removed by op2

Eg: mv x -> y vs rm x

I'm reasonably happy with 1. @michael-brade:

op2's intention is not to destroy a. (Well, it might be, but we don't know, do we? And I would interpret ops in a way that is least destructive)

We sort of do know. From op2's point of view, the document contained the x.a property when they submitted their rm operation. So I think the intent is to remove x.a, and I feel comfortable safely preserving that.

Judging from the way you quoted me, I think you may have misunderstood what I meant. The intent certainly is to remove x.a, but the question remains if the intent also includes to completely get rid of any part of x.a from anywhere else in the tree, especially if someone else just tried to move a part of it somewhere else.

I can't actually think of a common example (which may mean that you are right), but here is a possible example: say you have an address book database or a user management database. x.a means: x is the database, a is user Alice. Alice has a name, an address, photo, etc. x.b is Bob, who lives in the same house and has been created in the database recently. Now one operation means to delete Alice (x.a) from the database/address book (b/c she doesn't work at the company anymore or whatever), and another operation wants to move her address (x.a.s) to Bob (x.b) before deleting Alice so not to have to retype the same address again.

As I said, maybe not the most realistic example... but it should explain what I meant.

As for points 2 and 3, you surprised me here:

2. op1 and op2 both move the same object to different locations
   [...] Can anyone think of any use cases where you'd explicitly want this to conflict?

Always? I cannot even think of a single use case where it shouldn't conflict... I mean, what would the
automatic transform do in this case?

3. op2 does an embedded edit on an object that op1 removed

Yup, that means we'd lose op2's data. But no surprise here for me. If you still want a conflict to be safe, then to be consistent I would also expect at least a conflict in case 1 (op2 moves part of an object that op1 removed).

[josephg]

Now one operation means to delete Alice (x.a) from the database/address book (b/c she doesn't work at the company anymore or whatever), and another operation wants to move her address (x.a.s) to Bob (x.b) before deleting Alice so not to have to retype the same address again.

Hm, I hear what you're saying. I guess the way I see it is that if you want to save Alice's address, you should move it out of her record before deleting her information from the database. Once you delete her, the data is gone regardless of what subsequent operations are submitted. I think thats a reasonable restriction.

op1 and op2 both move the same object to different locations [...]
Can anyone think of any use cases where you'd explicitly want this to conflict?

Always? I cannot even think of a single use case where it shouldn't conflict... I mean, what would the
automatic transform do in this case?

😂

Well, we pick one of the two move destinations in an arbitrary but consistent way and move the object there. In this case, only the op that is considered the 'left' op will have its move honored. The data you've moved hasn't been lost or anything - you can always move it back if that was the wrong choice. And you have the other operation, so you have all the information you need to figure out where it went.

There is some weirdness with this behaviour if the operations are also configuring the object. Consider:

• doc: {people: {alice: {}}, employees:{}, contractors:{} }
• op1: move people.alice -> employees.alice, insert employees.alice.employeeID
• op2: move people.alice -> contractors.alice, insert contractor.alice.contractorID

By the rule of "one of those moves wins", alice will end up either in employees or contractors. But wherever she ends up, she'll be given both an employee id and a contractor id. Thats definitely strange.

But yeah; if its obvious to you that this should conflict I'll just add a conflict for it. Its a pretty easy thing to detect & recover from in any case.

Meanwhile, I think the API is going to end up as something like this:

const json1 = require('json1')
const type = json1.autoRecoverFromConflicts(conflict => {
  // Automatically recover from all conflicts except blackholes (or whatever logic you want)
  return (conflict.type !== json1.CONFLICT_BLACKHOLE)
})

// ...

// Ok because drop collisions are allowed by the predicate above
type.transform(['x', {i:5}], ['x', {i:6}], 'left')

// But this throws:
type.transform([['a', 'b', {p:0}], ['b', {d:0}]], [['a', {d:0}], ['b', 'a', {p:0}]], 'left')
// -> TransformConflictError type=BLACKHOLE

[michael-brade]

Once you delete her, the data is gone regardless of what subsequent operations are submitted. I think thats a reasonable restriction.

Fair enough. I think I agree 😄

[move same object to different locations]
But yeah; if its obvious to you that this should conflict I'll just add a conflict for it. Its a pretty easy thing to detect & recover from in any case.

Cool! I think it makes sense to at least have the option. Even better if it is easy to recover from.

Meanwhile, I think the API is going to end up as something like this:

const json1 = require('json1')
const type = json1.autoRecoverFromConflicts(conflict => {
  // Automatically recover from all conflicts except blackholes (or whatever logic you want)
  return (conflict.type !== json1.CONFLICT_BLACKHOLE)
})

// ...

// Ok because drop collisions are allowed by the predicate above
type.transform(['x', {i:5}], ['x', {i:6}], 'left')

// But this throws:
type.transform([['a', 'b', {p:0}], ['b', {d:0}]], [['a', {d:0}], ['b', 'a', {p:0}]], 'left')
// -> TransformConflictError type=BLACKHOLE

Great! Looks good to me! 👍

[josephg]

Small update: the code is pretty much correct now. The fuzzer gets up to about 2 million iterations before finding some invalid input. The failure case it found is legit, but we're well into the long tail of obscure bugs that (I hope) you'd have to get very unlucky to run into in practice.

At this point the code is usable, although there's a chance I might change some of the output slightly. I'd like to throw a preview release up on npm so we (certainly I) can start using it and messing around with it. It'd also be nice to launch that release with a function to modify a path by an operation. Thats super useful in practice, and I'll want that almost immediately when I'm actually using the code.

I also think porting it to typescript at some point is a good idea. Once its all correct I'd like to do a big cleanup pass. There's plenty of small pieces in there that could do with some spring cleaning, and moving the code to TS at the same time might be a good call. (And I'd like to port the tests away from coffeescript too!)

Other changes:

• I've made a text-unicode type, which is the same as the older text type except characters are counted using unicode code points rather than utf16 offsets. This makes it easier to interoperate with to other languages, although there's a performance cost in JS because native strings are utf16. Its also now written in typescript. Embedded json1 string edits use this type instead of the old text type.
• I did a tiny bit of bundle cleanup. The bundle size is now 23k minified, or 8k gzipped. It could be reduced further, but that seems acceptable.

I had a good look at how I can implement the MOVED_SAME_OBJECT conflict we were talking about above. Its way harder than I expected. For other conflicts, I recover from the conflict by modifying the operations then attempting to transform again. This is correct in all cases.

For example:

• op1: insert doc.a = "hi"
• op2: insert doc.a = "yo"
• result: insert conflict at doc.a
• left automatic recovery: op2 = op2 + delete doc.a
• right automatic recovery: op1 = op1 + delete doc.a

... This results in a consistent output for all input, while also letting the user modify the operation and resubmit something else. If you're curious, the code itself is here. Its pretty straight forward.

But the question then is, how could I modify operations to recover when both ops move the same object? The problem is that there's no safe intermediate place to move the object to. The edge cases to consider are these:

• a->b vs a->c
• a.b->b, del a vs a.b->c
• a.b->b, del a vs a.b->a.c
• a->b.b vs a->c, del b
• a.b, insert c=5 vs a->c, insert b=5

I'm sure there's a way to do it, but the answer might be that my nice neat abstraction gets broken in the process. The code currently recovers from these issues just fine - maybe I'll end up passing a whitelist of allowed mutual moves into the transform function or something. It wouldn't be the first special case. Not by a long shot.

[michael-brade]

Great, the first part is really great :-)

About the second part: I have to admit, I don't understand it completely. Two things:

• when you write left automatic recovery: op2 = op2 + delete doc.a, does that mean this will end up doing insert doc.a = "hi"; insert doc.a = "yo"; delete doc.a? I.e., op1, recover(op2)? Doesn't seem correct to me, so I must have misunderstood something.

• why do you need an intermediate place to move the object to? I am assuming with "the object" you are referring to a in your first example, for instance. So when recovering, you want to transform the operation, not apply it yet, or do you? (Yes, I don't know yet how tryTransform works :-p)

[josephg]

op1 is transformed into recovery2 + transform(op1+recovery1, op2+recovery2). In this case recovery1 is null and recovery2 is delete doc.a. In the insert doc.a='hi' vs insert doc.a='yo' case when transforming left, recovery1=null and recovery2=delete doc.a. So transform resolves to:

delete doc.a + transform(insert doc.a = 'hi', insert doc.a = 'yo' + delete doc.a)
= delete doc.a + transform(insert doc.a = 'hi', null)
= delete doc.a + insert doc.a = 'hi'

And that is the correct result.

Can we use the same approach for mutual move conflicts? I don't think so. Consider a->b vs a->c. The obvious recovery for left operations is to cancel op2's move (recovery2 = c->a) and for right operations to cancel our own move (recovery1 = b->a). You can run through the construction above to verify you get the right result (op1 becomes c->b and op2 becomes null).

But this doesn't work for all of the edge cases I listed above. Consider a->b vs a->c, insert a=5. The desired result from transform(op1, op2, left) is c->b. But op2+recovery2 becomes a->c, insert a=5 + c->a, which is invalid. You can't insert a=5 when there is already content there. (And if you did it anyway, transform would return the wrong result).

But I think thats ok. The answer is probably just to abandon this recovery trick for mutual moves, and instead detect it and throw (or whatever) before calling the internal transform function. And then if its marked as allowed, we let the existing transform code handle the complexity of resolving this sort of thing.

And it is complex - for example, consider transform(a->a vs a->b, insert a=6, left). In this case, even though op1 seems to be a noop, the rule that op1's moves override op2's moves still comes into effect. So op2's a->b move is cancelled by op1 and a ends up colliding with op2's insert. In the end, this should generate a MUTUAL_MOVE conflict at a followed by a DROP_COLLISION conflict with a->a vs insert a=6. Transform already deals with this case correctly in every way except it doesn't flag the move conflict. So yeah, if we flag the conflict before calling transform then I think we're golden.

---

Anyway, sorry that was probably a longer explanation than you expected. That probably wasn't super useful for you, but it was useful for me to talk about this stuff to get my thoughts straight.

In other news:

• I have a working conversion function from the old json0 operations to json1, but there's a few caveats. First, I tried to port the json0 test suite with the new code but it turns out that convert(json0.transform(op1, op2)) != json1.transform(convert(op1), convert(op2)) in all cases. So I'm going to abandon the old test suite and mostly just use a simple fuzzer to verify correctness. This conversion function will end up in a separate package to minimise bundle bloat. I'm not writing a converter back from json1 operations to json0 operations because not all operations can be converted, and also because its actually a much harder problem than it sounds to decompose an operation into a list of parts. Also there's a bit of complexity around text operations and unicode, because json0 and json1 don't count the length of characters the same way. I'll expand on all of that in the conversion project's readme when I put it up on github.
• I got the fuzzer happily running past 10M operations (which took nearly 4 hours to check). But then I realised my genRandomOp function was never generating composite operations. When I fixed that it crashed at iteration 8. Anyway, I fixed bunch of small bugs in compose, and now its back up to ~500k iterations before crashing with a crazy complicated blackhole vs move issue. Working on this is like a combination of yak shaving and whack-a-mole. And I suspect that by the time the code is actually bug free I'll need a few weeks of CPU time to convince myself of the fact.
• The only other piece of work left will be to write a function that modifies a path by an operation and then it'll be done.

[michael-brade]

Oh yes, that was super useful! This is a really good explanation. I just don't know where the formula to transform op1 comes from: recovery2 + transform(op1+recovery1, op2+recovery2) but it is quite elegant :)

But maybe here is a typo:

But this doesn't work for all of the edge cases I listed above. Consider a->b vs a->c, insert a=5. The desired result from transform(op1, op2, left) is c->b. But op2+recovery2 becomes a->c, insert a=5 + c->a, which is invalid. You can't insert a=5 when there is already content there.

I guess you meant: The desired result from transform(op1, op2, left) is a->b, insert a=5? Because the left op wins the conflict, then insert a. If not, you lost me here. And then: why is recovery2 insert a=5 + c->a and not delete a + c->a?

Also, I don't think a conversion from json1 to json0 is neccessary. Who would want to use it anyway?

Just out of curiosity: Am I right to assume that your fuzzer is using a fixed "random" sequence so that it is always reproducible? It is an amazing tool and a great way to replace monkeys having to type for 100 years or more :-D And yes, I know that feeling about yak shaving and whack-a-mole -- but I love it because it involves a limited complicated problem that fits in my head and I can spend a long time thinking about it. Much more fulfilling than being dragged in this direction and that direction just to get something sort of trivial to run.

[josephg]

I guess you meant: The desired result from transform(op1, op2, left) is a->b, insert a=5 ? Because the left op wins the conflict, then insert a. If not, you lost me here. And then: why is recovery2 insert a=5 + c->a and not delete a + c->a?

No thats not how transform works. Transform returns whatever op1 would be if it was applied after op2. So consider the simple case of op1:a->b vs op2:a->c (left). The desired result is c->b because a has already been moved to c. So we need to move it back to where we want it to go.

So now consider again op1:a->b vs op2:a->c, insert a=5. Lets imagine the document before any of these operations have happened is {a:1}. After op2 has applied, the document is {a:5, c:1}. The question transform answers is, what should the a->b operation look like if its creator knew about op2 when creating this operation? And the answer is that their intent was to move the thing that used to be a to be at b. So since that thing is at c now, transform returns c->b. It doesn't matter that op2 inserted something else at a. (But unfortunately, that insert at a does ruin my beautiful little recovery trick)

I guess you meant: The desired result from transform(op1, op2, left) is a->b, insert a=5?

Transform just returns what op1 would look like if it were applied after op2. If you want a combined operation, you can make it either by calling op2 + transform(op1, op2, left) or op1 + transform(op2, op1, right) (+ is the compose function). A correct OT system must return the same thing in both cases, and this is one of the core invariants that the fuzzer checks each iteration.

In this case:

• op2 + transform(op1, op2, left) = (a->c, insert a=5) + c->b = a->b, insert a=5.
• op1 + transform(op2, op1, right) = a->b + transform((a->c, insert a=5), a->b, right) = a->b + (insert a=5). (Since we're right, transform lets the other move win and we discard the a->c move).

Also, I don't think a conversion from json1 to json0 is neccessary. Who would want to use it anyway?

Anyone using json0 who wants to migrate to the new json1 code. For example, everyone using sharedb.

[michael-brade]

Transform returns whatever op1 would be if it was applied after op2.

Ah, of course! How could I have missed that, now it all makes sense. Which really means: tough luck for your little recovery trick 😂

But really, maybe there is a way... If I understand the code correctly, recovery1 and recovery2 are what resolveConflict(...)[x] returns for x==0 and x==1, respectively. I.e., you can choose/code anything you want for those in resolveConflict. So when you wrote:

But this doesn't work for all of the edge cases I listed above. Consider a->b vs a->c, insert a=5. The desired result from transform(op1, op2, left) is c->b. But op2+recovery2 becomes a->c, insert a=5 + c->a, which is invalid.

Why not fix the recovery2 to be a composite recovery: delete a, c->a. Wouldn't that work?

Also, I don't think a conversion from json1 to json0 is neccessary. Who would want to use it anyway?

Anyone using json0 who wants to migrate to the new json1 code. For example, everyone using sharedb.

No no, this time you misunderstood 😁 I meant the other way around, from new (json1) to old (json0). We need old to new to use an existing sharedb with json1, but what would be the use for new to old? You said you won't do it, but I was wondering why you even considered it...

[michael-brade]

One more question: Semantically, what does left or right mean in the third arg (side) of transform? I thought it determines which operation wins in case of a conflict, is that correct? If so, what decides if it is right or left?

EDIT: hm... I saw that left/right also exists in text OT. So I'm sure my assumption is not the whole picture.

[josephg]

Why not fix the recovery2 to be a composite recovery: delete a, c->a. Wouldn't that work?

Well, we don't actually want to delete anything in the output. No information is lost when we resolve this conflict. And if we lie about what op2 looks like to transform like this, it'll break other stuff. In this case, if we were dealing with a list instead of an object, any later list indexes will end up wrong in the transformed result because they wouldn't be modified by op2's insert at a (well, at whatever list index that correlates to).

I could probably come up with an example test case demonstrating this, but I'm not going to.

One more question: Semantically, what does left or right mean

Its a tag for breaking symmetry. Eg, imagine if two operations both insert at the same index in a list. It doesn't matter which insert ends up first in the output and which ends up second, but whichever way around they end up, all peers need to make the same decision otherwise they won't end up with the same document afterwards. Transform's caller is responsible for tagging one operation as 'right' and one as 'left' in a consistent way. In google wave and stuff I've worked on, the left op is whichever one reached the server first. In other systems I've heard about people using unique client IDs and 'left' is the op generated by the client with the lower id. It doesn't matter so long as the decision is made consistently. If all peers don't break ties the same way the system can't converge.

[josephg]

I've fixed some more bugs, added an initial pass of a applyPath(path, operation) function and published 0.1 preview to npm! 🎉

I'll probably close this issue soon. Normal bugs should be filed as regular issues here and we can address them as they come up.

I've also updated the readme describing some of the things that still need work. It'd also be great to get some usage examples up for people to play with.

[EvanSchalton]

When I realised all that I put it down until I have more time to come at it fresh. If someone paid me to work on it it might be a different story.

Are you interested in consulting on some OT work?

[jacwright]

@EvanSchalton haha, I was wondering the same thing. What are you working on?

@josephg looking for OT consulting? You may have a couple options.

[EvanSchalton]

@jacwright I'm more or less building a serverless version of sharedb in typescript

[jacwright]

@EvanSchalton I think that's a great idea. CKEditor is providing something like that for rich text documents. A service that provided those features for text as well as objects & arrays etc. could be really cool.

[josephg]

@jacwright @EvanSchalton Not looking for consulting work at the moment, unless its aligned with my current plans. But feel free to shoot me an email if you want to get in touch. (My email address is in all my git commits.)

[EvanSchalton]

@jacwright yeah -- I don't like the pay-per-user model; it doesn't align well with client priorities. When it comes to collaboration tools, IMO, they're more effective the more people can collaborate, and inversely -- not effective if people aren't collaborating. I've seen too many failed pilots because the cost structure (per user) caused the piloting company to select a smaller initial user pool -- resulting in diminished returns and ultimately a decommissioning of the project.

I think per-user pricing is a vestige of a bygone era -- you used to have to worry about server capacity so you effectively had to charge customers for the potential of their usage, but now we have easily scalable cloud compute -- especially with serverless architecture -- that frees me up to be more creative with my pricing strategy, unfortunately that means I also can't entertain vendors who have per user pricing models.

It seems like we're on similar bents; I'd love to connect on LinkedIn -- bounce ideas off each other as we implement

@josephg thanks, will do!

[jacwright]

@EvanSchalton I just meant a service like that would be great. Pricing model is whatever.

…

On Thu, Feb 4, 2021 at 9:41 PM Evan Schalton ***@***.***> wrote: @jacwright <https://github.com/jacwright> yeah -- I don't like the pay-per-user model; it doesn't align well with client priorities. When it comes to collaboration tools, IMO, they're more effective the more people can collaborate, and inversely -- not effective if people aren't collaborating. I've seen too many failed pilots because the cost structure (per user) caused the piloting company to select a smaller initial user pool -- resulting in diminished returns and ultimately a decommissioning of the project. I think per-user pricing is a vestige of a bygone era -- you used to have to worry about server capacity so you effectively had to charge customers for the potential of their usage, but now we have easily scalable cloud compute -- especially with serverless architecture -- that frees me up to be more creative with my pricing strategy, unfortunately that means I also can't entertain vendors who have per user pricing models. It seems like we're on similar bents; I'd love to connect on LinkedIn <https://www.linkedin.com/in/evanschalton/> -- bounce ideas off each other as we implement @josephg <https://github.com/josephg> thanks, will do! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA5LZX4GWT64RPLLOEK733S5NZHHANCNFSM4CJKOB7Q> .