Preview: Automerge binary data format

[ept]

As we all know, Automerge's current performance on large documents is terrible (#89) — loading them is very slow, the data files on disk are huge, and they use huge amounts of memory and network bandwidth. Improving this situation is one of my top priorities. However, most of the low-hanging fruit had already been picked, and a more fundamental rethink of Automerge's data structures was needed.

It turns out that CRDTs are very easy to implement badly, but actually quite difficult to make fast and efficient. A lot of this is due to the metadata overhead: for example, in a text document, every single character needs a unique identifier. Various schemes have been designed to reduce this metadata cost, but some of them behave badly under concurrent insertion. Others rely on periodic cleanup operations, but the cleanup depends on communicating with all nodes, and it stops if some of the nodes are offline for a long time.

Almost a year ago I first wrote up a proposal for a compressed binary encoding format for Automerge, taking a new approach that I've not yet seen in the CRDT research literature. Borrowing ideas from column-oriented database systems, this is in the first instance a way of saving disk space and network bandwidth, but it also paves the way towards some big potential performance improvements (and it doesn't suffer from the aforementioned problems).

I have been working on this project on and off since, and while it's not yet finished, I wanted to share an update on where we're at and where we're heading. I am excited about the progress so far and I'm pretty sure this is the right path for the future.

This work has been happening on the performance branch. My intention is to stabilise the features and data formats on this branch over the coming weeks, and to then turn it into a release candidate for ✨Automerge 1.0✨. There are a bunch of compatibility-breaking changes on this branch, both in the APIs and the data formats, and I want to wrap up all of these breaking changes into a single bump of the major version number, so as to minimise the number of further breaking changes in the future. There will be a migration tool for Automerge 0.* users to convert their existing documents to the Automerge 1.0 format, and the new data formats are carefully designed with extensibility in mind, allowing us to add new features to Automerge while retaining interoperability between clients running different versions of Automerge.

Automerge.getChanges() and Automerge.save() now return Uint8Array objects using the binary data format. The Automerge frontend APIs are largely unchanged (except for a few tweaks as documented in the CHANGELOG). The communication protocol between frontend and backend has changed a lot (fixing some historical design mistakes, and allowing the frontend to be simplified, which should make it easier to create frontends in other languages).

There are also some changes of the "might as well also do this now" sort:

• Every change now has a millisecond-resolution timestamp of when it was made. This timestamp has no bearing on the conflict resolution; it exists only to enable better inspection of the editing history (allowing you to see when a change was made). I'm not sure why we didn't add a timestamp from the beginning — it seems like quite an obvious omission in retrospect.
• Every change can be identified by the SHA-256 hash of its binary encoding, and this hash is now used instead of actorId+sequence number (vector clocks) to express dependencies between changes — much like in Git or blockchains (see discussion in Hash-based integrity checking of operation sets #27 and Using hash chaining to encode the dependency graph #200). ActorIds still exist, because using hashes for everything totally destroys compression (see below).

The switch to using hash chaining will have a bearing on network protocols that sync up Automerge replicas, such as Automerge.Connection, which currently relies on vector clocks. This is something I would like to discuss with the community. By the way, I have split Automerge.Connection into a separate repository, because I think we might want to evolve it and assign it version numbers independently from the Automerge core.

Anyway, time to report some numbers on how the binary format compares to Automerge's existing JSON data format. First of all, let me emphasise that I have not yet solved the performance problems on this branch (despite the branch name "performance") — so far I have primarily worked on making the binary format compact. Therefore the analysis below focusses mostly on encoded data sizes. Also, I will use text editing examples, since text editing tends to accumulate changes very quickly (one change per keystroke), so it's a challenging type of workload. Nevertheless, the binary format should be good for all types of data supported by Automerge.

The binary-encoded length of a typical single-character text insertion is 105 bytes. Sounds like a lot, but only 35 of those bytes are the encoding of the actual operation. The rest is made up of the hash of the previous change (33 bytes), the actorId (17 bytes), the timestamp (6 bytes), a checksum (4 bytes), and miscellaneous other header fields (10 bytes). It might be possible to squeeze that a little further, but with diminishing returns. A single-character deletion takes 109 bytes. There is no point gzipping these short byte sequences, as doing so actually increases the size.

	Current JSON format	JSON + gzip	New binary format	Binary + gzip
Single-character insertion	280 bytes	187 bytes	105 bytes	138 bytes
Single-character deletion	183 bytes	159 bytes	109 bytes	141 bytes
10,000-character insertion	2,476,915 bytes	105,296 bytes	10,124 bytes	4,380 bytes
10,000-character deletion	1,118,968 bytes	28,386 bytes	133 bytes	151 bytes

The benefits of the binary format show up much more starkly when inserting or deleting longer runs of characters in one go. When inserting 10,000 ASCII characters at once into a text document (a medium-sized copy&paste), the JSON format uses about 248 bytes per inserted character, leading to a change about 2.5 MB in size. On the other hand, the binary format contains the raw ASCII text (10,000 bytes) plus a small, near-constant overhead of 124 bytes for the whole change. Although the JSON data compresses well with gzip, with a 24:1 compression ratio, the gzipped JSON is still 10 times the size of the un-gzipped binary data. The binary data can still be gzipped further for an additional 2.3:1 compression.

The situation is even more extreme for a 10,000 character deletion: the binary encoding is almost constant-size, while the JSON format takes over 100 bytes per deleted character. In this example, the JSON encoding is 8,400 times the size of the binary encoding.

So far for the microbenchmarks. A more interesting question is how the encoding fares with more complex editing patterns. For this, I used a dataset that we captured a few years ago, when we wrote the LaTeX source of an entire paper using a homegrown text editor. The result is an editing history containing 332,702 keystrokes (of which 182,315 are single-character insertions; the rest are single-character deletions and cursor movements). The final text file (without any editing history) is 104,852 bytes in size.

I converted this dataset to the new binary format, with each keystroke as a separate change, and the result is as follows:

	JSON format	JSON + gzip	Binary format	Binary + gzip
Paper (individual changes)	146,406,415 bytes	71,347,241 bytes	51,620,565 bytes	N/A
Paper (whole document)	146,406,415 bytes	6,132,895 bytes	1,119,341 bytes	664,268 bytes

The first row shows what happens when we treat each of the 332,702 changes separately: 440 bytes per change in JSON, and 152 bytes per change for the binary encoding — a bit more than the per-change numbers above, but not wildly so. However, treating each change individually drastically limits the compression we can apply. Applying gzip compression to each JSON change individually yields a compression ratio of only 2:1.

It is much more effective to compress the document as a whole. Simply gzipping the JSON change history as a whole yields a 24:1 compression ratio. But the big deal is the whole-document binary encoding, which is about a megabyte (3 bytes per change), and which further gzips to 664 kB. When working with the whole document, binary encoding + gzip is almost 10 times more compact than JSON + gzip!

I should emphasise that in all of these examples, the compression is lossless: it's not just a snapshot of the latest state, but it preserves the entire character-by-character editing history. In fact, from the whole-document binary format it is possible to reconstruct the bitwise identical change history that created it, recompute all the SHA-256 hashes for the dependency chains between changes, and end up with exactly the same hashes for the "heads" (in Git terminology). Note that hashes need to be recomputed, not stored: merely storing the hash of each of the 332,702 changes would require about 10MB, ten times the size of the binary-encoded file.

The next interesting question is: what makes up those 1,119,341 bytes of binary data? Can we reduce it further? Well, it breaks down as follows:

• 662,388 bytes for the timestamps on each change. Yes, you read that correctly: two thirds of the file is just timestamps! The reason is: timestamps are millisecond resolution, and for each change we store the number of milliseconds since the previous change. When typing, the time between keystrokes is usually a few hundred milliseconds. The binary format encodes integers between 64 and 8,191 using two bytes, so most changes require two bytes to represent their timestamp delta to the previous change. Two bytes times 332,702 changes costs about 664 kB. We could reduce this to one byte per change if we reduce the timestamp resolution to integer seconds instead of milliseconds.
• 182,315 bytes for the ASCII characters of the 182,315 insertion operations. This is just ASCII text, so it further gzips pretty well. This example document only contains ASCII, but obviously the binary format handles all of Unicode using UTF-8.
• 112,726 bytes for the CRDT metadata on the text object. That is about 0.44 bytes per text editing operation — pretty awesome!
• 124,306 bytes for the cursor movement operations (1.2 bytes per operation).
• 36,954 bytes to store how the operations are grouped into changes.
• 652 bytes for miscellaneous headers.

Comparing the whole-document binary encoding (~1 MB without gzip) to the final text document without any CRDT metadata (~100 kB without gzip), we see there is still a substantial cost. But we do gain a very detailed change history, and of course the CRDT merging ability. And we have made big progress compared to the JSON encoding (two orders of magnitude smaller without gzip, one order of magnitude smaller with gzip). We could compress it by another factor of ~2 by reducing timestamps to 1-second resolution and leaving out the cursor movements (they could be maintained in separate transient storage, and omitted from the persistent document history).

So far all of the compression has been lossless. Of course we can also consider approximate forms of compression, such as combining several changes with similar timestamps into a more coarse-grained change (reducing the number of timestamps we have to encode), or discarding some of the change history entirely. However, any sort of lossy compression would mean losing the ability to reconstruct the original change history, and thus losing the ability to check the hash chains. Depending on how we use the hash chains (to check integrity, to verify authenticity?), such a trade-off will need to be considered very carefully.

A key insight from the experiments above is that when dealing with large change histories, it is much more efficient to encode the document as a whole than to encode each change separately (in the example above, it makes a factor of 50 difference). This has implications for network protocols that sync Automerge nodes: for example, Hypermerge is currently based on append-only logs of changes, assuming that we're working with one change at a time.

Future protocols will need to detect how far out of sync the two nodes are: if they are almost in sync, it is more efficient to just send the last few missing changes, as each change is typically a few hundred bytes. But if they are far apart (or if one of the nodes is lacking the document entirely) it is more efficient to send the whole-document encoding. Multiple versions of the whole-document encoding can be merged efficiently, without having to turn the document into a log of changes and back again. Essentially, the CRDT can run in two modes: an operation-based mode (where we send changes over the network) and a state-based mode (where we send the whole-document encoding). Network protocols will need to be adapted to take advantage of this choice, by figuring out when to use which mode.

One last thing about the whole-document encoding: while its primary design goal is to be compact, the close second goal is to allow fast loading of the current state from disk. For example, fetching the current value of a particular field (e.g. the title of a document) should not require decoding the whole file, but only reading a bit of metadata and then seeking to the appropriate place in the file. On some platforms, we might even be able to avoid loading the whole file into memory by memory-mapping it and seeking only to those bits that we need.

To test this, I wrote a hacky experimental decoder that reads the latest text of the LaTeX paper source example (that is, it extracts the 104,852-character final text string from the 1,119,341-byte encoded file). Here are the timings from running that decoder 10 times on my laptop:

12.919ms
6.203ms
55.147ms
1.702ms
1.960ms
4.270ms
8.165ms
5.617ms
1.977ms
1.753ms

I don't know why the running time fluctuates so much (GC maybe?), but a median time of ~5 milliseconds is pretty encouraging. Loading a file with hundreds of thousands of changes would probably take minutes with the current Automerge implementation, if it completes at all.

I will write up a detailed specification of the binary format sometime soon. A lot of thought has gone into it, e.g. designing for future extensibility. For now, I want to put this update out there to share what's happening.

By the way, Orion and Alex have been doing an excellent job of tracking this progress in their Rust port of Automerge. This means interop between the JavaScript and Rust backends is not far off!

[lightsprint09]

For me it is really hard to reason about an optional bool.
A bool should either true or false. If you really need to express three states, I would prefer a enum with named states.

[lightsprint09]

This is not the only place in the type specification where optional bool is used. Maybe we could clean this up if possible

[scotttrinh]

Isn't this a side-effect of writing something in idiomatic JavaScript? Certainly good advice for TypeScript code, but encoding these states as 0, 1, 2 would be pretty opaque to the JavaScript consumer, and using strings would bloat the in-memory cost. In practice, I imagine you'd use a falsy check here, so undefined would be equivalent to false. I assume this is still just two states (falsy, true) rather than three (undefined, false, true) in any meaningful sense.

[ept]

It's not tristate, it's just a boolean that defaults to false when absent. I don't know how to declare that in TypeScript. In 02271f0 I've changed operations such that they always have an insert property that is explicitly true or false.

[pvh]

Hi Martin, I've had some time now to digest all this and give it some thought.

First, on the subject of timestamps, paying for per-millisecond timestamps on all commits is pretty expensive, and it's not clear to me that there is much utility there. I think per-second timestamps would be a quite-reasonable optimization, and I'd further consider making their insertion optional.

Next, as for key movements or other expensive but less valuable data, I've thought quite a bit about "non-dependent actors" as a useful option here. The concept is (in old-automerge lingo) an actor which depends on other actors but is never depended-upon. It could thus track the main document but would not need to be loaded unless it was necessary. Users could provision and synchronize this kind of cursor history during interactive sessions but throw them away afterwards or simply not bother to share them during asynchronous collaboration. It's not clear to me whether this would still be feasible with the new design but it seems in principle like it should be (simply bump the dependency hash as necessary).

As to the future for Automerge Connection I have been aware of this as an upcoming problem for quite some time and have only recently begun giving it serious consideration. The vector clock model is desirable because calculating the necessary synchronization work between two documents is a trivial operation that only needs to examine the clock. On the face of it, calculating a delta between hash-chained commits would be tricky. We'd need to calculate the least-common-ancestor for the peers and then generate a special commit that caught them up. Streaming the changes to allow a client to receive deltas incrementally would be quite expensive in comparison but perhaps it's a desirable feature to send the result in a single exchange.

My mind does wander to the rsync/noms/jump-rope style architecture for synchronization. Is there a mechanical sympathy with a binary-encoding-level synchronization? Could we use some kind of probablistic-blocks method to determine what needs to be exchanged in each direction at the storage-layer? It's unclear to me right now.

All of this stuff also intersects with privacy concerns. We need to think carefully about how we query between peers and what we expose and to whom. Content hashes are, in general, pretty sensitive.

All that said, I'm very excited that we're getting to the point of working on these problems! I'm eager to dig into all of this in more detail as the binary layers come into greater focus. We'll need to find a way to commission a project to work on this stuff sometime soon, but that's a topic for a different venue.

[mlockett42]

Minisketch (https://github.com/sipa/minisketch) might be a useful library for dealing with synchronizing sets of hashes efficiently. This could be more efficient than storing and exchanging a whole lot of vector clock metadata.

[HerbCaudill]

First, on the subject of timestamps, paying for per-millisecond timestamps on all commits is pretty expensive, and it's not clear to me that there is much utility there. I think per-second timestamps would be a quite-reasonable optimization, and I'd further consider making their insertion optional.

+1 on per-second timestamps, and +1 on making timestamps optional altogether.

[HerbCaudill]

The switch to using hash chaining will have a bearing on network protocols that sync up Automerge replicas, such as Automerge.Connection, which currently relies on vector clocks. This is something I would like to discuss with the community.

For what it's worth, I've got a lot of code built on top of the vector clock model; so I'd hate to see it thrown out entirely. I'd understood from this conversation that the plan was to keep vector clocks but augment them with chain hashing, is that no longer the case?

By the way, I have split Automerge.Connection into a separate repository, because I think we might want to evolve it and assign it version numbers independently from the Automerge core.

That definitely strikes me as the right move - it's always seemed a bit out of place in the main codebase.

[HerbCaudill]

A key insight from the experiments above is that when dealing with large change histories, it is much more efficient to encode the document as a whole than to encode each change separately (in the example above, it makes a factor of 50 difference). This has implications for network protocols that sync Automerge nodes: for example, Hypermerge is currently based on append-only logs of changes, assuming that we're working with one change at a time.

Just to be clear, we'll always have access to the individual changes, right? Cevitxe also stores an append-only log of changes. Maybe there need to be different strategies available for the multiuser-text-editing use case, where you have more individual changes with tiny payloads, vs. the replicated-database use case, where you have fewer changes with more substantial payloads.

[ept]

Okay, I've changed it to use second-resolution timestamps. Here is the updated file size:

	Binary format	Binary + gzip
Paper (whole document)	695,298 bytes	302,067 bytes

I am really pleased with this: the gzipped document uses less than one byte per change, while still capturing the full keystroke-by-keystroke change history!

Just to be clear, we'll always have access to the individual changes, right?

Yes, the full log of individual changes will always be available (e.g. through Automerge.getChanges() and Automerge.getAllChanges()). The compressed document format is only returned by Automerge.save().

I've got a lot of code built on top of the vector clock model; so I'd hate to see it thrown out entirely. I'd understood from this conversation that the plan was to keep vector clocks but augment them with chain hashing

Agree, I think we should keep vector clocks but augment them with hash chains. Vector clocks are still useful because they allow two nodes to sync up in one round-trip, while resolving hash chains sometimes requires several round-trips.

[ichorid]

May I suggest using lz4 compression instead of gzip? Of course, lz4's compression ratio is considerably worse than that of gzip, but the speed of compression is about 6x times higher. Also lz4 is able to compressed data incrementally which is very nice to e.g. fit as much data as possible into network packet without resorting to backtracking.

[ept]

May I suggest using lz4 compression instead of gzip?

This compression step happens outside of Automerge, so you can combine Automerge with the compression algorithm that best suits your requirements.

[dmonad]

I was curious to test the performance improvements in crdt-benchmarks. This is quite a leap!

You can compare the results here: dmonad/crdt-benchmarks@73f3f65

It is very impressive how well your new algorithm compresses meta information! The only thing that I'm worried about is that parsing of the compressed document (parseTime) now takes longer than before. There seems to be quite some overhead in encoding the document in this way. In my experience, the true problem of CRDTs is that they are too slow to parse long edit-histories. Network traffic and storage are not really a problem.

I see that you implemented a similar module for encoding/decoding as I did for Yjs. If you are interested you could make use of lib0/encoding and lib0/decoding. I already put a crazy effort into optimizing read/write performance.

[ept]

Hi @dmonad, thanks for running the benchmarks! The loading of documents is not yet optimised at all, and currently takes a slow path through all the old data structures. That is the next thing on my list of things to fix.

[ankit-m]

@ept I was wondering if you are targeting a release date for v1.

[ept]

Hi @ankit-m, no definite schedule, but I hope to have a preview release within the next month or two. A final stable release may take a bit longer (since this branch has a lot of new code, there will be bugs that need to be ironed out) but I'm hoping well before the end of the year.

[ept]

All the compatibility-breaking changes that I've planned to make are now completed and merged into the performance branch, and so I'm declaring this branch done and merging it into main. No doubt there will be bugs, but we can address them on separate PRs. Hooray, it's shipping! 🚢