Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Output DnaWasm as gzip-compressed, base-64 encoded list of Strings #1958
Output .dna.json files are unwieldy and large, because WASM is output as a single base-64 encoded String. It is difficult to view .dna.json files in pagination utilities (eg.
Also, since WASM contains much redundancy, Gzip-encoding yields ~75% reduction in sizes.
Serializes to a list of Strings, instead of just a big String; small Zomes (< 1KB, eg. for tests) still round-trip to a single String.
Historical single String (eg.
( if any manual testing or benchmarking was/should be done, add notes and/or screenshots here )
( any new tickets/concerns that were discovered or created during this work but aren't in scope for review here )
- summary of change [PR#1234](https://github.com/holochain/holochain-rust/pull/1234)
@pjkundert this sounds like a cool idea, would you be interested in looking into
it claims ~3-4x speedup on encode/decode logic by using the AVX2 algo
i'm bringing it up now because it seems topical and also might produce incompatible hashes moving forward if the encoding character set is different
Hey, I would highly suggest we introduce a binary DNA format instead. Discussed several times already but never got to it because it wasn't a priority. But 99% of the bytes of a DNA are WASM binary code and don't make sense to be viewed in an editor anyway. Making it even bigger with base64 just to be able to stick to JSON, and then having to deal with the problems of that, feels very weird.
That is, if this working already, awesome and thanks for proactively taking this on. Just wanted to make visible that there might be another solution we might want to approach long-term anyway.
I really like the single .dna.json. Sure, a big blog of binary data in there is an eye-sore, but its a single, user-readable, easy to manage package. Separate binary files will present a more difficult DNA package consistency problem.
We seldom need to peer inside the DNA package, so even keeping the one long String is fine (as long as we compress it to keep the size down).
We've long had base-64 compressed binary data (see: usenet), and they're not bad. If we need to get that 5/4 expansion back (which is unlikely), it easily compresses away. Mostly, the change from big String to list of Strings is to support the rare occasion when we do need to take a quick look at some of the other properties in the Objects in the JSON, that's all.
I've left the
I think that the question of what
This maintains backward compatibility to load existing DNAs, while compressing and new DNAs in a list of strings, instead of one multi-megabyte long string, which is vastly more useful for any human interaction with the DNA file...
I think, most importantly, it restores the usage of the DnaWasm's serialization to control the output of the