Skip to content
Deterministic Binary Serialization
Clojure
Branch: master
Clone or download

Latest commit

Fetching latest commit…
Cannot retrieve the latest commit at this time.

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
resources
src/debris
.gitignore
LICENSE
README.md
deps.edn

README.md

debris


Debris is an alpha-state serialization format that does not focus on speed or size.

Its focus is on being deterministic, simple and predictable.

It should write binary data in a way that:

  • No setup is required;
  • Any implementation generates the same result;
  • Unordered collections are serialized in a sorted manner;
  • Data is never lost, but the data types can be widened on de-serialization.

In addition to usual scenarios for serializing data, this properties would allow any data serialized in debris to:

  • Produce reliable hashes or signatures;
  • Diff against other serialized chunk for updates;

Layout

Every data type in debris is serialized according to the same schema:

  |   1 byte  ||   4 bytes   ||  ...  |
  [type header][size in bytes][payload]

This means that:

  • The top-level collection can hold up to 4GB of data;
  • Collections can hold heterogeneous data;
  • Strings are serialized in utf-8;
  • Numbers are serialized a as Decimal representation;

The types are:

Header Prefix Data type Obs
0x00 Byte/Bytes
0x01 Boolean
0x02 Number Serialized as Decimal
0x03 Text Serialized as UTF-8
0x10 Unordered Sequence Sorted before serialized
0x11 Unordered Map Sorted by key before serialized
0x20 Ordered Sequence
0x21 Ordered Map

To sort 0x10 and 0x11, the header is compared first and then the payload:

;; Using commas to improve readability
(debris/serialize {true 10 false 20})

;; 1: [true 10], 2: [false 20]
;; 1: [[0x01,0x01,0x01] [0x02,0x02,0x31,0x30]] 2: [[0x01,0x01,0x00] [0x02,0x02,0x32,0x30]]
;;
;; Final serialization:
;; |---map---|----------false 20---------------|-------------true 10-------------------|
;; [0x11,0x0E,0x01,0x01,0x0,0x02,0x02,0x32,0x30,0x01,0x01,0x01,0x01,0x02,0x02,0x31,0x30]
You can’t perform that action at this time.