From 5c5f15c82c2e47af6b340da809c5f908ffe7ef2e Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 11:54:23 +0100 Subject: [PATCH 01/22] Rewrite SSZ spec * Implement tuples and a chunk-size reduction to 32 bytes (see #665 and #679) * Simplify presentation where appropriate. For example, deserialisation is implicit from serialisation (similar to `bls_sign` being implicit from `bls_verify`) and is left as an implementation exercise. * Dramatically reduce spec size and hopefully improve readability. --- specs/simple-serialize.md | 470 ++++++++------------------------------ 1 file changed, 93 insertions(+), 377 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 109ee289e9..9cee8a5075 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -1,424 +1,140 @@ -# [WIP] SimpleSerialize (SSZ) Spec - -This is the **work in progress** document to describe `SimpleSerialize`, the -current selected serialization method for Ethereum 2.0 using the Beacon Chain. - -This document specifies the general information for serializing and -deserializing objects and data types. - -## ToC - -* [About](#about) -* [Variables and Functions](#variables-and-functions) -* [Constants](#constants) -* [Overview](#overview) - + [Serialize/Encode](#serializeencode) - - [uintN](#uintn) - - [bool](#bool) - - [bytesN](#bytesn) - - [List/Vectors](#listvectors) - - [Container](#container) - + [Deserialize/Decode](#deserializedecode) - - [uintN](#uintn-1) - - [bool](#bool-1) - - [bytesN](#bytesn-1) - - [List/Vectors](#listvectors-1) - - [Container](#container-1) - + [Tree Hash](#tree-hash) - - [`uint8`..`uint256`, `bool`, `bytes1`..`bytes32`](#uint8uint256-bool-bytes1bytes32) - - [`uint264`..`uintN`, `bytes33`..`bytesN`](#uint264uintn-bytes33bytesn) - - [List/Vectors](#listvectors-2) - - [Container](#container-2) - + [Signed Roots](#signed-roots) -* [Implementations](#implementations) - -## About - -`SimpleSerialize` was first proposed by Vitalik Buterin as the serialization -protocol for use in the Ethereum 2.0 Beacon Chain. - -The core feature of `ssz` is the simplicity of the serialization with low -overhead. - -## Variables and Functions - -| Term | Definition | -|:-------------|:-----------------------------------------------------------------------------------------------| -| `little` | Little endian. | -| `byteorder` | Specifies [endianness](https://en.wikipedia.org/wiki/Endianness): big endian or little endian. | -| `len` | Length/number of bytes. | -| `to_bytes` | Convert to bytes. Should take parameters ``size`` and ``byteorder``. | -| `from_bytes` | Convert from bytes to object. Should take ``bytes`` and ``byteorder``. | -| `value` | The value to serialize. | -| `rawbytes` | Raw serialized bytes. | -| `deserialized_object` | The deserialized data in the data structure of your programming language. | -| `new_index` | An index to keep track the latest position where the `rawbytes` have been deserialized. | +# [WIP] SimpleSerialiZe (SSZ) + +This is a **work in progress** describing typing, serialisation and Merkleisation of Ethereum 2.0 objects. + +## Table of contents + +- [Constants](#constants) +- [Types](#types) + - [Primitive types](#primitive-types) + - [Composite types](#composite-types) + - [Notation](#notation) + - [Aliases](#aliases) +- [Serialization](#serialization) + - [`uintN`](#uintn) + - [`bool`](#bool) + - [Containers](#containers) + - [Tuples](#tuples) + - [Lists](#lists) +- [Deserialization](#deserialization) +- [Merkleization](#merkleization) +- [Signed containers](#signed-containers) +- [Implementations](#implementations) ## Constants -| Constant | Value | Definition | -|:------------------|:-----:|:--------------------------------------------------------------------------------------| -| `LENGTH_BYTES` | 4 | Number of bytes used for the length added before a variable-length serialized object. | -| `SSZ_CHUNK_SIZE` | 128 | Number of bytes for the chunk size of the Merkle tree leaf. | +| Name | Value | Definition | +|-|:-:|-| +| `LENGTH_BYTES` | 4 | Number of bytes used for the length added before a variable-length serialized object. | -## Overview +## Types -### Serialize/Encode +### Primitive types -#### uintN +* `uintN`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) +* `bool`: 1-bit unsigned integer -| uint Type | Usage | -|:---------:|:-----------------------------------------------------------| -| `uintN` | Type of `N` bits unsigned integer, where ``N % 8 == 0``. | +### Composite types -Convert directly to bytes the size of the int. (e.g. ``uint16 = 2 bytes``) +* **Containes**: ordered heterogenous collection of values +* **Tuple**: ordered fixed-size homogeneous collection of values +* **List**: ordered variable-size homogenous collection of values -All integers are serialized as **little endian**. +### Notation -| Check to perform | Code | -|:-----------------------|:----------------------| -| Size is a byte integer | ``int_size % 8 == 0`` | +* **Containes**: key-pair notation `{}`, e.g. `{'key1': uint64, 'key2': bool}` +* **Tuple**: angle-braket notation `[]`, e.g. `uint64[]` +* **List**: angle-braket notation `[N]`, e.g. `uint64[N]` -```python -assert(int_size % 8 == 0) -buffer_size = int_size / 8 -return value.to_bytes(buffer_size, 'little') -``` - -#### bool - -Convert directly to a single 0x00 or 0x01 byte. - -| Check to perform | Code | -|:------------------|:---------------------------| -| Value is boolean | ``value in (True, False)`` | - -```python -assert(value in (True, False)) -return b'\x01' if value is True else b'\x00' -``` - -#### bytesN - -A fixed-size byte array. +### Aliases -| Checks to perform | Code | -|:---------------------------------------|:---------------------| -| Length in bytes is correct for `bytesN` | ``len(value) == N`` | +For convenience we alias: -```python -assert(len(value) == N) - -return value -``` - -#### List/Vectors - -Lists are a collection of elements of the same homogeneous type. - -| Check to perform | Code | -|:--------------------------------------------|:----------------------------| -| Length of serialized list fits into 4 bytes | ``len(serialized) < 2**32`` | +* `byte` to `uint8` +* `bytes` to `byte[]` +* `bytesN` to `byte[N]` +* `bit` to `bool` -1. Serialize all list elements individually and concatenate them. -2. Prefix the concatenation with its length encoded as a `4-byte` **little-endian** unsigned integer. +## Serialization -We define `bytes` to be a synonym of `List[bytes1]`. +We reccursively define a `serialize` function. In the code below `value` refers to a value of the specified type. -**Example in Python** +### `uintN` ```python - -serialized_list_string = b'' - -for item in value: - serialized_list_string += serialize(item) - -assert(len(serialized_list_string) < 2**32) - -serialized_len = (len(serialized_list_string).to_bytes(LENGTH_BYTES, 'little')) - -return serialized_len + serialized_list_string +assert N in [8, 16, 32, 64, 128, 256] +return value.to_bytes(N / 8, 'little') ``` -#### Container - -A container represents a heterogenous, associative collection of key-value pairs. Each pair is referred to as a `field`. To get the value for a given field, you supply the key which is a symbol unique to the container referred to as the field's `name`. The container data type is analogous to the `struct` type found in many languages like C or Go. - -To serialize a container, obtain the list of its field's names in the specified order. For each field name in this list, obtain the corresponding value and serialize it. Tightly pack the complete set of serialized values in the same order as the field names into a buffer. Calculate the size of this buffer of serialized bytes and encode as a `4-byte` **little endian** `uint32`. Prepend the encoded length to the buffer. The result of this concatenation is the final serialized value of the container. - -| Check to perform | Code | -|:----------------------------------------------|:----------------------------| -| Length of serialized fields fits into 4 bytes | ``len(serialized) < 2**32`` | - -To serialize: - -1. Get the list of the container's fields. - -2. For each name in the list, obtain the corresponding value from the container and serialize it. Place this serialized value into a buffer. The serialized values should be tightly packed. - -3. Get the number of raw bytes in the serialized buffer. Encode that number as a `4-byte` **little endian** `uint32`. - -4. Prepend the length to the serialized buffer. - -**Example in Python** +### `bool` ```python -def get_field_names(typ): - return typ.fields.keys() - -def get_value_for_field_name(value, field_name): - return getattr(value, field_name) - -def get_type_for_field_name(typ, field_name): - return typ.fields[field_name] - -serialized_buffer = b'' - -typ = type(value) -for field_name in get_field_names(typ): - field_value = get_value_for_field_name(value, field_name) - field_type = get_type_for_field_name(typ, field_name) - serialized_buffer += serialize(field_value, field_type) - -assert(len(serialized_buffer) < 2**32) - -serialized_len = (len(serialized_buffer).to_bytes(LENGTH_BYTES, 'little')) - -return serialized_len + serialized_buffer -``` - -### Deserialize/Decode - -The decoding requires knowledge of the type of the item to be decoded. When -performing decoding on an entire serialized string, it also requires knowledge -of the order in which the objects have been serialized. - -Note: Each return will provide: -- `deserialized_object` -- `new_index` - -At each step, the following checks should be made: - -| Check to perform | Check | -|:-------------------------|:-----------------------------------------------------------| -| Ensure sufficient length | ``len(rawbytes) >= current_index + deserialize_length`` | - -At the final step, the following checks should be made: - -| Check to perform | Check | -|:-------------------------|:-------------------------------------| -| Ensure no extra length | `new_index == len(rawbytes)` | - -#### uintN - -Convert directly from bytes into integer utilising the number of bytes the same -size as the integer length. (e.g. ``uint16 == 2 bytes``) - -All integers are interpreted as **little endian**. - -```python -byte_length = int_size / 8 -new_index = current_index + byte_length -assert(len(rawbytes) >= new_index) -return int.from_bytes(rawbytes[current_index:current_index+byte_length], 'little'), new_index -``` - -#### bool - -Return True if 0x01, False if 0x00. - -```python -assert rawbytes in (b'\x00', b'\x01') -return True if rawbytes == b'\x01' else False -``` - -#### bytesN - -Return the `N` bytes. - -```python -assert(len(rawbytes) >= current_index + N) -new_index = current_index + N -return rawbytes[current_index:current_index+N], new_index -``` - -#### List/Vectors - -Deserialize each element in the list. -1. Get the length of the serialized list. -2. Loop through deserializing each item in the list until you reach the -entire length of the list. - -| Check to perform | code | -|:------------------------------------------|:----------------------------------------------------------------| -| ``rawbytes`` has enough left for length | ``len(rawbytes) > current_index + LENGTH_BYTES`` | -| list is not greater than serialized bytes | ``len(rawbytes) > current_index + LENGTH_BYTES + total_length`` | - -```python -assert(len(rawbytes) > current_index + LENGTH_BYTES) -total_length = int.from_bytes(rawbytes[current_index:current_index + LENGTH_BYTES], 'little') -new_index = current_index + LENGTH_BYTES + total_length -assert(len(rawbytes) >= new_index) -item_index = current_index + LENGTH_BYTES -deserialized_list = [] - -while item_index < new_index: - object, item_index = deserialize(rawbytes, item_index, item_type) - deserialized_list.append(object) - -return deserialized_list, new_index +assert value in (True, False) +return b'\x01' if value is True else b'\x00' ``` -#### Container - -Refer to the section on container encoding for some definitions. - -To deserialize a container, loop over each field in the container and use the type of that field to know what kind of deserialization to perform. Consume successive elements of the data stream for each successful deserialization. - -Instantiate a container with the full set of deserialized data, matching each member with the corresponding field. - -| Check to perform | code | -|:------------------------------------------|:----------------------------------------------------------------| -| ``rawbytes`` has enough left for length | ``len(rawbytes) > current_index + LENGTH_BYTES`` | -| list is not greater than serialized bytes | ``len(rawbytes) > current_index + LENGTH_BYTES + total_length`` | - -To deserialize: - -1. Get the list of the container's fields. -2. For each name in the list, attempt to deserialize a value for that type. Collect these values as they will be used to construct an instance of the container. -3. Construct a container instance after successfully consuming the entire subset of the stream for the serialized container. - -**Example in Python** +### Containers ```python -def get_field_names(typ): - return typ.fields.keys() - -def get_value_for_field_name(value, field_name): - return getattr(value, field_name) - -def get_type_for_field_name(typ, field_name): - return typ.fields[field_name] - -class Container: - # this is the container; here we will define an empty class for demonstration - pass - -# get a reference to the type in some way... -container = Container() -typ = type(container) - -assert(len(rawbytes) > current_index + LENGTH_BYTES) -total_length = int.from_bytes(rawbytes[current_index:current_index + LENGTH_BYTES], 'little') -new_index = current_index + LENGTH_BYTES + total_length -assert(len(rawbytes) >= new_index) -item_index = current_index + LENGTH_BYTES - -values = {} -for field_name in get_field_names(typ): - field_name_type = get_type_for_field_name(typ, field_name) - values[field_name], item_index = deserialize(data, item_index, field_name_type) -assert item_index == new_index -return typ(**values), item_index +serialized_elements = [serialize(element) for element in value] +serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) +assert len(serialized_bytes) < 2**32 +serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') +return serialized_length + serialized_bytes ``` -### Tree Hash - -The below `hash_tree_root_internal` algorithm is defined recursively in the case of lists and containers, and it outputs a value equal to or less than 32 bytes in size. For use as a "final output" (eg. for signing), use `hash_tree_root(x) = zpad(hash_tree_root_internal(x), 32)`, where `zpad` is a helper that extends the given `bytes` value to the desired `length` by adding zero bytes on the right: +### Tuples ```python -def zpad(input: bytes, length: int) -> bytes: - return input + b'\x00' * (length - len(input)) +serialized_elements = [serialize(element) for element in value] +serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) +return serialized_bytes ``` -Refer to [the helper function `hash`](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md#hash) of Phase 0 of the [Eth2.0 specs](https://github.com/ethereum/eth2.0-specs) for a definition of the hash function used below, `hash(x)`. - -#### `uint8`..`uint256`, `bool`, `bytes1`..`bytes32` - -Return the serialization of the value. - -#### `uint264`..`uintN`, `bytes33`..`bytesN` - -Return the hash of the serialization of the value. - -#### List/Vectors - -First, we define the Merkle tree function. +### Lists ```python -# Merkle tree hash of a list of homogenous, non-empty items -def merkle_hash(lst): - # Store length of list (to compensate for non-bijectiveness of padding) - datalen = len(lst).to_bytes(32, 'little') - - if len(lst) == 0: - # Handle empty list case - chunkz = [b'\x00' * SSZ_CHUNK_SIZE] - elif len(lst[0]) < SSZ_CHUNK_SIZE: - # See how many items fit in a chunk - items_per_chunk = SSZ_CHUNK_SIZE // len(lst[0]) - - # Build a list of chunks based on the number of items in the chunk - chunkz = [ - zpad(b''.join(lst[i:i + items_per_chunk]), SSZ_CHUNK_SIZE) - for i in range(0, len(lst), items_per_chunk) - ] - else: - # Leave large items alone - chunkz = lst - - # Merkleise - def next_power_of_2(x): - return 1 if x == 0 else 2**(x - 1).bit_length() - - for i in range(len(chunkz), next_power_of_2(len(chunkz))): - chunkz.append(b'\x00' * SSZ_CHUNK_SIZE) - while len(chunkz) > 1: - chunkz = [hash(chunkz[i] + chunkz[i+1]) for i in range(0, len(chunkz), 2)] - - # Return hash of root and data length - return hash(chunkz[0] + datalen) +serialized_elements = [serialize(element) for element in value] +serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) +assert len(serialized_elements) < 2**32 +serialized_length = len(serialized_elements).to_bytes(LENGTH_BYTES, 'little') +return serialized_length + serialized_bytes ``` -To `hash_tree_root_internal` a list, we simply do: +## Deserialization -```python -return merkle_hash([hash_tree_root_internal(item) for item in value]) -``` - -Where the inner `hash_tree_root_internal` is a recursive application of the tree-hashing function (returning less than 32 bytes for short single values). +Given a type, serialisation is an injective function from objects of that type to byte strings. That is, deserialisation—the inverse function—is well-defined. -#### Container +## Merkleization -Recursively tree hash the values in the container in the same order as the fields, and Merkle hash the results. +We first define helper functions: -```python -return merkle_hash([hash_tree_root_internal(getattr(x, field)) for field in value.fields]) -``` +* `pack`: Given ordered objects of the same basic type, serialise them, pack them into 32-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. +* `merkleise`: Given ordered 32-byte chunks, right-pad them with zero chunks to the closest power of two, Merkleize the chunks, and return the root. +* `mix_in_length`: Given a Merkle root `r` and a length `l` (32-byte little-endian serialisation) return `hash(r + l)`. -### Signed roots +Let `o` be an object. We now define object Merkleization `hash_tree_root(o)` recursively: -Let `field_name` be a field name in an SSZ container `container`. We define `truncate(container, field_name)` to be the `container` with the fields from `field_name` onwards truncated away. That is, `truncate(container, field_name) = [getattr(container, field)) for field in value.fields[:i]]` where `i = value.fields.index(field_name)`. +* `merkleize(pack(o))` if `o` is a basic object or a tuple of basic objects +* `mix_in_length(merkleize(pack(o)), len(o))` if `o` is a list of basic objects +* `merkleize([hash_tree_root(element) for element in o])` if `o` is a tuple of composite objects or a container +* `mix_in_length(merkleize([hash_tree_root(element) for element in o]), len(o))` if `o` is a list of composite objects -When `field_name` maps to a signature (e.g. a BLS12-381 signature of type `Bytes96`) the convention is that the corresponding signed message be `signed_root(container, field_name) = hash_tree_root(truncate(container, field_name))`. For example if `container = {"foo": sub_object_1, "bar": sub_object_2, "signature": bytes96, "baz": sub_object_3}` then `signed_root(container, "signature") = merkle_hash([hash_tree_root(sub_object_1), hash_tree_root(sub_object_2)])`. +## Signed containers -Note that this convention means that fields after the signature are _not_ signed over. If there are multiple signatures in `container` then those are expected to be signing over the fields in the order specified. If multiple signatures of the same value are expected the convention is that the signature field be an array of signatures. +Let `container` be a self-signed container object. The convention is that the signature (e.g. a `bytes96` BLS12-381 signature) be the last field of `container`. Further, the signed message for `container` is `signed_root(container) = hash_tree_root(truncate_last(container))` where `truncate_last` truncates the last element of `container`. ## Implementations -| Language | Implementation | Description | -|:--------:|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------| -| Python | [ https://github.com/ethereum/py-ssz ](https://github.com/ethereum/py-ssz) | Python implementation of SSZ | -| Rust | [ https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz ](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) | Lighthouse (Rust Ethereum 2.0 Node) maintained SSZ. | -| Nim | [ https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim ](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | Nim Implementation maintained SSZ. | -| Rust | [ https://github.com/paritytech/shasper/tree/master/util/ssz ](https://github.com/paritytech/shasper/tree/master/util/ssz) | Shasper implementation of SSZ maintained by ParityTech. | -| Javascript | [ https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js ](https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js) | Javascript Implementation maintained SSZ | -| Java | [ https://www.github.com/ConsenSys/cava/tree/master/ssz ](https://www.github.com/ConsenSys/cava/tree/master/ssz) | SSZ Java library part of the Cava suite | -| Go | [ https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz ](https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz) | Go implementation of SSZ mantained by Prysmatic Labs | -| Swift | [ https://github.com/yeeth/SimpleSerialize.swift ](https://github.com/yeeth/SimpleSerialize.swift) | Swift implementation maintained SSZ | -| C# | [ https://github.com/codingupastorm/csharp-ssz ](https://github.com/codingupastorm/csharp-ssz) | C# implementation maintained SSZ | -| C++ | [ https://github.com/NAKsir-melody/cpp_ssz](https://github.com/NAKsir-melody/cpp_ssz) | C++ implementation maintained SSZ | - -## Copyright -Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). +| Language | Implementation | Description | +|:-:|-|-| +| Python | [ https://github.com/ethereum/py-ssz ](https://github.com/ethereum/py-ssz) | Python implementation of SSZ | +| Rust | [ https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz ](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) | Lighthouse (Rust Ethereum 2.0 Node) maintained SSZ | +| Nim | [ https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim ](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | Nim Implementation maintained SSZ | +| Rust | [ https://github.com/paritytech/shasper/tree/master/util/ssz ](https://github.com/paritytech/shasper/tree/master/util/ssz) | Shasper implementation of SSZ maintained by ParityTech | +| Javascript | [ https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js ](https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js) | Javascript Implementation maintained SSZ | +| Java | [ https://www.github.com/ConsenSys/cava/tree/master/ssz ](https://www.github.com/ConsenSys/cava/tree/master/ssz) | SSZ Java library part of the Cava suite | +| Go | [ https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz ](https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz) | Go implementation of SSZ mantained by Prysmatic Labs | +| Swift | [ https://github.com/yeeth/SimpleSerialize.swift ](https://github.com/yeeth/SimpleSerialize.swift) | Swift implementation maintained SSZ | +| C# | [ https://github.com/codingupastorm/csharp-ssz ](https://github.com/codingupastorm/csharp-ssz) | C# implementation maintained SSZ | +| C++ | [ https://github.com/NAKsir-melody/cpp_ssz](https://github.com/NAKsir-melody/cpp_ssz) | C++ implementation maintained SSZ | From df50ac1adc76b0fe4cfecbd296c612db5f62df65 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 12:03:27 +0100 Subject: [PATCH 02/22] Update simple-serialize.md --- specs/simple-serialize.md | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 9cee8a5075..4f83af4a49 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -25,7 +25,8 @@ This is a **work in progress** describing typing, serialisation and Merkleisatio | Name | Value | Definition | |-|:-:|-| -| `LENGTH_BYTES` | 4 | Number of bytes used for the length added before a variable-length serialized object. | +| `LENGTH_BYTES` | 4 | Number of bytes for the length of variable-length serialized objects. | +| `MAX_LENGTH` | 2**(8 * LENGTH_BYTES) | Maximum serialization length. | ## Types @@ -36,15 +37,15 @@ This is a **work in progress** describing typing, serialisation and Merkleisatio ### Composite types -* **Containes**: ordered heterogenous collection of values -* **Tuple**: ordered fixed-size homogeneous collection of values -* **List**: ordered variable-size homogenous collection of values +* **Container**: ordered heterogenous collection of values +* **Tuple**: ordered fixed-length homogeneous collection of values +* **List**: ordered variable-length homogenous collection of values ### Notation -* **Containes**: key-pair notation `{}`, e.g. `{'key1': uint64, 'key2': bool}` -* **Tuple**: angle-braket notation `[]`, e.g. `uint64[]` -* **List**: angle-braket notation `[N]`, e.g. `uint64[N]` +* **Container**: key-pair notation `{}`, e.g. `{'key1': uint64, 'key2': bool}` +* **Tuple**: angle-braket notation `[N]`, e.g. `uint64[N]` +* **List**: angle-braket notation `[]`, e.g. `uint64[]` ### Aliases @@ -57,28 +58,28 @@ For convenience we alias: ## Serialization -We reccursively define a `serialize` function. In the code below `value` refers to a value of the specified type. +We reccursively define the `serialize` function which consumes an object `o` (of the type specified) and returns a byte string `[]byte`. ### `uintN` ```python assert N in [8, 16, 32, 64, 128, 256] -return value.to_bytes(N / 8, 'little') +return o.to_bytes(N / 8, 'little') ``` ### `bool` ```python -assert value in (True, False) -return b'\x01' if value is True else b'\x00' +assert o in (True, False) +return b'\x01' if o is True else b'\x00' ``` ### Containers ```python -serialized_elements = [serialize(element) for element in value] +serialized_elements = [serialize(element) for element in o] serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) -assert len(serialized_bytes) < 2**32 +assert len(serialized_bytes) < MAX_LENGTH serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') return serialized_length + serialized_bytes ``` @@ -86,7 +87,7 @@ return serialized_length + serialized_bytes ### Tuples ```python -serialized_elements = [serialize(element) for element in value] +serialized_elements = [serialize(element) for element in o] serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) return serialized_bytes ``` @@ -94,9 +95,9 @@ return serialized_bytes ### Lists ```python -serialized_elements = [serialize(element) for element in value] +serialized_elements = [serialize(element) for element in o] serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) -assert len(serialized_elements) < 2**32 +assert len(serialized_elements) < MAX_LENGTH serialized_length = len(serialized_elements).to_bytes(LENGTH_BYTES, 'little') return serialized_length + serialized_bytes ``` From 313fe46a94f0790bfb1be024e4ebf8f921a0fbdc Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 12:06:06 +0100 Subject: [PATCH 03/22] Update simple-serialize.md --- specs/simple-serialize.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 4f83af4a49..7d3d69725c 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -1,6 +1,6 @@ # [WIP] SimpleSerialiZe (SSZ) -This is a **work in progress** describing typing, serialisation and Merkleisation of Ethereum 2.0 objects. +This is a **work in progress** describing typing, serialization and Merkleization of Ethereum 2.0 objects. ## Table of contents @@ -25,8 +25,8 @@ This is a **work in progress** describing typing, serialisation and Merkleisatio | Name | Value | Definition | |-|:-:|-| -| `LENGTH_BYTES` | 4 | Number of bytes for the length of variable-length serialized objects. | -| `MAX_LENGTH` | 2**(8 * LENGTH_BYTES) | Maximum serialization length. | +| `LENGTH_BYTES` | `4` | Number of bytes for the length of variable-length serialized objects. | +| `MAX_LENGTH` | `2**(8 * LENGTH_BYTES)` | Maximum serialization length. | ## Types @@ -104,15 +104,15 @@ return serialized_length + serialized_bytes ## Deserialization -Given a type, serialisation is an injective function from objects of that type to byte strings. That is, deserialisation—the inverse function—is well-defined. +Given a type, serialization is an injective function from objects of that type to byte strings. That is, deserialization—the inverse function—is well-defined. ## Merkleization We first define helper functions: -* `pack`: Given ordered objects of the same basic type, serialise them, pack them into 32-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. -* `merkleise`: Given ordered 32-byte chunks, right-pad them with zero chunks to the closest power of two, Merkleize the chunks, and return the root. -* `mix_in_length`: Given a Merkle root `r` and a length `l` (32-byte little-endian serialisation) return `hash(r + l)`. +* `pack`: Given ordered objects of the same basic type, serialize them, pack them into 32-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. +* `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the closest power of two, Merkleize the chunks, and return the root. +* `mix_in_length`: Given a Merkle root `root` and a length `length` (32-byte little-endian serialization) return `hash(root + length)`. Let `o` be an object. We now define object Merkleization `hash_tree_root(o)` recursively: From 5116645dfa802044d56c63a9f77220a7ec7b1e9e Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 12:40:08 +0100 Subject: [PATCH 04/22] Update simple-serialize.md --- specs/simple-serialize.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 7d3d69725c..f972952d52 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -18,7 +18,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Lists](#lists) - [Deserialization](#deserialization) - [Merkleization](#merkleization) -- [Signed containers](#signed-containers) +- [Self-signed containers](#self-signed-containers) - [Implementations](#implementations) ## Constants @@ -121,21 +121,21 @@ Let `o` be an object. We now define object Merkleization `hash_tree_root(o)` rec * `merkleize([hash_tree_root(element) for element in o])` if `o` is a tuple of composite objects or a container * `mix_in_length(merkleize([hash_tree_root(element) for element in o]), len(o))` if `o` is a list of composite objects -## Signed containers +## Self-signed containers Let `container` be a self-signed container object. The convention is that the signature (e.g. a `bytes96` BLS12-381 signature) be the last field of `container`. Further, the signed message for `container` is `signed_root(container) = hash_tree_root(truncate_last(container))` where `truncate_last` truncates the last element of `container`. ## Implementations -| Language | Implementation | Description | -|:-:|-|-| -| Python | [ https://github.com/ethereum/py-ssz ](https://github.com/ethereum/py-ssz) | Python implementation of SSZ | -| Rust | [ https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz ](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) | Lighthouse (Rust Ethereum 2.0 Node) maintained SSZ | -| Nim | [ https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim ](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | Nim Implementation maintained SSZ | -| Rust | [ https://github.com/paritytech/shasper/tree/master/util/ssz ](https://github.com/paritytech/shasper/tree/master/util/ssz) | Shasper implementation of SSZ maintained by ParityTech | -| Javascript | [ https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js ](https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js) | Javascript Implementation maintained SSZ | -| Java | [ https://www.github.com/ConsenSys/cava/tree/master/ssz ](https://www.github.com/ConsenSys/cava/tree/master/ssz) | SSZ Java library part of the Cava suite | -| Go | [ https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz ](https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz) | Go implementation of SSZ mantained by Prysmatic Labs | -| Swift | [ https://github.com/yeeth/SimpleSerialize.swift ](https://github.com/yeeth/SimpleSerialize.swift) | Swift implementation maintained SSZ | -| C# | [ https://github.com/codingupastorm/csharp-ssz ](https://github.com/codingupastorm/csharp-ssz) | C# implementation maintained SSZ | -| C++ | [ https://github.com/NAKsir-melody/cpp_ssz](https://github.com/NAKsir-melody/cpp_ssz) | C++ implementation maintained SSZ | +| Language | Project | Maintainer | Implementation | +|-|-|-|-| +| Python | Ethereum 2.0 | Ethereum Foundation | [https://github.com/ethereum/py-ssz](https://github.com/ethereum/py-ssz) | +| Rust | Lighthouse | Sigma Prime | [https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) | +| Nim | Nimbus | Status | [https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | +| Rust | Shasper | ParityTech | [https://github.com/paritytech/shasper/tree/master/util/ssz](https://github.com/paritytech/shasper/tree/master/util/ssz) | +| Javascript | Lodestart | Chain Safe Systems | [https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js](https://github.com/ChainSafeSystems/ssz-js/blob/master/src/index.js) | +| Java | Cava | ConsenSys | [https://www.github.com/ConsenSys/cava/tree/master/ssz](https://www.github.com/ConsenSys/cava/tree/master/ssz) | +| Go | Prysm | Prysmatic Labs | [https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz](https://github.com/prysmaticlabs/prysm/tree/master/shared/ssz) | +| Swift | Yeeth | Dean Eigenmann | [https://github.com/yeeth/SimpleSerialize.swift](https://github.com/yeeth/SimpleSerialize.swift) | +| C# | | Jordan Andrews | [https://github.com/codingupastorm/csharp-ssz](https://github.com/codingupastorm/csharp-ssz) | +| C++ | | | [https://github.com/NAKsir-melody/cpp_ssz](https://github.com/NAKsir-melody/cpp_ssz) | From 28cf5860ea139bb0953075083d3f5f638541946a Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 12:53:24 +0100 Subject: [PATCH 05/22] Update simple-serialize.md --- specs/simple-serialize.md | 42 +++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index f972952d52..780f1cc90b 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -1,11 +1,11 @@ -# [WIP] SimpleSerialiZe (SSZ) +# SimpleSerialiZe (SSZ) This is a **work in progress** describing typing, serialization and Merkleization of Ethereum 2.0 objects. ## Table of contents - [Constants](#constants) -- [Types](#types) +- [Typing](#typing) - [Primitive types](#primitive-types) - [Composite types](#composite-types) - [Notation](#notation) @@ -23,12 +23,12 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ## Constants -| Name | Value | Definition | -|-|:-:|-| +| Name | Value | Description | +|-|-|-| | `LENGTH_BYTES` | `4` | Number of bytes for the length of variable-length serialized objects. | | `MAX_LENGTH` | `2**(8 * LENGTH_BYTES)` | Maximum serialization length. | -## Types +## Typing ### Primitive types @@ -43,9 +43,9 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ### Notation -* **Container**: key-pair notation `{}`, e.g. `{'key1': uint64, 'key2': bool}` -* **Tuple**: angle-braket notation `[N]`, e.g. `uint64[N]` -* **List**: angle-braket notation `[]`, e.g. `uint64[]` +* **Container**: key-pair curly braken notation `{}` (e.g. `{'key1': uint64, 'key2': bool}`) +* **Tuple**: angle braket notation `[N]` (e.g. `uint64[N]`) +* **List**: angle braket notation `[]` (e.g. `uint64[]`) ### Aliases @@ -58,26 +58,26 @@ For convenience we alias: ## Serialization -We reccursively define the `serialize` function which consumes an object `o` (of the type specified) and returns a byte string `[]byte`. +We reccursively define the `serialize` function which consumes an object `object` (of the type specified) and returns a byte string of type `bytes`. ### `uintN` ```python assert N in [8, 16, 32, 64, 128, 256] -return o.to_bytes(N / 8, 'little') +return object.to_bytes(N / 8, 'little') ``` ### `bool` ```python -assert o in (True, False) -return b'\x01' if o is True else b'\x00' +assert object in (True, False) +return b'\x01' if object is True else b'\x00' ``` ### Containers ```python -serialized_elements = [serialize(element) for element in o] +serialized_elements = [serialize(element) for element in object] serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) assert len(serialized_bytes) < MAX_LENGTH serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') @@ -87,7 +87,7 @@ return serialized_length + serialized_bytes ### Tuples ```python -serialized_elements = [serialize(element) for element in o] +serialized_elements = [serialize(element) for element in object] serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) return serialized_bytes ``` @@ -95,7 +95,7 @@ return serialized_bytes ### Lists ```python -serialized_elements = [serialize(element) for element in o] +serialized_elements = [serialize(element) for element in object] serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) assert len(serialized_elements) < MAX_LENGTH serialized_length = len(serialized_elements).to_bytes(LENGTH_BYTES, 'little') @@ -114,12 +114,12 @@ We first define helper functions: * `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the closest power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (32-byte little-endian serialization) return `hash(root + length)`. -Let `o` be an object. We now define object Merkleization `hash_tree_root(o)` recursively: +Let `object` be an object. We now define object Merkleization `hash_tree_root(object)` recursively: -* `merkleize(pack(o))` if `o` is a basic object or a tuple of basic objects -* `mix_in_length(merkleize(pack(o)), len(o))` if `o` is a list of basic objects -* `merkleize([hash_tree_root(element) for element in o])` if `o` is a tuple of composite objects or a container -* `mix_in_length(merkleize([hash_tree_root(element) for element in o]), len(o))` if `o` is a list of composite objects +* `merkleize(pack(object))` if `object` is a basic object or a tuple of basic objects +* `mix_in_length(merkleize(pack(object)), len(object))` if `object` is a list of basic objects +* `merkleize([hash_tree_root(element) for element in object])` if `object` is a tuple of composite objects or a container +* `mix_in_length(merkleize([hash_tree_root(element) for element in object]), len(object))` if `object` is a list of composite objects ## Self-signed containers @@ -128,7 +128,7 @@ Let `container` be a self-signed container object. The convention is that the si ## Implementations | Language | Project | Maintainer | Implementation | -|-|-|-|-| +|:-:|-|-| | Python | Ethereum 2.0 | Ethereum Foundation | [https://github.com/ethereum/py-ssz](https://github.com/ethereum/py-ssz) | | Rust | Lighthouse | Sigma Prime | [https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) | | Nim | Nimbus | Status | [https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | From 07922a63fbb52cd3f42441cb78909e8a6bc4c0c9 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 12:54:56 +0100 Subject: [PATCH 06/22] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 780f1cc90b..eb1d3cace4 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -128,7 +128,7 @@ Let `container` be a self-signed container object. The convention is that the si ## Implementations | Language | Project | Maintainer | Implementation | -|:-:|-|-| +|-|-|-|-| | Python | Ethereum 2.0 | Ethereum Foundation | [https://github.com/ethereum/py-ssz](https://github.com/ethereum/py-ssz) | | Rust | Lighthouse | Sigma Prime | [https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz](https://github.com/sigp/lighthouse/tree/master/beacon_chain/utils/ssz) | | Nim | Nimbus | Status | [https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim](https://github.com/status-im/nim-beacon-chain/blob/master/beacon_chain/ssz.nim) | From e698d7e29bb85122d33290bb255457ca8bcc0de1 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 12:57:06 +0100 Subject: [PATCH 07/22] Update simple-serialize.md --- specs/simple-serialize.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index eb1d3cace4..7da967a25a 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -30,24 +30,24 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ## Typing -### Primitive types +#### Primitive types * `uintN`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `bool`: 1-bit unsigned integer -### Composite types +#### Composite types -* **Container**: ordered heterogenous collection of values -* **Tuple**: ordered fixed-length homogeneous collection of values -* **List**: ordered variable-length homogenous collection of values +* **container**: ordered heterogenous collection of values +* **tuple**: ordered fixed-length homogeneous collection of values +* **list**: ordered variable-length homogenous collection of values -### Notation +#### Notation -* **Container**: key-pair curly braken notation `{}` (e.g. `{'key1': uint64, 'key2': bool}`) -* **Tuple**: angle braket notation `[N]` (e.g. `uint64[N]`) -* **List**: angle braket notation `[]` (e.g. `uint64[]`) +* **container**: key-pair curly braket notation `{}` (e.g. `{'key1': uint64, 'key2': bool}`) +* **tuple**: angle braket notation `[N]` (e.g. `uint64[N]`) +* **list**: angle braket notation `[]` (e.g. `uint64[]`) -### Aliases +#### Aliases For convenience we alias: @@ -60,21 +60,21 @@ For convenience we alias: We reccursively define the `serialize` function which consumes an object `object` (of the type specified) and returns a byte string of type `bytes`. -### `uintN` +#### `uintN` ```python assert N in [8, 16, 32, 64, 128, 256] return object.to_bytes(N / 8, 'little') ``` -### `bool` +#### `bool` ```python assert object in (True, False) return b'\x01' if object is True else b'\x00' ``` -### Containers +#### Containers ```python serialized_elements = [serialize(element) for element in object] @@ -84,7 +84,7 @@ serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') return serialized_length + serialized_bytes ``` -### Tuples +#### Tuples ```python serialized_elements = [serialize(element) for element in object] @@ -92,7 +92,7 @@ serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) return serialized_bytes ``` -### Lists +#### Lists ```python serialized_elements = [serialize(element) for element in object] From 0df8f8b3c68bc95647c004b89b7decf6d61a8863 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 13:04:36 +0100 Subject: [PATCH 08/22] Update simple-serialize.md --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 7da967a25a..41640aabea 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -6,7 +6,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Constants](#constants) - [Typing](#typing) - - [Primitive types](#primitive-types) + - [Basic types](#basic-types) - [Composite types](#composite-types) - [Notation](#notation) - [Aliases](#aliases) @@ -30,7 +30,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ## Typing -#### Primitive types +#### Basic types * `uintN`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `bool`: 1-bit unsigned integer From 828146cbf8b37fd4455df664c45c8a7cb5160889 Mon Sep 17 00:00:00 2001 From: jannikluhn Date: Wed, 27 Feb 2019 17:15:46 +0100 Subject: [PATCH 09/22] Apply suggestions from code review Co-Authored-By: JustinDrake --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 41640aabea..a87f477ae4 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -64,7 +64,7 @@ We reccursively define the `serialize` function which consumes an object `object ```python assert N in [8, 16, 32, 64, 128, 256] -return object.to_bytes(N / 8, 'little') +return object.to_bytes(N // 8, 'little') ``` #### `bool` @@ -111,7 +111,7 @@ Given a type, serialization is an injective function from objects of that type t We first define helper functions: * `pack`: Given ordered objects of the same basic type, serialize them, pack them into 32-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. -* `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the closest power of two, Merkleize the chunks, and return the root. +* `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (32-byte little-endian serialization) return `hash(root + length)`. Let `object` be an object. We now define object Merkleization `hash_tree_root(object)` recursively: From 95fa9d56b85d04e9153c7ec5774a8505ddb2e949 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 17:35:26 +0100 Subject: [PATCH 10/22] Update simple-serialize.md --- specs/simple-serialize.md | 45 +++++++++++---------------------------- 1 file changed, 12 insertions(+), 33 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index a87f477ae4..3dc9b0b077 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -13,9 +13,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Serialization](#serialization) - [`uintN`](#uintn) - [`bool`](#bool) - - [Containers](#containers) - - [Tuples](#tuples) - - [Lists](#lists) + - [Containers, tuples, lists](#containers-tuples-lists) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Self-signed containers](#self-signed-containers) @@ -58,50 +56,31 @@ For convenience we alias: ## Serialization -We reccursively define the `serialize` function which consumes an object `object` (of the type specified) and returns a byte string of type `bytes`. +We reccursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. #### `uintN` ```python assert N in [8, 16, 32, 64, 128, 256] -return object.to_bytes(N // 8, 'little') +return value.to_bytes(N // 8, 'little') ``` #### `bool` ```python -assert object in (True, False) -return b'\x01' if object is True else b'\x00' +assert value in (True, False) +return b'\x01' if value is True else b'\x00' ``` -#### Containers +#### Containers, tuples, lists ```python -serialized_elements = [serialize(element) for element in object] -serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) +serialized_bytes = ''.join([serialize(element) for element in value]) assert len(serialized_bytes) < MAX_LENGTH serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') return serialized_length + serialized_bytes ``` -#### Tuples - -```python -serialized_elements = [serialize(element) for element in object] -serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) -return serialized_bytes -``` - -#### Lists - -```python -serialized_elements = [serialize(element) for element in object] -serialized_bytes = reduce(lambda x, y: x + y, serialized_elements) -assert len(serialized_elements) < MAX_LENGTH -serialized_length = len(serialized_elements).to_bytes(LENGTH_BYTES, 'little') -return serialized_length + serialized_bytes -``` - ## Deserialization Given a type, serialization is an injective function from objects of that type to byte strings. That is, deserialization—the inverse function—is well-defined. @@ -114,12 +93,12 @@ We first define helper functions: * `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (32-byte little-endian serialization) return `hash(root + length)`. -Let `object` be an object. We now define object Merkleization `hash_tree_root(object)` recursively: +Let `value` be an object. We now define object Merkleization `hash_tree_root(value)` recursively: -* `merkleize(pack(object))` if `object` is a basic object or a tuple of basic objects -* `mix_in_length(merkleize(pack(object)), len(object))` if `object` is a list of basic objects -* `merkleize([hash_tree_root(element) for element in object])` if `object` is a tuple of composite objects or a container -* `mix_in_length(merkleize([hash_tree_root(element) for element in object]), len(object))` if `object` is a list of composite objects +* `merkleize(pack(value))` if `value` is a basic object or a tuple of basic objects +* `mix_in_length(merkleize(pack(value)), len(value))` if `value` is a list of basic objects +* `merkleize([hash_tree_root(element) for element in value])` if `value` is a tuple of composite objects or a container +* `mix_in_length(merkleize([hash_tree_root(element) for element in value]), len(value))` if `value` is a list of composite objects ## Self-signed containers From aa676354a049b8a2812c237861b0d7e9c02a83c6 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 17:54:19 +0100 Subject: [PATCH 11/22] Update simple-serialize.md --- specs/simple-serialize.md | 29 +++++++---------------------- 1 file changed, 7 insertions(+), 22 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 3dc9b0b077..639231b303 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -4,11 +4,9 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ## Table of contents -- [Constants](#constants) - [Typing](#typing) - [Basic types](#basic-types) - [Composite types](#composite-types) - - [Notation](#notation) - [Aliases](#aliases) - [Serialization](#serialization) - [`uintN`](#uintn) @@ -19,31 +17,18 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Self-signed containers](#self-signed-containers) - [Implementations](#implementations) -## Constants - -| Name | Value | Description | -|-|-|-| -| `LENGTH_BYTES` | `4` | Number of bytes for the length of variable-length serialized objects. | -| `MAX_LENGTH` | `2**(8 * LENGTH_BYTES)` | Maximum serialization length. | - ## Typing -#### Basic types +### Basic types * `uintN`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) * `bool`: 1-bit unsigned integer -#### Composite types - -* **container**: ordered heterogenous collection of values -* **tuple**: ordered fixed-length homogeneous collection of values -* **list**: ordered variable-length homogenous collection of values - -#### Notation +### Composite types -* **container**: key-pair curly braket notation `{}` (e.g. `{'key1': uint64, 'key2': bool}`) -* **tuple**: angle braket notation `[N]` (e.g. `uint64[N]`) -* **list**: angle braket notation `[]` (e.g. `uint64[]`) +* **container**: ordered heterogenous collection of values (key-pair curly braket notation `{}`, e.g. `{'foo': uint64, 'bar': bool}`) +* **tuple**: ordered fixed-length homogeneous collection of values (angle braket notation `[N]`, e.g. `uint64[N]`) +* **list**: ordered variable-length homogenous collection of values (angle braket notation `[]`, e.g. `uint64[]`) #### Aliases @@ -52,7 +37,6 @@ For convenience we alias: * `byte` to `uint8` * `bytes` to `byte[]` * `bytesN` to `byte[N]` -* `bit` to `bool` ## Serialization @@ -75,8 +59,9 @@ return b'\x01' if value is True else b'\x00' #### Containers, tuples, lists ```python +LENGTH_BYTES = 4 serialized_bytes = ''.join([serialize(element) for element in value]) -assert len(serialized_bytes) < MAX_LENGTH +assert len(serialized_bytes) < 2**(8 * LENGTH_BYTES) serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') return serialized_length + serialized_bytes ``` From 54a81a8ebf7f328f0d2183f9f822854a756e37ae Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 17:56:51 +0100 Subject: [PATCH 12/22] Update simple-serialize.md --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 639231b303..2c0bb8f6a1 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -59,8 +59,8 @@ return b'\x01' if value is True else b'\x00' #### Containers, tuples, lists ```python -LENGTH_BYTES = 4 serialized_bytes = ''.join([serialize(element) for element in value]) +LENGTH_BYTES = 4 assert len(serialized_bytes) < 2**(8 * LENGTH_BYTES) serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') return serialized_length + serialized_bytes @@ -78,7 +78,7 @@ We first define helper functions: * `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (32-byte little-endian serialization) return `hash(root + length)`. -Let `value` be an object. We now define object Merkleization `hash_tree_root(value)` recursively: +We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: * `merkleize(pack(value))` if `value` is a basic object or a tuple of basic objects * `mix_in_length(merkleize(pack(value)), len(value))` if `value` is a list of basic objects From e76619358828638cf2cdf495e5029611a16861d9 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 17:59:49 +0100 Subject: [PATCH 13/22] Update simple-serialize.md --- specs/simple-serialize.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 2c0bb8f6a1..7f6866cb5d 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -26,9 +26,12 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ### Composite types -* **container**: ordered heterogenous collection of values (key-pair curly braket notation `{}`, e.g. `{'foo': uint64, 'bar': bool}`) -* **tuple**: ordered fixed-length homogeneous collection of values (angle braket notation `[N]`, e.g. `uint64[N]`) -* **list**: ordered variable-length homogenous collection of values (angle braket notation `[]`, e.g. `uint64[]`) +* **container**: ordered heterogenous collection of values + * key-pair curly braket notation `{}`, e.g. `{'foo': uint64, 'bar': bool}` +* **tuple**: ordered fixed-length homogeneous collection of values + * angle braket notation `[N]`, e.g. `uint64[N]` +* **list**: ordered variable-length homogenous collection of values + * angle braket notation `[]`, e.g. `uint64[]` #### Aliases @@ -40,7 +43,7 @@ For convenience we alias: ## Serialization -We reccursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. +We reccursively define the serialisation `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. #### `uintN` From 99074d09ecba1c02182e23496fcc638f289b2722 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 18:00:49 +0100 Subject: [PATCH 14/22] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 7f6866cb5d..565b0d788c 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -33,7 +33,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio * **list**: ordered variable-length homogenous collection of values * angle braket notation `[]`, e.g. `uint64[]` -#### Aliases +### Aliases For convenience we alias: From bac352e70f825da9ac5a7155a5a8581033de15c3 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 18:02:53 +0100 Subject: [PATCH 15/22] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 565b0d788c..e2ce4f208e 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -43,7 +43,7 @@ For convenience we alias: ## Serialization -We reccursively define the serialisation `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. +We reccursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. #### `uintN` From 8176cc4cf0c9eaee68abbf24ca4344dd1fac90e1 Mon Sep 17 00:00:00 2001 From: Danny Ryan Date: Wed, 27 Feb 2019 22:23:42 +0100 Subject: [PATCH 16/22] Apply suggestions from code review Co-Authored-By: JustinDrake --- specs/simple-serialize.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index e2ce4f208e..221a481441 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -43,23 +43,23 @@ For convenience we alias: ## Serialization -We reccursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. +We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. -#### `uintN` +### `uintN` ```python assert N in [8, 16, 32, 64, 128, 256] return value.to_bytes(N // 8, 'little') ``` -#### `bool` +### `bool` ```python assert value in (True, False) return b'\x01' if value is True else b'\x00' ``` -#### Containers, tuples, lists +### Containers, tuples, lists ```python serialized_bytes = ''.join([serialize(element) for element in value]) @@ -79,7 +79,7 @@ We first define helper functions: * `pack`: Given ordered objects of the same basic type, serialize them, pack them into 32-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. * `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. -* `mix_in_length`: Given a Merkle root `root` and a length `length` (32-byte little-endian serialization) return `hash(root + length)`. +* `mix_in_length`: Given a Merkle root `root` and a length `length` (`uint256` little-endian serialization) return `hash(root + length)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: From 74349eacd25c96d2678d4a0afbc2ce8f4436dcc4 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 22:33:36 +0100 Subject: [PATCH 17/22] Update simple-serialize.md --- specs/simple-serialize.md | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 221a481441..fb5ee23ae7 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -4,6 +4,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ## Table of contents +- [Constants](#constants) - [Typing](#typing) - [Basic types](#basic-types) - [Composite types](#composite-types) @@ -17,8 +18,14 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Self-signed containers](#self-signed-containers) - [Implementations](#implementations) -## Typing +## Constants + +| Name | Value | Description | +|-|-|-| +| `BYTES_PER_CHUNK` | `32` | Number of bytes per chunk. +| `BYTES_PER_LENGTH_PREFIX` | `4` | Number of bytes per serialized length prefix. | +## Typing ### Basic types * `uintN`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) @@ -27,7 +34,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ### Composite types * **container**: ordered heterogenous collection of values - * key-pair curly braket notation `{}`, e.g. `{'foo': uint64, 'bar': bool}` + * key-pair curly braket notation `{}`, e.g. `{'foo': "uint64", 'bar': "bool"}` * **tuple**: ordered fixed-length homogeneous collection of values * angle braket notation `[N]`, e.g. `uint64[N]` * **list**: ordered variable-length homogenous collection of values @@ -63,9 +70,8 @@ return b'\x01' if value is True else b'\x00' ```python serialized_bytes = ''.join([serialize(element) for element in value]) -LENGTH_BYTES = 4 -assert len(serialized_bytes) < 2**(8 * LENGTH_BYTES) -serialized_length = len(serialized_bytes).to_bytes(LENGTH_BYTES, 'little') +assert len(serialized_bytes) < 2**(8 * BYTES_PER_LENGTH_PREFIX) +serialized_length = len(serialized_bytes).to_bytes(BYTES_PER_LENGTH_PREFIX, 'little') return serialized_length + serialized_bytes ``` @@ -77,8 +83,8 @@ Given a type, serialization is an injective function from objects of that type t We first define helper functions: -* `pack`: Given ordered objects of the same basic type, serialize them, pack them into 32-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. -* `merkleize`: Given ordered 32-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. +* `pack`: Given ordered objects of the same basic type, serialize them, pack them into BYTES_PER_CHUNK-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. +* `merkleize`: Given ordered BYTES_PER_CHUNK-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (`uint256` little-endian serialization) return `hash(root + length)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: From 3e6934b602d58656ffe4d5a31b0c238ce74cbb66 Mon Sep 17 00:00:00 2001 From: Justin Date: Wed, 27 Feb 2019 22:44:20 +0100 Subject: [PATCH 18/22] Update simple-serialize.md --- specs/simple-serialize.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index fb5ee23ae7..6f3186aee5 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -83,8 +83,8 @@ Given a type, serialization is an injective function from objects of that type t We first define helper functions: -* `pack`: Given ordered objects of the same basic type, serialize them, pack them into BYTES_PER_CHUNK-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. -* `merkleize`: Given ordered BYTES_PER_CHUNK-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. +* `pack`: Given ordered objects of the same basic type, serialize them, pack them into `BYTES_PER_CHUNK`-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. +* `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (`uint256` little-endian serialization) return `hash(root + length)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: From 7ab7abe2ca7f66ac75e854c20b4ff0ef26cab1f7 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 28 Feb 2019 11:32:54 +0100 Subject: [PATCH 19/22] Update simple-serialize.md --- specs/simple-serialize.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 6f3186aee5..8830ae7e41 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -34,11 +34,11 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ### Composite types * **container**: ordered heterogenous collection of values - * key-pair curly braket notation `{}`, e.g. `{'foo': "uint64", 'bar': "bool"}` + * key-pair curly bracket notation `{}`, e.g. `{'foo': "uint64", 'bar': "bool"}` * **tuple**: ordered fixed-length homogeneous collection of values - * angle braket notation `[N]`, e.g. `uint64[N]` + * angle bracket notation `[N]`, e.g. `uint64[N]` * **list**: ordered variable-length homogenous collection of values - * angle braket notation `[]`, e.g. `uint64[]` + * angle bracket notation `[]`, e.g. `uint64[]` ### Aliases @@ -50,7 +50,9 @@ For convenience we alias: ## Serialization -We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a byte string of type `bytes`. +We recursively define the `serialize` function which consumes an object `value` (of the type specified) and returns a bytestring of type `bytes`. + +*Note*: In the function definitions below (`serialize`, `hash_tree_root`, `signed_root`, etc.) objects implicitly carry their type. ### `uintN` @@ -77,14 +79,14 @@ return serialized_length + serialized_bytes ## Deserialization -Given a type, serialization is an injective function from objects of that type to byte strings. That is, deserialization—the inverse function—is well-defined. +Because serialization is an injective function (i.e. two distinct objects of the same type will serialize to different values) any bytestring has at most one object it could deserialize to. Efficient algorithms for computing this object can be found in [the implementations](#implementations). ## Merkleization We first define helper functions: * `pack`: Given ordered objects of the same basic type, serialize them, pack them into `BYTES_PER_CHUNK`-byte chunks, right-pad the last chunk with zero bytes, and return the chunks. -* `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, right-pad them with zero chunks to the next power of two, Merkleize the chunks, and return the root. +* `merkleize`: Given ordered `BYTES_PER_CHUNK`-byte chunks, if necessary append zero chunks so that the number of chunks is a power of two, Merkleize the chunks, and return the root. * `mix_in_length`: Given a Merkle root `root` and a length `length` (`uint256` little-endian serialization) return `hash(root + length)`. We now define Merkleization `hash_tree_root(value)` of an object `value` recursively: @@ -96,7 +98,7 @@ We now define Merkleization `hash_tree_root(value)` of an object `value` recursi ## Self-signed containers -Let `container` be a self-signed container object. The convention is that the signature (e.g. a `bytes96` BLS12-381 signature) be the last field of `container`. Further, the signed message for `container` is `signed_root(container) = hash_tree_root(truncate_last(container))` where `truncate_last` truncates the last element of `container`. +Let `value` be a self-signed container object. The convention is that the signature (e.g. a `bytes96` BLS12-381 signature) be the last field of `value`. Further, the signed message for `value` is `signed_root(value) = hash_tree_root(truncate_last(value))` where `truncate_last` truncates the last element of `value`. ## Implementations From 67793edc8d6c1b9c831c0b33246a442ceb69d3d3 Mon Sep 17 00:00:00 2001 From: Justin Date: Thu, 28 Feb 2019 11:34:24 +0100 Subject: [PATCH 20/22] Update simple-serialize.md --- specs/simple-serialize.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 8830ae7e41..faba9e762d 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -29,7 +29,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio ### Basic types * `uintN`: `N`-bit unsigned integer (where `N in [8, 16, 32, 64, 128, 256]`) -* `bool`: 1-bit unsigned integer +* `bool`: `True` or `False` ### Composite types From 25ef2553c790add9b66f0c5d5eb67bfe0ddc4839 Mon Sep 17 00:00:00 2001 From: Justin Date: Fri, 1 Mar 2019 12:02:29 +0100 Subject: [PATCH 21/22] Update simple-serialize.md --- specs/simple-serialize.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index faba9e762d..31839b27ef 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -12,7 +12,8 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Serialization](#serialization) - [`uintN`](#uintn) - [`bool`](#bool) - - [Containers, tuples, lists](#containers-tuples-lists) + - [Tuples](#lists) + - [Containers, lists](#containers-lists) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Self-signed containers](#self-signed-containers) @@ -68,7 +69,13 @@ assert value in (True, False) return b'\x01' if value is True else b'\x00' ``` -### Containers, tuples, lists +### Tuples + +```python +return ''.join([serialize(element) for element in value]) +``` + +### Containers, lists ```python serialized_bytes = ''.join([serialize(element) for element in value]) From 57971aacb42a406c7c547a754e4682d727383550 Mon Sep 17 00:00:00 2001 From: Justin Date: Tue, 5 Mar 2019 15:20:36 +0100 Subject: [PATCH 22/22] Update simple-serialize.md --- specs/simple-serialize.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/specs/simple-serialize.md b/specs/simple-serialize.md index 31839b27ef..a452213b5f 100644 --- a/specs/simple-serialize.md +++ b/specs/simple-serialize.md @@ -12,8 +12,7 @@ This is a **work in progress** describing typing, serialization and Merkleizatio - [Serialization](#serialization) - [`uintN`](#uintn) - [`bool`](#bool) - - [Tuples](#lists) - - [Containers, lists](#containers-lists) + - [Tuples, containers, lists](#tuples-containers-lists) - [Deserialization](#deserialization) - [Merkleization](#merkleization) - [Self-signed containers](#self-signed-containers) @@ -69,13 +68,15 @@ assert value in (True, False) return b'\x01' if value is True else b'\x00' ``` -### Tuples +### Tuples, containers, lists + +If `value` is fixed-length (i.e. does not embed a list): ```python return ''.join([serialize(element) for element in value]) ``` -### Containers, lists +If `value` is variable-length (i.e. embeds a list): ```python serialized_bytes = ''.join([serialize(element) for element in value])