ssz: `byte` type and canonical JSON mapping #3506

arnetheduck · 2023-09-19T12:12:52Z

This PR introduces a new byte type equivalent in all aspects to uint8 except that it has additional intent and display semantics attached.

On top of this, the PR adds a canonical JSON mapping to the SSZ specification, documenting current usage of JSON in tests, API:s and simplifying future interop work between clients and adjacent specifications such as the Beacon API. The encoding is appropriate to use with YAML as well.

As an important property, this mapping contains a 1:1 mapping of SSZ type to JSON encoding - this allows round-tripping any object between JSON and SSZ based on the SSZ schema and usage of the core SSZ types alone.

The encoding presented in this PR is used in tests and API:s with one exception: the ParticipationFlags type from the Altair spec - it is recommended we switch encoding in tests and eventually the beacon API to address this irregularity, so as to avoid a proliferation "special" primitive types in the SSZ spec that only appear in particular schemas (and thus making validating general-purpose SSZ/JSON parsers more complex) as well as differences in encoding between fields of the same SSZ type.

Edit: the change in encoding has been removed from this PR (could be done separately down the line if desired)

The PR also clarifies that the introduction of new aliases does not lead to changes in their canonical JSON specification - this allows building general SSZ/JSON libraries that do not further depend on open-ended knowledge about aliases.

This PR should be seen as an alternative to
#2983.

This PR introduces a new `byte` type equivalent in all aspects to `uint8` except that it has additional intent and display semantics attached. On top of this, the PR adds a canonical JSON mapping to the SSZ specification, documenting current usage of JSON in tests, API:s and simplifying future interop work between clients and adjacent specifications such as the Beacon API. The encoding is appropriate to use with YAML as well. As an important property, this mapping contains a 1:1 mapping of SSZ type to JSON encoding - this allows round-tripping any object between JSON and SSZ based on the SSZ schema and usage of the core SSZ types alone. The encoding presented in this PR is used in tests and API:s with one exception: the `ParticipationFlags` type from the Altair spec - it is recommended we switch encoding in tests and eventually the beacon API to address this irregularity, so as to avoid a proliferation "special" primitive types in the SSZ spec that only appear in particular schemas (and thus making validating general-purpose `SSZ/JSON` parsers more complex) as well as differences in encoding between fields of the same SSZ type. The PR also clarifies that the introduction of new aliases does not lead to changes in their canonical JSON specification - this allows building general SSZ/JSON libraries that do not further depend on open-ended knowledge about aliases. This PR should be seen as an alternative to ethereum#2983.

arnetheduck · 2023-09-19T12:16:46Z

cc @ajsutton @dapplion @mcdee

protolambda · 2023-09-19T12:40:17Z

ssz/simple-serialize.md

+Integers are encoded as strings to avoid loss of precision in 64-bit values.
+
+Aliases are encoded as their underlying type.
+
+`hex-byte-string` is a `0x`-prefixed hex encoding of byte data, as it would appear in an SSZ stream.
+
+`List` and `Vector` of `byte` (and aliases thereof) are encoded as `hex-byte-string`. `Bitlist` and `Bitvector` similarly map their SSZ-byte encodings to a `hex-byte-string`.
+
+`Union` is encoded as an object with a `selector` and `data` field, where the contents of `data` change according to the selector.
+
+> This encoding is used in [beacon-APIs](https://github.com/ethereum/beacon-APIs) with one exception: the `ParticipationFlags` type for the `getStateV2` response, although it is an alias of `uint8`, is encoded as a list of numbers. Future versions of the beacon API may address this incompatibility.


Was it not previously agreed to encode uint8, uint16, uint32 as numbers (because they fit in the JS size), and only encode uint64 as string? I believe that's why ParticipationFlags is the way it is.
AFAIK the specs do not encode uint8, uint16, uint32 anywhere else, so changing it wouldn't be too difficult, but I still prefer to avoid unnecessary number stringification.

Was it not previously agreed to encode uint8, uint16, uint32 as numbers

no idea and no strong opinion either way.

AFAIK the specs do not encode uint8

https://github.com/ethereum/consensus-specs/blob/dev/specs/deneb/beacon-chain.md#blob is the only (other) offender I can find that potentially also should be byte-ified.

I believe that's why ParticipationFlags is the way it is.

I'd pin it on lack of interest since the two types were the same in this repo ;) ie the python code liberally uses uint8 where byte is intended for simplicity since the two are after all... the same. Flags are not numbers in most CS-inspired interpretations of numbers that I can think of so byte it becomes. We'll have to deal with the fallout at some point maybe, but this way we start with a clean slate at spec level.

specs/altair/beacon-chain.md

mcdee · 2023-09-21T08:50:51Z

Was it not previously agreed to encode uint8, uint16, uint32 as numbers (because they fit in the JS size), and only encode uint64 as string?

No, all integers are meant to be encoded as quoted decimal values.

I'm fine with the change, except retrospectively altering the spec which will have downstream effects and I don't see bringing any value.

dapplion · 2023-09-25T21:22:51Z

I'm fine with the change, except retrospectively altering the spec which will have downstream effects and I don't see bringing any value.

It's very useful to have a canonical representation of SSZ as JSON/YAML. All spec tests render participation flags as byte strings, so naming adjustment aligns with current usage.

mcdee · 2023-09-25T23:08:57Z

All spec tests render participation flags as byte strings, so naming adjustment aligns with current usage.

I don't see this in values within the consensus spec tests repo, for example:

grep participation tests/mainnet/altair/ssz_static/BeaconState/ssz_random/case_0/value.yaml
previous_epoch_participation: [93, 102]
current_epoch_participation: [126, 191, 230]

And querying /eth/v2/debug/beacon/states/finalized on beacon nodes also returns lists of (quoted) integers.

This can be done separately in the future, if wanted

djrtwo

No fundamental opposition from be, barring any significant downstream affects in tools, apis, etc.

dapplion · 2023-11-08T11:25:50Z

barring any significant downstream affects in tools, apis, etc.

PR's content just document existing practices so should not cause any practical change

ssz/simple-serialize.md

rolfyone · 2023-11-09T22:07:36Z

No fundamental opposition from be, barring any significant downstream affects in tools, apis, etc.

This exactly. I don't have a strong objection with the assumption its not going to need a ton of downstream changes.

dapplion · 2024-01-09T05:27:02Z

To further motivate this PR

Example of how this canonical mapping can be used to derive beacon APIs OpenAPI type schemas from the pyspec Derive OpenAPI type schema from pyspec beacon-APIs#402

djrtwo · 2024-01-11T14:13:36Z

to be merged after ACDC 2024/1/11

spelling bee and lint torture

f2ad012

protolambda reviewed Sep 19, 2023

View reviewed changes

hwwhww added scope:SSZ Simple Serialize Altair aka HF1 labels Sep 21, 2023

mcdee reviewed Sep 21, 2023

View reviewed changes

specs/altair/beacon-chain.md Outdated Show resolved Hide resolved

arnetheduck mentioned this pull request Sep 21, 2023

Move state to beacon ethereum/beacon-APIs#357

Closed

revert ParticipationFlags back to uint8

66082e3

This can be done separately in the future, if wanted

arnetheduck force-pushed the canonical-json-byte branch from f8c173c to 66082e3 Compare October 17, 2023 09:41

arnetheduck mentioned this pull request Nov 1, 2023

Consensus-layer Call 121 ethereum/pm#898

Closed

djrtwo reviewed Nov 1, 2023

View reviewed changes

dapplion approved these changes Nov 8, 2023

View reviewed changes

mcdee reviewed Nov 8, 2023

View reviewed changes

ssz/simple-serialize.md Outdated Show resolved Hide resolved

rkapka approved these changes Nov 8, 2023

View reviewed changes

remove obsolete comment

04f5ec5

mcdee approved these changes Nov 9, 2023

View reviewed changes

dapplion mentioned this pull request Jan 8, 2024

Derive OpenAPI type schema from pyspec ethereum/beacon-APIs#402

Open

1 task

djrtwo approved these changes Jan 11, 2024

View reviewed changes

djrtwo merged commit 9f533cf into ethereum:dev Jan 11, 2024
26 checks passed

djrtwo mentioned this pull request Jan 16, 2024

release v1.4.0-beta.6 #3578

Merged

8 tasks

rkapka mentioned this pull request Jan 24, 2024

Make REST VC the default prysmaticlabs/prysm#13519

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ssz: `byte` type and canonical JSON mapping #3506

ssz: `byte` type and canonical JSON mapping #3506

arnetheduck commented Sep 19, 2023 •

edited

arnetheduck commented Sep 19, 2023 •

edited

protolambda Sep 19, 2023

arnetheduck Sep 19, 2023

mcdee commented Sep 21, 2023

dapplion commented Sep 25, 2023

mcdee commented Sep 25, 2023

djrtwo left a comment

dapplion commented Nov 8, 2023

rolfyone commented Nov 9, 2023

dapplion commented Jan 9, 2024

djrtwo commented Jan 11, 2024

ssz: byte type and canonical JSON mapping #3506

ssz: byte type and canonical JSON mapping #3506

Conversation

arnetheduck commented Sep 19, 2023 • edited

arnetheduck commented Sep 19, 2023 • edited

protolambda Sep 19, 2023

Choose a reason for hiding this comment

arnetheduck Sep 19, 2023

Choose a reason for hiding this comment

mcdee commented Sep 21, 2023

dapplion commented Sep 25, 2023

mcdee commented Sep 25, 2023

djrtwo left a comment

Choose a reason for hiding this comment

dapplion commented Nov 8, 2023

rolfyone commented Nov 9, 2023

dapplion commented Jan 9, 2024

djrtwo commented Jan 11, 2024

ssz: `byte` type and canonical JSON mapping #3506

ssz: `byte` type and canonical JSON mapping #3506

arnetheduck commented Sep 19, 2023 •

edited

arnetheduck commented Sep 19, 2023 •

edited