Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consistent JSON serialization of List[uint8, ...] #219

Open
arnetheduck opened this issue May 23, 2022 · 3 comments
Open

Consistent JSON serialization of List[uint8, ...] #219

arnetheduck opened this issue May 23, 2022 · 3 comments

Comments

@arnetheduck
Copy link
Contributor

arnetheduck commented May 23, 2022

List[uint8, ...] can be thought of either as byte string or a collection of (small) integers. In the former interpretation, one would serialize it as "0xabcd" whereas the latter would be [0xab, 0xcd] (string vs list).

In bellatrix, we encode ExtraData as a string - we also encode all Vector[byte..] as strings, for example hashes. However, in the BeaconState for bellatrix+, *_epoch_participation is also a List[byte, ...] (by virtue of ParticipationFlags and byte being aliases for uint8) - the encoding of the Beacon State is left unspecified opening the scene for ambiguity uses the list encoding (correction).

In general, it seems preferable to encode all List[byte, N] as "0x..." together with Vector[byte, ..] as strings for consistency - the alternative is that each field that ventures near the uint8 type gets its own serialization is messier.

@ralexstokes
Copy link
Member

There is a specification for how to encode the data:

For epoch_participation in Altair and later:
https://github.com/ethereum/beacon-APIs/blob/master/types/altair/state.yaml#L74-L77
https://github.com/ethereum/beacon-APIs/blob/master/types/altair/epoch_participation.yaml#L6
https://github.com/ethereum/beacon-APIs/blob/master/types/primitive.yaml#L71

has a JSON list of strings with decimal digits for a uint8 type

and for ExtraData:
https://github.com/ethereum/beacon-APIs/blob/master/types/primitive.yaml#L89


separately I do agree with you that "tightly packing" the List[uint8, ...] is better for performance so I'd support a PR that added this change :)

@arnetheduck
Copy link
Contributor Author

There is a specification for how to encode the data:

Ah, I missed that one for some reason 👍 Hm. Changing is messy, but so is having different encodings for the same thing :/

@ajsutton
Copy link
Contributor

Worth noting that in Teku we generate our JSON representations based off of the SSZ schema definitions and check they're right by parsing the YAML files in the reference tests (we use Jackson which supports both JSON and YAML support based on the same definitions). So I've got no problem changing and a more compact representation would certainly be good - especially since the participation flags are particularly human friendly even in List[number] form. But please can we keep the representation the same between the API and the reference tests.

@arnetheduck arnetheduck changed the title Ambiguous JSON serialization of List[uint8, ...] Consistent JSON serialization of List[uint8, ...] May 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants
@ajsutton @arnetheduck @ralexstokes and others