Working document to capture requirements and proposed requirements for concrete syntax(es)

[jar398]

It is proposed that someone prepare a document that lists agreed (consensus) and proposed (non-consensus) requirements on whatever will end up being the proposed favored concrete syntax and binary syntax, regardless of its particular pedigree (smallcaps, syrup, etc.) and details. Ideally, accompany each putative requirement with use cases and brief discussion.

[jar398]

I wanted to cross reference #3 which seems closely related to this issue.

The requirement for a serialization format might be pretty simple, like some version of a round tripping requirement. Of course descriptions (designs) of particular serializations would be more involved than that.

[jar398]

The question of requirements has been raised again (#107)... I think there are a lot of assumptions bottled up in various minds that ought to be surfaced and written down (at least as an issue comment here).

[jar398]

We have made progress on abstract syntax / data model, see #92.

I suppose one requirement is that a concrete syntax/serialization should permit expression of the abstraction syntax. What else?

[jar398]

(Copying Dan's comment because it seems important here, since these other issues may bear on serialization requirements.)

@dckc writes on #107:

Is there any particular issue for concrete serialization format?

I find

• CapTP AST / data representation and serialization #3

which is perhaps broader, and

• Proposed wire format #63

which is maybe on target

and tangentially:

• Concrete syntax ordering guarantees #56

[tsyesika]

In March's OCapN meeting there were five tasks, several of them assigned to Spritely folks. The tasks were:

1. synthesize existing conversation from presentation and dialogue at meeting
2. Spritely proposes set of requirements based on that conversation (noting where we have consensus and non-consensus)
3. Write up "where is canonicalization useful" and "does c-list entries destroy canonicalization use"
4. Agoric reviews syrup/molasses for next meeting?
5. Reach out to implementers and get feedback.

So in an effort to tackle 1 and 2, here's my write up to summarise the discussion we had in the meeting and the requirements we landed on:

Requirements (MUST haves)

• It can express all the agreed abstract types
• Binary Data (part of abstract types but worth calling out on its own point).
• Canonicalization* (Don't have consensus!)

Nice to haves (but not hard requirements)

• Support for Range Queries
• Compactness
• Textual format for readability (both of us have this)
• Easy access through multiple languages
• Has access to good tooling to work with it (e.g. jq)

It seems like the biggest questions here are how does the format express binary data and should canonicalization be a requirement? Goblin's OCapN implementation relies upon canonicalization for its 3rd party handoff implementation (certificate based).

[tsyesika]

In March's OCapN meeting there were five tasks, several of them assigned to Spritely folks. The tasks were:

3. Write up "where is canonicalization useful" and "does c-list entries destroy canonicalization use"

So this is really two questions:

1. Where is canonicalization useful?

Canonicalization is used by Goblin's OCapN implementation to generate handoff certificates that are used in Third Party handoffs. These certificates are OCapN data structures which include a signature, the data can be deserialized, understood by OCapN and then re-serialized by check the signature.

Obviously a machine could keep around the blob it got on the wire, in addition to the serialized version, this way if it turned out it contained a certificate it could verify the blob it got rather than rely on being able to produce the same blob using a format with canonicalization. The issue here is that you must then store it twice so you can always check the signature.

2. How does this effect c-lists

The issue with c-lists and a signing a canonical serializtion that it contains references to a side table is that you're not signing the actual data, just references to some unsigned, unverified blob. To avoid this security issue you could use content addressing (referring to blobs by their hash therefore allowing checking against tampering). This is however a much more complicated solution than just using a binary format which allows for expressing the content inline and signing the entire document.

[dckc]

As discussed in the May meeting #114,

Agoric withdraws the ordering proposed requirement and accepts that the concrete syntax should carry bytes.

Hence the March 2023 syrup spec of March 2023
97bfd6b meets requirements that the OCapN group has gathered to date.

See also issue that @tsyesika agreed to open re adding to the locator syntax a name, or version marker, for concrete syntax.