Structured responses section for streaming loading

[nyaxt]

Goal

• Easier streaming consumption of large bundles.
• Easier non-streaming consumption of the small bundles.

Background

In the current bundle spec, the structure of the responses section is allowed to be very flexible and not required to be a CBOR construct.

Streaming loading algorithm for current format

When loading a bundle in a streamed fashion, currently the parser will:

1. Consume the index section
2. Sort the exchanges by their response range offset
3. Consume the other sections up till the responses section
4. Discard the bytes until it reaches the first exchange in the sorted list
5. Buffer the bytes until it reaches the end of the the first exchange
6. The first exchange becomes ready for its use:
7. Parse its responses header CBOR map
8. Read its payload
9. Append bytes to the buffer until it reaches the end of the next item.
10. The second exchange becomes ready for its use.
11. ...

Since there may be overlap in the response exchange, we need to keep around the original responses section in the buffer (which probably will reside on memory), since they may theoretically be used more than 1 time in a different CBOR interpretation. It is non-trivial to compute the "done" part of the responses section which are not referenced in later exchanges to partially free the original responses bytes buffer.

Non-streaming consumption of the small bundles

If the entire bundle is guaranteed to be a valid CBOR, it will become very easy to consume small packages which will easily fit in memory. If efficiency is not an issue, you can feed the entire package to CBOR parser, and consume the exchanges without implementing the full parsing algorithm.

For example, @irori created a live web demo of unbundler which assumes that the bundler output a CBOR array as response section.

Proposals

• Specify the structure of the responses section
  • A valid CBOR array?
• An index section entry references the response via array index, not offset
  • Enforces the start of the response is aligned to the start of the item in the responses CBOR array
  • Can save an alignment assert in the parser
• Require the responses section to be always after the index section