DAG-CBOR/CBOR vs DAG-JSON/JSON format inconsistencies

[achingbrain]

If I have a DAG-CBOR block with a CID in it, it has a tag 42. This tag is not present in "vanilla" CBOR.

Currently if I request a DAG-CBOR block that contains tag 42 in the cbor format from Kubo, it happily serves me binary with tag 42 present which I then cannot deserialize it using only a vanilla CBOR parser, so the returned value is invalid, or at least close to invalid.

Compare this to DAG-JSON which may contain serialized CIDs as { "/": "QmFoo" }, which can be lossily deserialized using a vanilla JSON parser.

If the user has explicitly requested a DAG-CBOR CID in the cbor format, perhaps any CIDs present should be converted to A1 61 2F 65 Qmfoo (e.g. { "/": "QmFoo"}) so it is parsable by a CBOR parser?

Then the user could arrive back at the original bytes if they wanted to via:

1. Convert DAG-CBOR bytes to CBOR bytes
2. Deserialize CBOR bytes as object
3. Serialize object to JSON
4. Deserialize JSON to object using DAG-JSON parser
5. Serialize object to DAG-CBOR

Alternatively the gateway should return a 406 if tag 42 is present in a block requested in the cbor format, since it cannot be represented as vanilla CBOR, and the user can request dag-cbor, which means they are ready and prepared to handle tag 42.

[lidel]

This is tricky space.

Every dag-cbor is a valid cbor (every square is a valid rectangle). Tags are part of CBOR and tag 42 is at IANA. Parsers are free to error or skip unknown ones. I don't think we should mutate valid CBOR.

IIUC turning Tag 42 into a map that mirrors { "/": "QmFoo" } convention from DAG-JSON would be our invention, not part of any spec. This means we would have to take over maintenance of IPLD specs for DAG-CBOR and update them. We could do it, but it is extra work.

Taking step back:

• Is the problem that the library you use does not support unknown tags?
• Are you attempting to parse if CID has codec cbor?

If useful, what we do in golang, is we skip CBOR parsing if codec in requested CID matches the requested format.

For example, iirc, if CID codec is cbor or dag-cbor, and that CID is requested as ?format=cbor boxo/gateway returns exactly the same bytes as ?format=raw. This might've changed since I looked at this code, but the intention was to not attempt to parse block if response format matched, and just returns it as-is.

Could we do the same in JS to avoid changing existing behavior?

I don't think we have bandwidth to fix spec gaps IPLD project failed to address half a decade ago, so my intuition is to either do what GO does, or error, but never mutate a valid CBOR.

Thoughts?

[achingbrain]

Parsers are free to error or skip unknown ones. I don't think we should mutate valid CBOR

I guess a better analogy here is HTML, which is designed to be extensible rather than JSON which isn't. That is, a server can supply any old rubbish (or new tags, etc) and call it text/html and it's up to the browser to render what it can and ignore what it cannot. Maybe it's fine. On balance I do agree it's a good idea not to mutate data.

but the intention was to not attempt to parse block

The server should make a basic effort to ensure that the returned content can be represented in the format requested by the client, otherwise it should 406. It should return the underlying bytes though, rather than (for example) re-serializing the parsed object as it would cause it to omit any tags it doesn't understand when the client may be equiped to parse them.