canonicalize protobufs somehow.

[zenhack]

It would be nice if we could canonicalize typed objects somehow, so that logically identical values produce byte-for-byte identical blobs, allowing them to be correctly de-duped. I don't know if there's a standard way of canonicalizing protobufs? I wasn't able to find anything after a bit of searching.

Relatedly: @bobg, how wedded are you to protobufs in particular? If there is indeed no standard canonicalization algorithm, it may be worth considering other formats (I'm somewhat partial to capnproto).

[bobg]

Can't believe I've gotten this far without having heard about (a) capnproto or (b) the non-determinism of protobuf encoding.

I am now educating myself on these subjects. Thanks, and stay tuned.

[bobg]

I'm finally back to looking at this. I like the deterministic encoding of capnproto, but I hate the unidiomatic, type-unsafe Go produced by the capnproto Go compiler - everything is a capn.Struct. That makes it a non-starter as far as I'm concerned.

There are proposals out there for making protobuf encoding deterministic; e.g. https://docs.cosmos.network/master/architecture/adr-027-deterministic-protobuf-serialization.html. Maybe that's the way to go?

[zenhack]

type-unsafe Go produced by the capnproto Go compiler - everything is a capn.Struct

Not sure what your referring to? It should generate typed wrappers? I will grant that having to use getters & setters is less ergonomic than being able to just use struct literals though...

But if there is a canonicalization algorithm for pb out there that does what is needed, then that solves the problem too.

[bobg]

I have done away with the idea of typing blobs, and so protobufs are no longer first-class objects in BS. There are convenience functions for putting and getting them, and some structures (hashsplit trees, maps, etc) are defined using them, but they are no longer intrinsic to the BS data model.