Is consensus encoding necessary to serialise contracts?

[chm-diederichs]

As RGB contracts are not parsed by Bitcoin nodes directly, is it necessary to serialise using consensus encoding? Consensus encoding seems underspecified, and structs are only defined implicitly in the custom encoder.
Protocol buffers, as used in LN, provide freedom to update contract headers in the future as more contract types are introduced. They are also well defined structures, making it easier to integrate across platforms.

[dr-orlovsky]

I think this question is more related to the specification than the actual implementation, following the spec. So I will move it to the spec repo – we have a "Networking" part there (unwritten), where it will be nice to address this issue and provide some explanations.

[dr-orlovsky]

Citing my explanation elsewhere, we had this discussion some time ago in RGB Telegram channel.

1. starting here: https://t.me/rgbtelegram/1470
2. continuation here: https://t.me/rgbtelegram/1501
3. End result/conclusion: https://t.me/rgbtelegram/1520

[petertodd]

Protocol buffers isn't well designed for hashed data; consensus encoding is. I'm not too familiar with exactly what LN is using protocol buffers for. But it's likely not applicable to this usecase.

…

On June 26, 2019 11:47:54 PM GMT+01:00, Christophe Diederichs ***@***.***> wrote: As RGB contracts are not parsed by Bitcoin nodes directly, is it necessary to serialise using consensus encoding? Consensus encoding seems underspecified, and structs are only defined implicitly in the custom encoder. Protocol buffers, as used in LN, provide freedom to update contract headers in the future as more contract types are introduced. They are also well defined structures, making it easier to integrate across platforms. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #84

[dr-orlovsky]

@petertodd Last time I checked, LND used protobufs for building its gRPC interface. It has nothing to do with P2P communications, just one of endpoints (additionally to usual JSON-RPC).

Originally, LND was playing with protobufs for P2P in early releases, but lately, they've taken the decision to replace it with Bitcoin consensus encoding. Why? I didn't find their arguments, but I found arguments from Satoshi himself on protobufs:

The reason I didn't use protocol buffers or boost serialization is because they looked too complex to make absolutely airtight and secure. Their code is too large to read and be sure that there's no way to form an input that would do something unexpected.
I hate reinventing the wheel and only resorted to writing my own serialization routines reluctantly. The serialization format we have is as dead simple and flat as possible. There is no extra freedom in the way the input stream is formed. At each point, the next field in the data structure is expected. The only choices given are those that the receiver is expecting. There is versioning so upgrades are possible.
https://bitcointalk.org/index.php?topic=632.msg7090#msg7090

I do agree with Satoshi on cons:

1. higher security risks in terms of shifting control to a third-party that is hard to audit
2. higher security risk due to a hardness of auditing plenties of an automatically generated code (so basically we introduce a new layer of complexity/unpredictability/Turing completeness, which is really, really bad in terms of security)

Because financial systems are about security first, there is no reason for using protobufs or any other layered third-party solutions for building quite simple network communications.

Cons «weight» is much higher than of pros

[emilbayes]

@petertodd do you know if there is any work on formalising the specification of consensus encoding? Right now we are going with whatever the rust crate does, but it is not clearly defined anywhere what valid encodings look like (eg tuples)

[dr-orlovsky]

I'm afraid there is no specification — I've made my search and found none. The bitcoin encoding is defined as a way bitcoin core nodes encode their data in the bitcoin wire protocol — and thus Bitcoin Core C code is the only specification available. The encoding format is not designed to be universal encoding format supporting some different data types. We are free to extend it and call "RGB encoding" wherever we need; bitcoin encoding code from the rust library is just a very good starting point supporting nearly all data structures we need to support.

You can read more here rust-bitcoin/rust-bitcoin#262 (also follow the links in the discussion)