Documentation omission: compactSize encodings must be canonical

[daira]

The documentation of a compactSize field in the Bitcoin Developer Reference describes four possible encodings based on length. The ranges of integers expressible in each encoding overlap. Therefore it is possible in principle to represent a compactSize with a non-canonical encoding that uses more than the minimum number of bytes.

The actual implementation of ReadCompactSize excludes this possibility. This is also tested. However, it is not documented. Since it affects consensus, it should be documented.

This issue was filed first at bitcoin/bitcoin#8721 and moved here. I believe the assertion by @TheBlueMatt on that issue that this is only a p2p network rule rather than a consensus rule is inaccurate. For example, the Raw Transaction Format section says:

Bitcoin transactions are broadcast between peers in a serialized byte format, called raw format. It is this form of a transaction which is SHA256(SHA256()) hashed to create the TXID and, ultimately, the merkle root of a block containing the transaction—making the transaction format part of the consensus rules.

and this does not make clear that the uses of the serialized form in consensus checks are of a reserialization, not the serialization that was "broadcast between peers". But this is in some sense academic since it should be documented either way.

Ref: zcash/zcash#842 , which is a potential security bug that the Zcash fork of Bitcoin would have had if canonicity were not enforced.

[wbnns]

Thanks for mentioning, @daira - would you be able to submit a PR to document this?

[wbnns]

Bounty: 5,000 bits / ~$10 USD