Limit the commitment length

[A60AB5450353F40E]

I've had a chat with Calin, the length limit should be independent of Script length instead of eating into the Script limit with token data, much less mess, and 520 is a sane limit since anything above that would be unusable by contracts. If a future hard-fork adjusts the limit of stack items, then it will be an opportunity to adjust this in the same go.

With that outputs could have full 10,000b of Script + UTXO state (satoshi amount, token annotation)

see here: https://gitlab.com/bitcoin-cash-node/bitcoin-cash-node/-/merge_requests/1580/diffs?commit_id=cc3a38ca01a09427b9fc217692b19cea638730d5

[cculianu]

Just to expand on this -- the following is true currently:

• The only place that the MAX_SCRIPT_SIZE limit is applied in current BCH consensus is when evaluating the script and it's only applied to the byte-code being evaluated.
• This means it's possible to mine a non-standard txn with any script size, up to ~1MB (max transaction size).
• The output's locking will be >10,000 and will sit on the chain forever unspendable, but it will get mined and consensus accepts such scripts!
• Again, I stress -- the limit on scripts is only applied to the active byte code being executed as a check done before the interpreter "dives in" to the script itself (later when it is spent, not when it is created!).

As such it makes sense to not conflate anything about MAX_SCRIPT_SIZE with anything remotely to do with token data. The two pieces of data live independently in the node as concepts and as data.

The fact that we did a serialization "hack" to encode the token data within the serialized script blob is completely unrelated to the script limit!. Again.. the limit is applied only upon evaluation of the script byte code -- by that time the token data is completely decoupled from the script byte code. The fact that we happened to get the token data and the script some time ago using a "script blob serialization hack" is not relevant to the script interpreter at the point at which it is invoked, in other words.

So please @bitjson correct the spec to reflect this and to clarify this very important point.

Now, speaking to the rest of this issue that @A60AB5450353F40E was so kind to raise -- yes -- right now the max NFT commitment really should just be 520 bytes. Whenever we raise consensus limits on MAX_SCRIPT_ELEMENT_SIZE we can just raise that one too...

Also right now I am throwing hard deserialization error if I see a commitment >520. The spec should recommend refusing to even deserialize such "malformed" token data off the bat... So yes for now hard limit of 520 for the commitment would be nice.

Finally I reiterate: please correct the spec @bitjson not to conflate MAX_SCRIPT_SIZE with anything to do with tokens. They are (almost) unrelated concepts (for the reasons outlined in the bullet point list above).

[bitjson]

Hey @cculianu and @A60AB5450353F40E, thanks for opening this issue! Please see the rationale section I'm working on here:

Non-Fungible Token Commitment Length

This specification limits the length of non-fungible token commitments in standard transactions to 40 bytes; this restrains excess growth of the UTXO set and reduces the resource requirements of typical wallets and indexers.

By committing to a hash, contracts can commit to an unlimited collection of data (e.g. using a merkle tree). For resistance to birthday attacks, covenants should typically use 32 byte hashes. This proposal expands this minimum requirement by 8 bytes to include an additional (padded, maximum-length) VM number. This is particularly valuable for covenants that are optimized to use multiple types of commitment structures and must concisely indicate their current internal "mode" to other contracts (e.g. re-organizing an unbalanced merkle tree of contract state for efficiency when the covenant enters "voting" mode). This additional 8 bytes also provides adequate space for higher-level standards to specify prefixes for committed hashes (e.g. marking identifiers to content-addressable storage).

For nonstandard transactions, commitment length is constrained only by the existing 10,000 byte limit on locking bytecode length; this is the maximum commitment length that can be encoded while maintaining backwards-compatibility with existing transaction parsing implementations.

As nonstandard transactions may already contain outputs with maximum-length locking bytecode, any additional limitation on nonstandard commitment lengths would have no impact on UTXO set growth, but such a limit may prevent the standardness limit from being increased in future, backwards-compatible upgrades, resulting in unnecessary protocol complexity and higher implementation costs.

Specifically, a future upgrade may relax the standardness requirements of transaction outputs to allow arbitrary locking bytecode, allowing covenants to operate without the unnecessary overhead of Pay to Script Hash (P2SH). For a covenant to validate a P2SH output, the redeem bytecode must be public at payment time; including the hash of the redeem bytecode is strictly wasteful – it's only purpose is to ensure the transaction passes output standardness validation. For this same reason, some commitments managed by covenants will also be wasteful if contracts are forced (by standardness validation) to commit to hashes instead of slightly larger segments of raw data. Therefore if future upgrades relax locking bytecode standardness requirements, such upgrades may also relax commitment length standardness requirements. By supporting larger nonstandard commitment lengths, this proposal minimizes future protocol complexity and technical debt.

Alternatively, this proposal could limit commitment lengths in nonstandard transactions (by consensus) to 520 bytes – this lower limit would ensure that OP_UTXOTOKENCOMMITMENT and OP_OUTPUTTOKENCOMMITMENT can never fail due to an oversized push (exceeding the 520 byte stack item length limit). Limiting commitment lengths to prevent inspection operation failure would break from the existing behavior of all other inspection operations, where excessive length pushes already cause evaluation failure. Evaluation failure also enables backward-compatible expansion in future upgrades: if the stack item length limit is increased, the OP_*TOKENCOMMITMENT operations can safely support the new larger limit without impacting existing contracts (contracts can be written to rely on the failing behavior, but these contracts can only be interpreted as intentional; see Interpretation of Signature Preimage Inspection). Notably, most currently-known, post-quantum cryptographic signature schemes require public keys and signatures with lengths exceeding the existing stack item length limit; if a future upgrade added support for one or more of these signature schemes, it will likely also require an increase in the stack item length limit.

Because nonstandard transactions may include commitment lengths that exceed the current stack item length limit (520 bytes), the OP_*TOKENCOMMITMENT operations must be used with the same care required of OP_ACTIVEBYTECODE, OP_UTXOBYTECODE, OP_INPUTBYTECODE, and OP_OUTPUTBYTECODE; covenant developers must ensure that an attacker cannot force the covenant into a state in which further covenant execution is impossible (e.g. all spending paths require reading from an invalid input/output index or reading a commitment longer than 520 bytes). Though this is not typically a risk for covenants that validate output token commitments, it could be possible in rare cases where, for example, 1) the covenant requires reading from it's own token commitment for every available spending path, and 2) users are somehow allowed to attach arbitrary, unrelated tokens to the covenant without first reading from the attached token's commitment (in the transaction that initially attaches the token).

---

To respond directly to the discussion above:

Limiting nonstandard to 520 bytes would ensure that any future upgrade increasing size beyond 520 bytes will introduce significant technical debt:

1. It would require a new token encoding e.g. PREFIX_TOKEN2, as existing transaction decoding implementations cannot later be expected to support larger sizes, and
2. It would also break the "API" exposed by OP_*TOKENCOMMITMENT to contracts. Previously, contracts can reasonably expect OP_*TOKENCOMMITMENT to ensure that the commitment is less than or equal to 520 bytes, and importantly, they can reasonably expect OP_*TOKENCOMMITMENT to never fail, regardless of commitment size. It's more conservative for a VM operation to fail evaluation on unexpected behavior, as future hard forks can cleanly add functionality in that "gap" without breaking contracts. E.g. all the reactivated opcodes, schnorr support, bigger VM numbers, new opcodes, etc.

@cculianu On MAX_SCRIPT_SIZE – that's very interesting, I wasn't thinking about it that way! So the true limit on locking bytecode in a nonstandard transaction is actually the consensus MAX_TX_SIZE (1MB) rather than MAX_SCRIPT_SIZE (limitations during spending don't impact our encoding). In that case, maybe we should limit commitment lengths to MAX_SCRIPT_SIZE rather than subtracting the token encoding overhead (currently 38 bytes). It's certainly nice to avoid adding new "magic numbers" to the system (like 9962), and MAX_SCRIPT_SIZE is a good guide for setting a MAX_COMMITMENT_SIZE: commitments occupy a very similar place in decentralized application design and they contain the same kind of data that might otherwise be encoded (much less efficiently) in scripts. (E.g. if some future upgrade wanted to expand one of the two limits, it should reasonably expand both.) In fact, the most conservative choice is probably to leave MAX_COMMITMENT_SIZE in the same position as MAX_SCRIPT_SIZE with respect to future upgrades: no limit other than MAX_TX_SIZE for commitments in transaction outputs, and only validate MAX_COMMITMENT_SIZE during spending.

And just an example to make these hypothetical longer commitments more concrete: many future DEX designs could contain order book information in relatively-long commitments held by covenants that are only moved once after many other users have interacted with the "owning" covenant. A single long commitment moved rarely would ultimately use less network bandwidth/chain space than the same system designed to use clusters of small commitments, so it could be in the network's interest to eventually reduce this waste by supporting much longer commitments.

Would love to hear what you two think given the above – I'm leaning toward MAX_COMMITMENT_SIZE = MAX_SCRIPT_SIZE, but I'll keep thinking about it.

[cculianu]

@bitjson -- i quickly skimmed this to the end. I will respond in tdetail later but basically -- why are youc ontinuing to conflate MAX_SCRIPT_SIZE and anything to do with token data?

I thought I was very clear in my writing above -- the token data has nothing to do with the script itself. There is no connection between the two things other than the "serialization trick" we are doing to embed the token data in the place where un-upgraded software expects the script blob.

Can you kindly do me a favor and divorce the idea that the token data and the script itself have any connection other than the fact that they happened to be serialized together?

The token data is not part of the script after deserialization. And the script is not part of the token data either. They are separate things. Please divorce the two concepts in your mind.

---

That being said, imposing a limit of MAX_SCRIPT_SIZE on token data's commitment length seems arbitrary. Why that number in particular and not another number? Where does this number come from?

[cculianu]

For nonstandard transactions, commitment length is constrained only by the existing 10,000 byte limit on locking bytecode length; this is the maximum commitment length that can be encoded while maintaining backwards-compatibility with existing transaction parsing implementations.

This is wrong. Please update your mental model to account for what I wrote above.

[bitjson]

This is wrong.

Yes, I'll have to update that part of the rationale section. I wrote that before you brought to my attention that MAX_TX_SIZE enforces the only real limit. Thanks!

That being said, imposing a limit of MAX_SCRIPT_SIZE on token data's commitment length seems arbitrary. Why that number in particular and not another number? Where does this number come from?

I was trying to think through that here:

MAX_SCRIPT_SIZE is a good guide for setting a MAX_COMMITMENT_SIZE: commitments occupy a very similar place in decentralized application design and they contain the same kind of data that might otherwise be encoded (much less efficiently) in scripts. (E.g. if some future upgrade wanted to expand one of the two limits, it should reasonably expand both.) In fact, the most conservative choice is probably to leave MAX_COMMITMENT_SIZE in the same position as MAX_SCRIPT_SIZE with respect to future upgrades: no limit other than MAX_TX_SIZE for commitments in transaction outputs, and only validate MAX_COMMITMENT_SIZE during spending.

So I'm now thinking that a MAX_COMMITMENT_SIZE should be set to the same value as MAX_SCRIPT_SIZE and enforced in the same way (not enforced during creation, only during spending). I'm not sure if that was intentional on Satoshi's part, but its a very clever way to keep MAX_SCRIPT_SIZE/MAX_COMMITMENT_SIZE upgradable in the future without requiring existing implementations to deal with larger sizes.

[A60AB5450353F40E]

Ok so still 2 limits, properly "divorced" but sharing the concept of how we'd enforce it. Meaning our UTXO could be 10,000 (script) + 10,000 (commitment) big if we want it to be spendable. Or bigger - but then it will be unspendable (until some future upgrade) because the limit would be enforced only when time comes to spend.

[imaginaryusername]

I want to chime in on one more thing: We generally do not want to introduce more things that are "enforced" through standardness, that's practically useless in the short term and really inviting trouble down the line in the long term. Standardness is a terrible way to upgrade on a chain where hardforks are generally possible, it was originally meant as a method to preserve upgrade paths without forks.

Please remove https://github.com/bitjson/cashtokens/blob/f8b500a051f82d42dbf9e9e890bc6cdc14592307/readme.md?plain=1#L183 , https://github.com/bitjson/cashtokens/blob/f8b500a051f82d42dbf9e9e890bc6cdc14592307/readme.md?plain=1#L136 and other associated nonstandard-but-consensus parts. I'd prefer the consensus to be just 40, but 520 or any other length is fine too (you guys know the nuances better than I do) - just don't keep some nonsensically different nonstandard size around, it can be debated and upgraded later in network upgrades just like VM limits etc.

[cculianu]

Yeah just implementation-wise -- the standard vs nonstandard thing is just a headache to implement. The less of that stuff we add the better the code is and the less technical debt.

I vote for 520 as the limit. It can be raised later if we decide to raise the limit on MAX_SCRIPT_ELEMENT_SIZE (the max size of VM pushes).

---

Other than that -- I still have yet to perceive a satisfactory explanation for the rationale behind the max commitment length proposed being MAX_SCRIPT_SIZE. Why not MAX_SCRIPT_SIZE + 2? Or why not 666666. Or why not the square root of pi, rounded up? You see -- the limit seems completely arbitrary. Again, note that token data is unrelated to script data (introspection opcodes notwithstanding).

I must stress that 520 seems like the only sane limit now. Anything else is potentially harmful and weakly justified, IMHO.

[cculianu]

I want to chime in on one more thing: We generally do not want to introduce more things that are "enforced" through standardness, that's practically useless in the short term and really inviting trouble down the line in the long term. Standardness is a terrible way to upgrade on a chain where hardforks are generally possible, it was originally meant as a method to preserve upgrade paths without forks.

Just to agree with you here -- standardness was added because the original Satoshi rules for the chain were DoS-ey. It was possible to craft chains of txns that completely raped miners and nodes during validation. So standardness was added to "limit" what untrusted non-mining nodes could do, so as to make the chain more regular. It was a trick to reign in originally insane rules to a modicum of sanity.

But when authoring new rules for consensus -- we don't have that problem! We can just conceive of sane rules from the get-go that are sane for everybody -- miners and non-miners alike should use the same rules.

So yes, I agree. Adding new rules to standardness vs non-standard makes no sense in the modern BCH context. Standardness was a band-aid historically... one that we will continue to live with. But adding new rules to standardness is ill-advised.

Just conceive of new rules from the get-go that are sane and not DoS-ey.

[imaginaryusername]

re: Using commitments of longer length, it really depends on which path we are committing to:

1. Commitments are expected to contain hash of data only; or
2. Commitments are expected to contain data themselves (as speculated in jedex design at Limit the commitment length #23 (comment) )

The problem with using (2) as rationale is there's no end to it: We will likely always be able to find usecases where it's more convenient to put longer and longer data into commitments, but UTXO size is a severely finite resource that cannot possibly accommodate such endless demands. Mandating hashes instead of data fields may make things significantly more complex in many cases including jedex, but it's the only possible way to future-proof the rationale without carving a special-case for jedex - and it looks like jedex itself won't be satisfied with 520 bytes even in the conceptual stage anyway.

If you ask me, I'd stick to 40 bytes for purity - anyone that actually want data in there can opt for more complexity at op_return, where scarce UTXO space isn't consumed.

[A60AB5450353F40E]

Re. the API problem for introspection - then we should set the limit to 521 lol, just so contract authors will be aware their contracts can get messed up if they don't sanitize inputs.

[cculianu]

FWIW I'm in camp 40. But can be persuaded to join camp 520 (or even 521 but maybe I'm too smoothe-brained to understand the joke there, ha ha).

Def. not in favor of anything significantly above 520 right now... (e.g. 10000 seems like asking for trouble).

[bitjson]

I should add more background about the thought process behind the current version of the spec:

• At initial publication, I proposed that CashToken commitments be simply limited by consensus to 40 bytes.
  • I recognized that increasing this limit (even in a "hard fork") would be very difficult if not impossible: This is a great example of the paradox with prefix codepoint upgrades – we can make backwards-compatible changes to the transaction format using prefix codepoints, but we generally can't later modify the encoding of those new elements without breaking transaction parsers.
  • But I failed to recognize exactly how limiting 40 byte commitments might ultimately be until I worked through the impact on some large examples (Jedex and a Drivechain-like sidechain covenant).
• In v2.1 I updated the proposal (based on discussion here) to take what I thought was the most conservative approach: limit to 40 bytes via standardness, but apply no further consensus limits.
• Above, @cculianu brought to my attention that locking bytecode encoding is limited only by MAX_TX_SIZE – MAX_SCRIPT_SIZE is only checked once spending is attempted. So the most conservative approach now seems to be: a standardness limit at 40 bytes, commitment encoding is limited only by MAX_TX_SIZE, and a MAX_COMMITMENT_SIZE is checked at spend.

---

From @imaginaryusername:

I want to chime in on one more thing: We generally do not want to introduce more things that are "enforced" through standardness, that's practically useless in the short term and really inviting trouble down the line in the long term. Standardness is a terrible way to upgrade on a chain where hardforks are generally possible, it was originally meant as a method to preserve upgrade paths without forks. [...] I'd prefer the consensus to be just 40, but 520 or any other length is fine too (you guys know the nuances better than I do) - just don't keep some nonsensically different nonstandard size around, it can be debated and upgraded later in network upgrades just like VM limits etc.

I agree in general with the sentiment that we should be moving away from "standardness" where possible, but this particular case is more nuanced. We're adding a new concept to the existing transaction format; without making room for future upgrades (using standardness), those future upgrades will be forced to either 1) break all wallet, indexers, and other ecosystem software, or 2) introduce a second, completely separate encoding for tokens (i.e. the most notable technical debt ever introduced into the protocol, see also: discussion here).

Maybe a useful question to clarify this: if we simply limit commitments to 40 bytes by consensus, what kind of upgrade would be required to increase that to 41 bytes? Satoshi's answer to this question (the behavior of MAX_SCRIPT_SIZE) was either very thoughtful or very lucky.

---

From @cculianu:

Other than that -- I still have yet to perceive a satisfactory explanation for the rationale behind the max commitment length proposed being MAX_SCRIPT_SIZE. Why not MAX_SCRIPT_SIZE + 2? Or why not 666666. Or why not the square root of pi, rounded up? You see -- the limit seems completely arbitrary. Again, note that token data is unrelated to script data (introspection opcodes notwithstanding).

Conceptually, commitments are closely related to – and inherit design requirements from – "scripts". Scripts include and operate on data that is either directly encoded (in locking bytecode) or provided and validated during spend (unlocking bytecode). Commitments allow that data to be extracted and maintained separately from the originating contract, even across multiple other UTXOs. In fact, for multi-covenant decentralized applications (e.g. Jedex), its significantly more efficient in terms of transaction sizes to extract practically all of this data, leaving only "constants" in the contract itself – the "variables" are maintained in commitments, allowing them to be efficiently read/modified by multiple contracts (or in the case of depository receipts/redeemable NFTs, held by users to completely simplify away the covenants' need to copy that state across many transactions).

So in practice, for efficiently-designed decentralized applications, commitments are the "variables" while locking/unlocking bytecode typically contains only the "constants" and validation procedures.

Notably, the VM currently limits those validation procedures to 201 opcodes, but each "script" may be 10,000 bytes; in effect, the current consensus limits of the BCH VM suggest that commitments should actually have limits ~50x those of scripts. I don't necessarily agree with that implicit expectation (because I think a future upgrade should replace the 201 opcode limit), but it's hard to make a strong argument that the protocol should apportion more capacity to either variables or procedures. As such, I think the most conservative solution is to assume an even split – setting MAX_COMMITMENT_SIZE equal to MAX_SCRIPT_SIZE. Likewise, if a future upgrade proposed a change to either limit, I would expect that both limits ought to be modified together.

And of course, the goal of limiting commitments to 40 bytes for standard transactions is to set a conservative-but-functional initial limit. Future proposals can make the argument for increasing that (or transitioning away from using standardness entirely).

[cculianu]

Above, @cculianu brought to my attention that locking bytecode encoding is limited only by MAX_TX_SIZE – MAX_SCRIPT_SIZE is only checked once spending is attempted. So the most conservative approach now seems to be: a standardness limit at 40 bytes, commitment encoding is limited only by MAX_TX_SIZE, and a MAX_COMMITMENT_SIZE is checked at spend.

Just to clarify the script size it checked at spending in the current rules. Nothing forces us to stick to this convention for token data. Again, token data is not script data... We can make up any rules we want here.

We don't have to stick to the weird way that script work for token data! In fact, I advise against it since I consider the way scripts work now to be messy and unintuitive.. What I mean specifically is we don't have to stick to the convention that the MAX_X_SIZE is enforced only upon spending, not upon creation of the UTXO.

I recommend we enforce the max commitment size both upon creation of the UTXO and again upon spending (as a belt and suspenders check in the latter case).

---

Commitments allow that data to be extracted and maintained separately from the originating contract ....
... setting MAX_COMMITMENT_SIZE equal to MAX_SCRIPT_SIZE ...
And of course, the goal of limiting commitments to 40 bytes for standard transactions is to set a conservative-but-functional initial limit. Future proposals can make the argument for increasing that (or transitioning away from using standardness entirely).

Well, @imaginaryusername already raised a great point in that we don't want to be adding any more exceptions/differences between standard and non-standard txns. See his comment earlier in the thread. I am forced to agree with him.

In light of that -- pick a size @bitjson . Is 520 good for you for now for MAX_COMMITMENT_SIZE? Or should we just go with 40? Note there is little value in a "more generous limit" for non-standard. As we discussed above -- the "standard vs non-standard" thing is a historical thing we inherited. We want to avoid that in the future.

So pick one: Do you want the max size to be 40 or 520? (With no difference between standard and non-std).