Rationale: `OP_UTXOTOKENCOMMITMENT` and `OP_OUTPUTTOKENCOMMITMENT` push `0` for zero-length commitments

[A60AB5450353F40E]

"If the UTXO includes a non-fungible token with a zero-byte commitment, push 0x00. If the UTXO does not include a non-fungible token, push a 0 (VM Number)."

If we encode a 1-byte commitment that encodes a commitment[0]=0x00 it would result in the same 0x00 value being pushed to the stack.

Suggestion: push an empty stack item in both no-NFT and no-commitment cases, and the user should check the capability to distinguish no-NFT from no-commitment.

[cculianu]

This is 1000% something I noticed as well and I recommend the spec be changed as you point out to remove this ambiguity.

[bitjson]

Thanks for opening this issue @A60AB5450353F40E! That's definitely a rough edge right now.

Related: we probably don't want it to be possible to encode a 0x00 commitment in two ways, so if we keep the idea at all, it might at least be valuable to forbid the longer encoding (0x0100).

Suggestion: push an empty stack item in both no-NFT and no-commitment cases, and the user should check the capability to distinguish no-NFT from no-commitment.

That's a good point, any contract that is just using zero-byte commitments could probably just use OP_*TOKENCATEGORY to confirm that the NFT exists. (They likely aren't going to issue fungible tokens for the same category that they're checking for zero-byte NFTs, but if needed, it's also pretty cheap to confirm that zero fungible tokens are attached to the output).

Some rationale: I'd originally selected 0x00 because it's short but not a valid number, so contracts can safely use both zero-byte NFTs and number commitments without extra validation. One downside of 0x00 is that it requires two bytes to push in contracts (0x0100). We could maybe use 0x81 to reduce that to a single byte (OP_1NEGATE) without making it confusable with commitments of positive numbers, but the that still runs the risk of being accepted by other validation looking for commitments with numbers.

But maybe I was trying too hard to make zero-byte NFTs efficient – tokens already cost at least 34 bytes to encode, adding a special case to encoding and inspection for zero-byte NFTs seems like a lot of complexity to save one byte in rare cases. (And like you said, most use cases will have other equally-efficient ways to look for a zero-byte NFT.)

---

Ya, seems like a good idea for zero-byte NFTs to push 0 for OP_*TOKENCOMMITMENT. I'll update that in the spec and work on describing it in the rationale, thanks!

(Will leave this issue open until a correction is merged.)

[A60AB5450353F40E]

No wait I think I misinterpreted "zero-byte" as zero-lenght, I think the spec is fine, just needs clarifications.

So when it comes to OP_*TOKENCOMMITMENT, lets go over all cases:

• no PFX, return EMPTY ("0x" in BitAuth) which is also numerical 0, or OP_0 right?
• commitment_length == 0, return EMPTY
• commitment_length == 1 and commitment == 0x00, return 0x00

the len=0 is not the same as having a len=1 commitment of 0x00, in v2.1 of the spec it means that the NFT doesn't use the feature, not that it uses it with 0 value.

[bitjson]

No wait I think I misinterpreted "zero-byte" as zero-length, I think the spec is fine, just needs clarifications.

That interpretation was correct! By "zero-byte NFT", I meant NFTs with commitments of length 0.

To clarify: if a particular decentralized application needed to share numbers using NFT commitments (rather than using fungible tokens), that application would likely need to use OP_NUM2BIN/OP_BIN2NUM to pad the number to a fixed width. E.g. in Jedex, threads retain information about what portion of their balance is collected_fees using a number padded to 7 bytes (see here in the application's "Token API").

So the diff for the description of OP_OUTPUTTOKENCOMMITMENT – before this change:

Pop the top item from the stack as an output index (VM Number). Push the token commitment of the output at that index to the stack. If the output includes a non-fungible token with a zero-byte commitment, push 0x00. If the output does not include a non-fungible token, push a 0 (VM Number). If the token commitment is longer than the stack item length limit, fail evaluation.

That description after this change:

Pop the top item from the stack as an output index (VM Number). Push the token commitment of the output at that index to the stack. If the output includes a non-fungible token with a zero-byte commitment, or if the output does not include a non-fungible token, push a 0 (VM Number). If the token commitment is longer than the stack item length limit, fail evaluation.

There's no longer a difference in behavior between "no NFT" and "an NFT with a zero-byte commitment". OP_OUTPUTTOKENCATEGORY and OP_OUTPUTTOKENAMOUNT offer sufficient flexibility to distinguish between those cases if needed. And as mentioned above/demonstrated in Jedex, a lot of "token category APIs" will already be much more constrained; it will often be more byte-efficient for contract designers to avoid creating situations where they need to distinguish between the two cases.

[cculianu]

Ok, I'll go ahead and update the code (and test cases) now to handle it this way..

[cculianu]

Just FYI: You know, there is no way to differentiate between NFT_Immutable with 0-byte commitment and NFT_Fungible. You can't ask for the capability byte directly. You can only ask for the categoryId with the capability byte appended. And if you do, NFT_Fungible and NFT_Immutable both end up without a capability byte appended to the categoryId! See the native introspection opcodes spec. Perhaps that needs changing?

[A60AB5450353F40E]

Then let's please separate the FORMAT (has FT, has NFT, has NFT commitment) from the CAPABILITY (immutable, mutable, mint), it will all be much clearer. See #28

And here I have a column specifying all 13 possibilities and what introspection should return for each: #27 (comment)

[cculianu]

Ok, created #29 .

[cculianu]

@A60AB5450353F40E I'm on with that approach too as discussed in #28. .. but waiting on Jason to OK that one. In the meantime I have a suggestion in #29...

[bitjson]

@cculianu

Just FYI: You know, there is no way to differentiate between NFT_Immutable with 0-byte commitment and NFT_Fungible.

That's the consideration with returning 0 rather than 0x00 for zero-length commitments. Returning 0 simplifies the BCH protocol (by removing that 0x00 edge case), but it means contracts must use situation-specific validation (OP_*TOKENCATEGORY and/or OP_*TOKENAMOUNT) to differentiate. Differentiating is not impossible though, and returning 0 actually requires fewer bytes in each case (that's what I was trying to get across above).

The critical point to understand is that the issuer of the token category (e.g. a set of covenants) controls the "API" within that category (and even in extreme cases where contracts might read tokens of arbitrary categories, there are other strategies for vetting them). Contract authors/compilers will only use zero-length commitments in cases where that 1 byte savings does not increase the cost of validation across other contracts, and in those cases 0 vs 0x00 saves an additional 2 bytes per validation (OP_NOT vs OP_PUSHBYTES_1 0x00 OP_EQUAL). Dropping the 0x00 behavior also removes the discontinuity that can occur if non-padded numbers are used alone in a commitment: with the 0x00 approach, such contracts would need to include a special case for 0 too (and pushing most contracts to just pad those numbers, wasting 2-7 bytes per output).

So after the discussion in this issue, I'm strongly leaning toward dropping OP_*TOKENCOMMITMENT's special handling for zero-length commitments. Returning 0x00 makes the protocol more complex and is unlikely to be used by well-optimized contract compilers (validating 0x00 is just as expensive as instead using a 1-byte, OP_1NEGATE-OP_16 commitment), while returning 0 for zero-length commitments will be a valuable optimization for many contracts (can save up to 7 bytes per output).

[A60AB5450353F40E]

Dropping the 0x00 behavior also removes the discontinuity that can occur if non-padded numbers are used alone in a commitment: with the 0x00 approach, such contracts would need to include a special case for 0 too (and pushing most contracts to just pad those numbers, wasting 2-7 bytes per output).

Ah right... from point of view of Script API it is a discontinuity because if you're using commitment for non-padded numbers the 0x00 would be the only number requiring a OP_BIN2NUM to convert to 0 as ScriptNum. Other values would be directly usable as ScriptNum - as long as the user would respect the convention to not pad the numbers. So, returning 0x00 instead of 0 would "break" our user even if he used only non-padded numbers and he'd need a special treatment for the 0.

I saw it as discontinuity if you're using the commitment for some string that may have the value 0x00. If the opcode returns 0, then the string user needs to catch that and convert it to 0x00. What do you think will be more common? I was confused for a sec., thought you wanted to return EMPTY when len=1 and 0x00 are encoded.

So after the discussion in this issue, I'm strongly leaning toward dropping OP_*TOKENCOMMITMENT's special handling for zero-length commitments. Returning 0x00 makes the protocol more complex and is unlikely to be used by well-optimized contract compilers (validating 0x00 is just as expensive as instead using a 1-byte, OP_1NEGATE-OP_16 commitment), while returning 0 for zero-length commitments will be a valuable optimization for many contracts (can save up to 7 bytes per output).

This answers it, and I'm now too leaning towards returning 0 (EMPTY stack item) and not "0x00". It's not even "special".. a 0-length string is an EMPTY string, and therefore returning an EMPTY stack item which also means 0 ScriptNum is adequate.

In any case - it must be well documented and clear :)

[A60AB5450353F40E]

Ok @cculianu @bitjson I think we can close it - it should continue to return 0 for len=0 or if commitment field not encoded, but we need to fix category opcode to return (ID+capability) in case of immutable instead of just the ID, see other issue.

Note re. encoding: the "no commitment" encoding should replace the "has 0-len commitment" which should be disabled, since the 2 encodings would mean the same thing from the point of view of Script API and we only need 1 encoding for the 0-commitment and the smallest one will be "no commitment" which can be a 34b encoding (pfx+ID+0x43) where OP*CATEGORY would return ID+0x03 (immutable nft), and OP*COMMITMENT would return 0 as ScriptNum.

What of len=1 and 0x00, that should be the way to encode a 0-byte when commitment is used as string, in which case the OP*COMMITMENT would return the 0x00 proper, right?