Clarification: "invalid token prefixes" can just be locking bytecode

[bitjson]

@thesquaregroot identified an ambiguity in the spec we need to be sure is handled explicitly: any locking bytecode can begin with 0xd0 (OP_UTXOTOKENAMOUNT), even for outputs that do not themselves contain tokens. There are many useful covenants that might begin with OP_UTXOTOKENAMOUNT, and those outputs may or may not also carry tokens.

I overlooked this in the current version of the CHIP. I'll clarify the spec to ensure starting with 0xd0 isn't assumed to make the output invalid, and I've added this to the set of test vectors that will be in the next release. 👍

[A60AB5450353F40E]

During parsing the PFX format, non-compliance should not fail the TX but instead just skip interpreting the output as token, and the "malformed" token PFX would then be handed over to Script VM instead of being snipped off, and then it may or may not fail when executed as locking bytecode - it would depend on what such Script does. The PFX is much similar to P2SH. Non-standard outputs starting with P2SH pattern but having extra opcodes are still valid but will be processed differently than P2SH "magic" pattern - simply executed as "bare" script. With tokens, we add one more such "magic" pattern, and deviation from the pattern format would have it interpreted as locking script.

[cculianu]

Yep, this is what I implemented already, since this ambiguity became obvious to me while implementing. Yes, as I currently have it I do what you expect: if it parses ok according to the rules we described, it is loaded as token data and the remaining bytes are assigned to the locking bytecode.

if it doesn't parse ok, ALL the bytes are just treated as locking bytecode.

Note that under no circumstances using current consensus is it possible to spend any locking bytecode that starts with 0xd0 .. and in the future after activation if it started with 0xd0 but failed to parse.. it would not be possible to spend it either. EDIT: It would definitely be possible to spend it (as a non-standard txn), provided the proper stack state is set up by the spending scriptSig. (See @thesquaregroot 's comment below)

I would suggest that we forbid ANY locking bytecode from even being created after activation.. if it failed to parse as token data but started with 0xd0. The reason for this: to keep things as simple as possible implementors.... but I am not married to this idea and it's not even clear to me if it's worth the trouble to be so restrictive so.. I say this quite diffidently. EDIT: See @thesquaregroot 's comment below

[thesquaregroot]

.. and in the future after activation if it started with 0xd0 but failed to parse.. it would not be possible to spend it either.

That's what I thought at first, too, since 0xd0 is currently a disabled opcode, but as Jason mentioned, 0xd0 is actually defined as a part of this CHIP to be OP_UTXOTOKENAMOUNT. It would make it a non-standard output/transaction, but I believe it would be spendable if a miner were to include it in a block. It's certainly an interesting use-case - a non-token output that depends on the data from token outputs - but I could see there being some interesting use-cases for it.

[cculianu]

Oh duh I'm stupid. You're right -- if the scriptSig contains a push for the index parameter to the opcode.. there you go. It is spendable. Hmm... apologies for my mistake.

[cculianu]

Note I implemented this already and it became very clear to me that we need to be clear on what is a parse error and what is not in the spec. As such, see some of my suggestions in #43.

[cculianu]

Stated again: If you really want this feature it opens up a pandora's box about what is parseable and what is not so the spec must be very clear on what to reject as a parse failure and what to not reject as a token data parse failure. This is because bytes that are rejected as a token data parse failure now would be prepended to the locking script and thus have completely different rules applied to them (just script VM rules). Whereas bytes that parse correctly are not subject to VM interpreter rules but are subject to token data validation consensus rules.

This consideration also applies to the PATFOs (pre-activation token forgery outputs) as well. So we must be very clear here.

I feel that the way the spec is presently worded, even after my #43 suggestion above is not sufficiently explicit and clear about these things..... and getting it wrong could mean consensus failure!

[cculianu]

See #44

[cculianu]

Hmm the more I think about this the more I think it’s not a great idea for prefix byte to do double duty like this.

Aside from the issues above… there is the possibility that some locking script that is not trying to do anything funny but that starts with 0xd0 ends up “accidentally” looking like token data and failing to be spendable (corner case, I know). This last point bugs me as well, tbh.

I think that either :

• we make a unique codepoint to denote token prefix (preferable)
• Or, we disallow locking scripts that don't parse correctly as token data (in effect forbidding 0xd0 as first opcode in non token scripts).

Tl;dr: not a fan of the ambiguity proposed here. It’s asking for trouble …

[A60AB5450353F40E]

Yeah, sharing the codepoint with introspection opcode seemed clever but it may be more trouble than it's worth.
As you say, we could just spec the PFX to be some codepoint that's NOT used as a valid opcode so there can be no double-duty.

[cculianu]

Yes.. thanks.

Note we could have two prefix bytes. 0x?? Which denotes “encapsulated payload” and a second byte for payload version, in this case maybe 0x0 or 0x1

This way we waste a code point .. but we gain the ability to embed extra different types of metadata going forward into a txn out.

And most importantly, we also gain the ability to someday extend the token data serialization format by defining a new version that directs the serializer to the correct code paths appropriate for the byte stream.

(Before anybody panics that TWO bytes for prefix is a waste of space — I personally strongly believe that being stingy with 1 byte here or there at the expense of extensibility is perhaps misguided optimization… 1 extra byte per token out is not a huge deal, imho… if it buys us extensibility.)