Indexer ignores inscriptions that are not on first input of TX, but otherwise valid (consensus)

[veryordinally]

While working on #783 @raphjaph and @veryordinally discovered what we thought to be a bug in the indexer that affects versions of ord up to and including 0.5.1: inscriptions that are not made on the first input are ignored during indexing. This was due to a simplifying assumptions in src/inscription.rs where only the first input was parsed for inscriptions. We addressed that with this commit as part of draft PR #1963

Discussed with @raphjaph and we suggest to disentangle this fix from the larger collections and provenance topic, and also to address this sooner, as it will cause inscription numbers to change since indexers with this fix applied will index those inscriptions that the previous indexer skipped. I will compile and add a list of inscriptions that are currently ignored to this issue.

[nothing0012]

I think this belongs to the consensus layer of the ord protocol. Should we have a activation blockHeight for this change? So that

Before block height X: only the first input was parsed for inscriptions
After block height X: consider all the inputs

And we give the community enough time to upgrade before block height X.

[veryordinally]

That would seem like a good general mechanism to do these kinds of updates/fixes to me. Would also be an option for collections/provenance. We should probably make an issue to implement an activation block height mechanism if we want t adopt this. @raphjaph what do you think? Downside would be that add in some sense arbitrary constraints on what gets parsed as an inscription and what doesn't which makes the protocol messier.

[gmart7t2]

While working on #783 @raphjaph and @veryordinally discovered a bug in the indexer that affects versions of ord up to and including 0.5.1: inscriptions that are not made on the first input are ignored during indexing

That is how the docs say it should be:

https://docs.ordinals.com/inscriptions.html

The docs and the code agree that only the first input/output is an inscription

that add in some sense arbitrary constraints on what gets parsed as an inscription

The whole thing is already arbitrary. Adding an extra constraint that allows multiple inscriptions per reveal tx after a certain block height is no different.

[timechainz]

The docs and the code agree that only the first input/output is an inscription

@gmart7t2 If I'm not mistaken, the docs you highlight refer to the first output on which the first sat contains the inscription, but the aforementioned bug relates to the INPUT of the reveal transaction. The code was only looking in the first input. That's not consistent with the doc which doesn't specify it's the first input. I believe that is why @veryordinally is referring to it as a bug.

[gmart7t2]

Is it meant to be possible to create multiple inscriptions per reveal tx? There are a few examples of people trying to make multiple inscriptions with a single reveal tx:

• 84c5ab8d08c769274213cb7554454a30a9d5c4a77ffee7fffdc8e11d6292568ci0 says "Hello, world - 1", while 84c5ab8d08c769274213cb7554454a30a9d5c4a77ffee7fffdc8e11d6292568ci1 says "Hello, world - 2".

• 092111e882a8025f3f05ab791982e8cc7fd7395afe849a5949fd56255b5c41cc has 116 inputs, all of which parse correctly as png images, the last 4 of which would presumably be inscribed on the fee sats.

[gmart7t2]

@gmart7t2 If I'm not mistaken, the docs you highlight refer to the first output on which the first sat contains the inscription

It says "the input of a reveal tx", suggesting that reveal transactions only have one input.

Are you suggesting that if there is an inscription in the 5th input of a reveal tx then the inscription should still be assigned to the first sat of the first output? It world seem more natural to apply ordinal theory, and assign the inscription to whatever sat in the output corresponds to the start of the inscription input. Then txs like 84c5ab8d08c769274213cb7554454a30a9d5c4a77ffee7fffdc8e11d6292568ci0 would make sense, creating 2 inscriptions with 1 reveal tx.

[timechainz]

I'm not saying anything in regard to the outputs. I'm simply saying that the fix this issue is addressing is related to selection of input. It selects the first inscription it finds in the inputs rather than only looking in the first input.

I think your feedback regarding multiple inscriptions in a tx is more applicable to #876

[DrJingLee]

Will the reindexing affect some special numbers such as inscription of 500,000, 1,000,000 ,99999,88888 etc. and some special numbers close to boundary such as 99,101, 999, 1000 9999,10000 etc ?
If so , May be it s good and fair to leave as it is and fix from the cetain height of blocks later on ? To Treat the bug as a force majeure ......

[BennyTheDev]

I like the negative ordinal number approach that I believe has been mentioned, as it would be a win-win: being in the negative range makes yours special and since literally everyone is relying on the positive numbers already, they would stay the same.

[timechainz]

I’ve thought a lot about this and it came up in a twitter space last night. For the record I have biases in both directions as I hold some inscriptions with nice round numbers and also valid inscriptions that are not showing. These are my current thoughts.

There are two concerns that should be addressed independently:

• What constitutes a valid inscription?
• How should valid inscriptions be numbered?

A transaction witness that has a valid inscription envelope as presently defined by the protocol should be considered a valid inscription. Pretending that valid inscriptions aren’t there because a sat is double-inscribed or before a certain block height seems very fiat and conflicts with the project ethos that Casey established from early on. My preference would be that regardless of what decisions are made on numbering, valid inscriptions should be present in the index, recognized, available on the site by inscription id, etc.

Numbering is a much more difficult question. I agree that it’s more a consensus layer than protocol layer function. There have already been at least a few incidents that have affected numbering (double-inscribed sats, spent as fees, inscriptions that are not on the first input) and there will likely be others. Per Casey, “this ship has sailed in the sense that people are attached to the existing inscription numbers”. While I think that’s a difficult promise to keep in perpetuity I understand the sentiment and think it’s the correct decision. That said, this increases the likelihood of a fork due to decisions made relating to the handling of how and when bugs should be fixed. There’s already (largely tongue-in-cheek) “orthodox inscription numbers” (lol) chatter happening on Twitter so I think it’s important to address the issue sooner rather than later.

@casey mentions the concept of hidden inscriptions in #1455 relating to the double-inscribed sats that are skipped:

Surfacing the hidden ordinals one day as an easter egg, maybe as nega inscriptions with negative numbers that count down from 0

I like that concept but would avoid a negative numbering system because it could suffer from the same renumbering issue, in reverse. People will get attached to the nega numbers again. They can just exist in eternal purgatory (rather than current state of damnation). A dumping ground for all past and future inscriptions that are valid but didn’t qualify for numbering inclusion for some reason.

My take is that the numbering system should be maintained (block height activation) but that surfacing hidden inscriptions without numbering should be prioritized so that unnumbered inscriptions are addressable by id.

[Psifour]

The optimal solution (for this, and future incidents) is one that:

• Minimizes the changing of existing inscription index numbers
• Only employs logic that can be ran sequentially without unnecessary overhead
• Acknowledges the experimentation by users who did things that should have been indexed, but found edge cases

My proposal that addresses all of these is the creation of a flag system. This can be implemented as a set of bit flags on the high-order byte of the inscription number number: u64 -> flags: u8 and number: u56, as a new field flags: u8/u16/u32, or some other arbitrary system (this is a technical decision and independent of the feasibility of this concept).

This would enable us to assign the number that an inscription 'would have been' with flags set to distinguish it from the standard numbering system. This satisfies each of the conditions I presented above (as I will demonstrate below):

• It not only minimizes the change of existing numbers, but fully eliminates any need to change them completely for this AND future incidents.
• This logic can be implemented as a single check for the abnormal condition during indexing. This makes it preferable to 'block height' based regressions.
• By exposing these flags indexes/indexers can add badges/modals/etc to indicate the event that led to these inscriptions being flagged. This makes it potentially educational as well.

[tonilainemusic]

i think the inscriptions should be added, and the inscriptions re-numbered.

If you don't re-number, you will set up a situation where people will go through and search for the 'real' 500k - the real 1 million. And probably somewhere down the line a correction to the 'real' numbers.

I say just fix it now.

[Magicking]

Renumbering can not be changed as it would break application living on this parallel bitcoin*.

I like the flags idea, it introduce the numbering in another dimension, I think this is some kind of versioning over a soft-fork of the ord protocol.

*Caveats to that @veryordinally mentioned on Twitter that inscription ordering is brittle. Which I would understand as the result of previous computation and thus subject to change if said computation is changing.

I would commend for it to be specified somewhere that inscription numbering is meant either as part of the Ordinal protocol or not but since it's part of the reference implementation, i think this may need clarification.

All of that said, if we consider the infancy which are in the Ordinals vs the years of maintenance it could be in, simple rules such as "consider all inputs for inscription numbering after X block" as proposed before would be an easy to implement and most understandable path forward. And both specifications and implementation shall need update.

[satoshi0770]

i believe we should address two points

1/ hidden inscriptions that were reinscribed on the same sat should be indexed and shown on the main explorer.
this can be with the negative numbers -0 first inscription on a sat that already had an inscription, -1 2nd inscription on a sat that already had an inscription ect.

2/ parent child inscriptions (tests) txs that have multiple reveals or txs where the reveal is not the first input,
i think the best way to fix this and keep compatibility is to follow a strict ruleset, first input first output should be a new inscription. parent should be 2nd input and 2nd output. and keep 1 reveal inscription per tx.
if im not mistaken this will not need any change to numbering.

so we are left with what to do with inscriptions that were revealed in any input other then the first input.
not sure on best way to address that. we can possibly completely ignore them as not following consensus of reveal being the first input and new inscription being the first output. but open to hearing other ideas on how to treat those transactions that were not following ord client consensus and possibly (up to debate) the social consensus (which gm quoted from the docs).

[earonesty]

agreed, best to ignore the reveals not following the spec, and this also prevents renumbering (reorg)

[BennyTheDev]

I fully support @satoshi0770's suggestion

[mikeest]

I believe that @timechainz proposal is the best compromise for everyone. He suggests that the numbering system should be maintained (block height activation), but that surfacing hidden inscriptions without numbering should be prioritized so that unnumbered inscriptions are addressable by ID.

I think we should not retroactively change the inscription numbers as they are an important part of the concept of Ordinals. As @casey has said before, THE ORDINALS PROJECT COMMITS TO IMUTABILITY OF INSCRIPTION NUMBERS. I would agree with that.

Even though inscription numbers are not stored on the blockchain, they have become an integral part for all users. I believe that the view of many individuals that the numbers are immutable and that all Ordinals are connected through their numbers has fascinated many users and is what makes Ordinals special.

As for the inscriptions that have been hidden due to various bugs, I agree with timechainz's suggestion that they should be made visible to users, but WITHOUT numbers. This is because we cannot exclude the possibility of new bugs causing new hidden inscriptions in the future, and we would be faced with the same dilemma as now.

I believe that most users would be satisfied with this proposal, as the inscriptions without numbers would be seen as misprints and have their own unique value. Furthermore, this system would be easily explainable to mainstream users, similar to stamps that are not usable for regular purposes due to production errors but still have their own collector's value.

Regardless of which system we ultimately decide to use, there is one important thing that has not been thoroughly discussed yet and many users may not be aware of:

So far, we have only been discussing making hidden inscriptions visible due to ONE bug. #2000 @veryordinally
However, as some may know, there was also a bug in the early stages between inscription numbers 500-600 where multiple inscriptions could be on one sat and are right now hidden. #1455

I believe it is crucial that if we decide to make hidden inscriptions visible, we make ALL current hidden inscriptions on the blockchain visible and not just the inscriptions from this particular bug. Otherwise, it would bring up discussions and discrepancies again as to why certain hidden inscriptions were not made visible.

Following the principle of "all or nothing."

[blieb]

If people don't want to renumber I think the best option is to not make them visible. And change the spec so other indexes will also not make them visible (else they will use other numbers).

Of we want to make them all visible. The order should be as they are on the blockchain. No strange "workarounds" to change numbers so numbers on ordinals will not change. This will make a bigger mess.

I also think most people who did inscribe this way also did accept that their inscription was not accepted when it did not appear on the ordinals website. So I would go for the first option.

[BTCAlchemist]

I agree with @gmart7t2 to pick an arbitrary future inscription # and insert the missing inscriptions into the index there. This would preserve the current inscription # immutability. The question is, who would need to agree to this to get it adopted? Maybe @casey, @gmart7t2, @raphjaph, or @terror know whose agreement would be needed or could suggest a suitable process for consensus.

As I see it the best solution is to pick an arbitrary inscription in the future, far enough ahead that people have time to update their software, and insert the missing inscriptions at that point.

We don't want to reward people who inscribed on inputs other than the first with "special" negative inscription numbers or we will see a spate of such inscriptions being made just to get the special low numbers.

So pick a number, say 1,543,606, and when we reach that number add in the missing 200 or so inscriptions at that point.

That way the inscriptions aren't lost and we don't encourage people to make weird inscriptions in an attempt to get special treatment.

[veryordinally]

Here's my current perspective after considering the various proposals and viewpoints brought forth:

The two main solutions being discussed focus on either retroactively changing inscription numbers or leaving them unchanged. However, I believe we can consider the possibility of embracing diversity in inscription numberings.

Some creative solutions have been proposed, such as using negative numbers or bitwise flags in the numbering system. These alternatives showcase the innovation and flexibility within our community and are worth exploring further.

Diverse indexing approaches would not cause a "fork" of the network, as inscription numbers are not intrinsic to what an inscription is. The Bitcoin blockchain will always remain the ultimate source of truth.

Embracing diverse perspectives on inscription numberings can actually improve decentralization. As more people run explorers embodying their preferred viewpoints on inscription numbers, it strengthens the ordinals ecosystem rather than weakening it IMO.

Our focus as a community should be on maintaining consensus around inscription IDs and inscription encodings, while recognizing that sequence numbers are less relevant. Building an ecosystem that embraces potentially divergent views on sequence numbers can lead to more resilience. Also, sequence numbers become less and less interesting now that we have surpassed 1 million inscriptions.

This approach aligns with the values of open source governance and decentralized systems, encouraging the community to explore alternative methods and interpretations while still reaching consensus on core aspects of the protocol.

[veryordinally]

@veryordinally wrote:

I will compile and add a list of inscriptions that are currently ignored to this issue.

Did you add the list yet? I'm not seeing it. I see a tweet where you give the number of such inscriptions but it doesn't match the number I arrived at and I'm interested to know where I went wrong in my analysis.

Sorry, forgot to attach the list. This was as of Monday, when I tweeted about 1206 "hidden" inscriptions. I obtained this list from running #1963 from @raphjaph and adding some logging.

hidden-inscriptions.log

[lgalabru]

@veryordinally Could you re-run your extract, and ignore the transaction if you have multiple inscriptions?
I inspected a small sample manually, and ALL of them already have an inscription in Input #0.
Which means that they would not count as valid inscriptions.

[lgalabru]

Again, per the spec:

There is some ambiguity on the first part (red), but the yellow part is crystal clear, and the protocol DO NOT support multiple inscriptions per transaction - as inscription on input #0 (or #N) would be attached to the 1st satoshi of the first output, any subsequent inscribed input would do the same - meaning we would be looking at re-inscriptions - not permitted.

[stet]

Again, per the spec:

There is some ambiguity on the first part (red), but the yellow part is crystal clear, and the protocol DO NOT support multiple inscriptions per transaction - as inscription on input #0 would be attached to the 1st satoshi of the first output, any any subsequent inscribed input would do the same - meaning we would be looking at re-inscriptions - not permitted.

Honestly, if we take this approach then we could also start making arguments that duplicate inscription content or content with json or other mime types before they were "officially supported" or even any experimentation at all with what is contained in the envelope.... would all be up for debate if they should or should not have a superficial sequential number based on total ord protocol specifier transactions.

[veryordinally]

@veryordinally Could you re-run your extract, and ignore the transaction if you have multiple inscriptions? I inspected a small sample manually, and ALL of them already have an inscription in Input #0. Which means that they would not count as valid inscriptions.

Will check, but would be surprised if that is the case. From my eyeballing of the affected inscriptions these were mostly inscriptions with a parent inscription in input and output 0, i.e. the inscription on input 1 would be the first new inscription.

[stet]

Also worth pointing out that the sequential Inscription Number being used on ordinals.com and then obv elsewhere only encouraged spam, duplicate content and fed into a rush to grab a spot as opposed to respecting the protocol to be used for digital artifacts and appreciating meaningful historical sats.

The sequential numbering system has value obviously but making this perceived value as paramount to everything that ordinals docs actually does discuss, is a stretch in the fomo direction that we all should consider as not nec healthy for the overall project. Collectors and speculators gonna set their own sort of rules and critieria for value but this is OUTSIDE the scope of Ordinal Theory.

[lgalabru]

@veryordinally I ran a quick script, assuming that the 1206 hidden inscriptions were correctly extracted, it looks like only 315 of them should be considered, all the other ones are falling in the multiple inscriptions / per transaction bin.

txids.log

[veryordinally]

@casey here's the issue converted to a discussion like you requested - wanted to preserve all the existing thoughts from the community. Your take would be helpful. Some specific questions:

1. Was it an intentional part of the protocol to only support inscriptions on the first input, or was that an implementation detail / simplification?
2. Assuming we do want to support inscriptions on inputs other than the first going forward (to support Collections and Provenance #783 and Create multiple inscriptions in a single transaction #876 and other future extensions), how should we handle upgrades to what is recognized as an inscription? Specifically, would an activation height mechanism make sense for such upgrades?
3. Depending on how we handle 2, how do we deal with implications to the inscription numbers - the main two perspectives are "timechain rules" (inscription numbers should be sequence numbers of inscriptions on chain, under whatever the latest definition of an inscription is), or "existing numbers need to be stable". You previously stated you want to keep existing numbers stable (https://github.com/casey/ord/issues/1455#issuecomment-1434980969)

[kevinfaveri]

1 - If it was never specified otherwise explicitly, then I think we should go with a change like this: https://github.com/casey/ord/discussions/2015#discussioncomment-5626854
2 - I think it is important even beyond the cited extensions: it unlocks minting of Ordinals at a big scale, which will help grow the pie and make it cheaper for creators to experiment with the technology without the need to invest thousands of dollars. Again, I think the activation height is not only the clear solution --- it is the most in line with BTC core development.
3 - I think maintaining / marketing inscription numbers as a somehow stable id complicates things a lot and don't improve UX that much (there are better stuff to focus on than maintaining a vanity number id). I suggest official comms start to use the official on chain id. Inscription count can be supported in explorers for ease of use/search, but ideally we have the definition clear that what is truly immutable is the on chain id (inscription number being just a convention). And in case we choose to discard other than first input inscriptions from before, there's nothing to be done on number 3, right? They stay stable.

Overall, I think that's okay to assume past mistakes and try to improve on them. Try to fix backwards introduces more problems than it is worth it. Bitcoin design recognizes this and is a proven path to success of managing and introducing breaking changes.

[Aervue]

My 2 cents:

Change the spec to allow multiple inscriptions per tx by including multiple inputs/outputs. Vanity numbers are not important in the long term. Don't make this a height upgrade and overcomplicate things.

[earonesty]

agreed vanity numbers are not important, but why allow multiple per tx? seems like people were just ignoring the spec.

[Aervue]

Pros for multiple per tx:

• Lower fees
• Consecutive vanity numbers
• Parent-child might require this (not sure?)

[satoshi0770]

There is inherent issue with this.
First parent child does not require because parent can be 2nd input 2nd output and the reveal is still first input first output.
The issue is when u have input 1 being 1k sats. Input 2 being 10k+fee
If output 1 is anything but 1k sats. Then the inscribed ends up not being from the the first say of the input reveal. So it breaks current specs.
If output 1 is 700 sats then output 2 has sats from input 1 and the inscribe sat is from input 1 and not the reveal input 2.
If output 1 has more then 1k sats, then output 2 and the inscribed sat will not be the first sat of input 2 where the reveal input is.
It breaks consensus.

[rot13maxi]

I think that there are two parts to the solution here: what to do about inscriptions that have already been made and were missed by the indexer because they did something non-standard, and what to do long-term about multiple inscriptions in a transaction.

Ordinal Theory tells us that the first sat in the output comes from the first sat in the input. The inscription docs also say that in a reveal transaction, the first sat of the first output is what is inscribed. I think that it follows then that the data to be inscribed needs to be in the witness of the first input of the reveal transaction. In other words, the software is currently working as intended. If people tried to put more inscriptions in other inputs, then that data isn't currently recognized by the protocol. In other words, there is no bug.

That said, I think there is a desire to have multiple inscriptions in a transaction. Those new rules will need to specify how to map revealed data to output sats. AFAIK, that part of the protocol hasn't been specified yet.

[Aervue]

I believe the current ord protocol is beta at best right now, changes that have more pros than cons should be encouraged until everything is more developed, standardized and tested.

[stet]

Its more important to set the right precedent for moving forward.

It is very simple. If transaction uses "ord" protocol specifier with envelope containing data, include it.
There will be experiments. Ord is alpha and so are the experimental inscriptions.
How many duplicate inscriptions are there?
How many were inscribed using a file format before officially supported?
Many debates could be started on what qualifies and doesn't qualify....
Or we can just stop the craziness and count all ord inscription transactions thus allowing for any future experiments.
Let indexers use filters and spit out different sequential numbers but ALL should obviously be default.

[satoshi0770]

If transaction uses "ord" protocol specifier with envelope containing data, include it.

My point exactly.
What is this "ord 'protocol' specifier"
How would you currently define it.

My understanding, currently ord protocol does not allow re-inscribes, does not allow multiple input inscribes. We need to have a clearly defined spec. Otherwise any ambiguity can be interpreted as desired.

[timechainz]

What is this "ord 'protocol' specifier"

I think it's quite clear that the protocol defines the concept of an envelope and an "ord" designation for inscriptions.

The only question is: Are there situations where we want to hide inscriptions that have a perfectly valid inscription envelope?

Hiding or censoring inscriptions really conflicts with the ethos of this project. With that in mind, it makes more sense to liberally detect and include all inscriptions in the index (according to current methods) and make them available by id - even if they have no number or negative number or whatever. If we apply flags (as @Psifour suggested) we can keep the indexing code simple and efficient while at the same time be hyper-conservative with the block height activation of new flags with regard to the default numbering system, optimizing for immutability of inscription numbers like @casey declared on 2/17. It's the best of both words.

There are already other projects starting to build custom indexes to route around this shortcoming. If the ord project is seen as gatekeeping it will lower the credibility of the project and increase the likelihood of forks. Let's be inclusive with inscriptions but keep the default numbering system as immutable as possible.

tldr

Hiding inscriptions is fiat, don't do that
Make a best effort to make inscription numbers immutable

[satoshi0770]

I think it's quite clear that the protocol defines the concept of an envelope and an "ord" designation for inscriptions.

Unfortunately the spec on the documentation is not the actual spec onchain. If you have a look at the witness data, its not correct according to the docs. (Should we included txs that are following the docs as inscriptions? Because currently no txs that follow the docs envelope are indexed)

Would it be fair to say the current definition of whats an inscription is what the ord client currently allows and recognizes? If so then anything outside of this is not an inscription?

Hiding inscriptions is fiat, don't do that
Make a best effort to make inscription numbers immutable

Agree with this, nothing should be hidden. But its reasonable to have clarity on what exactly is an inscription. Currently there is no clear defining spec outside of what ord client does.

There is another issue with multi input reveals, where the ordinals in the input reveal could, not end up being the ordinals inscribed. But which ordinals are inscribed is up to indexing/inscription rules which once again aren't clearly defined outside of ord client.

Tldr: I'm totally for reindexing including all inscriptions, renumbering if we need or have another way around that ect. But want to bring to light for clarity and a good precedent for the future.

[veryordinally]

Here's what Casey came up with: #2045 - I think this addresses all concerns brought up in the discussion here. Working on the implementation now.