Remove arbitrary restrictions on OP_RETURN by default

[petertodd]

Any number or size of data-carrying OP_RETURN outputs are allowed, and the -datacarrier option is removed. For those who want to limit data carrying outputs, -datacarriersize is still supported, and has the functionality of applying the specified data carrier limit as well as limiting the number of data carrying OP_RETURN outputs to one. If -datacarriersize=0 is set, no data carrying output is allowed.

Rational: there's lots of ways for people to publish data in the Bitcoin chain, and lots of data has been published. There's no reason for us to place artificial limits on this particular method.

Replaces #27261

[DrahtBot]

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.

Type	Reviewers
Concept NACK	luke-jr, recursive-rat4, Sjors, 1ma, RobinLinus, Retropex, ChrisMartl, Sun0fABeach, jesterhodl, davidlick, cbergqvist, HenrikJannsen, imacfan, ns-xvrn, dimitaracev, BBezaire, gzuuus
Concept ACK	Ayms, ChristopherA, fiatjaf, hsjoberg, vicariousdrama, zndtoshi, itme-brain, dangershony

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #28132 (policy: Enable full-rbf by default by petertodd)
• #27832 (doc: Clarify -datacarriersize, add -datacarriersize=2 tests by MarcoFalke)
• #26525 (Remove -mempoolfullrbf option by BitcoinErrorLog)
• #26291 (Update MANDATORY_SCRIPT_VERIFY_FLAGS by ajtowns)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

[luke-jr]

Concept NACK, the spam filters should be fixed, not removed or relaxed.

[recursive-rat4]

NACK until the data is discardable and the block size is increased.

[Ayms]

Concept ACK bitcoin surpasses eth in every domain except storage
Current storage methods need two txs and many deviant methods were used in the past to store things in bitcoin, as we saw recently also

[luke-jr]

except storage

Blockchains don't do storage. There's a tool for that called ext4, and BitTorrent for distribution.

[ChristopherA]

Concept ACK. Current limits on op_return force people to use hacks to store data hiding in op_codes. Using op_return is more `honest’. In my own case, current op_return limits keep me from posting a signed hash plus some metadata (<256 bytes) and thus I must use Taproot tricks instead.

[Ayms]

@luke-jr the idea is not to store a video on bitcoin but indeed a reference to it like bittorrent
Now if someone wants to store things in a full block nobody can avoid this
Then this change just make things easier to avoid unwanted practices

[maflcko]

From CI:

test/functional/mempool_datacarrier.py:31: error: Bracketed expression "[...]" is not valid as a type  [valid-type]
test/functional/mempool_datacarrier.py:31: note: Did you mean "List[...]"?
Found 1 error in 1 file (checked 269 source files)
^---- failure generated from lint-python.py

[Sjors]

Concept NACK unless you keep the default at MAX_OP_RETURN_RELAY. Depending on adoption we can always drop that default later. No need to make pull requests more controversial than necessary.

[petertodd]

Concept NACK unless you keep the default at MAX_OP_RETURN_RELAY. Depending on adoption we can always drop that default later. No need to make pull requests more controversial than necessary.

@ChristopherA is paying me $1000 for 10 hours work to do this pull-req. Since the other sub-change got mired in controversy, I decided to meet my client requirement in a time-efficient manner. Obviously, I can't promise to him that this will actually get merged. But I can promise to him that I'll do a mergable pull-req in the time allotted.

So frankly, at this point I don't care to split it up into two pull-reqs for such a minor reason. If someone else wants to, they can take this open-source licensed code and do it themselves.

[petertodd]

From CI:

test/functional/mempool_datacarrier.py:31: error: Bracketed expression "[...]" is not valid as a type  [valid-type]
test/functional/mempool_datacarrier.py:31: note: Did you mean "List[...]"?
Found 1 error in 1 file (checked 269 source files)
^---- failure generated from lint-python.py

From CI:

test/functional/mempool_datacarrier.py:31: error: Bracketed expression "[...]" is not valid as a type  [valid-type]
test/functional/mempool_datacarrier.py:31: note: Did you mean "List[...]"?
Found 1 error in 1 file (checked 269 source files)
^---- failure generated from lint-python.py

Hmm, I changed it to List[int], which still isn't working. Anyone with more Python knowledge know what it should be? https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html suggests that list[int] should be valid. But the lint still complained about that too.

[willcl-ark]

I’m on mobile and didn’t check the pull, but do you need to from typing import List too?

[Sjors]

So frankly, at this point I don't care to split it up into two pull-reqs for such a minor reason.

Making a significant change to default policy is not a minor reason.

is paying me $1000 for 10 hours work to do this pull-req

That's great, but others have to spend hours reviewing this stuff, plus occasionally having to deal with the brigading. I'm more likely to do that for a change that keeps the default, but I'm not the only one who can review of course.

[petertodd]

So frankly, at this point I don't care to split it up into two pull-reqs for such a minor reason.

Making a significant change to default policy is not a minor reason.

This is not a significant change. Large amounts of data are already published in the bitcoin blockchain in a variety of ways. Heck, -permitbaremultisig is still allowed by default, and people still use it to publish data in the form of spendable and even unspendable outputs. And unlike OP_Return outputs, bare multisig outputs have caused issued due to Bitcoin's broken sigop counting scheme; the OP_Return mechanism was carefully designed to avoid this problem years ago.

This proposed change simply cleans up a bit of remaining standardness paternalism; we've gotten rid of essentially all other standardness paternalism. I can't think of any standardness mechanism other than OP_Return and first-seen (which I'm arguing to remove by default in #28132) that doesn't have a strong technical reason to exist.

is paying me $1000 for 10 hours work to do this pull-req

That's great, but others have to spend hours reviewing this stuff, plus occasionally having to deal with the brigading. I'm more likely to do that for a change that keeps the default, but I'm not the only one who can review of course.

As I explain above, this is not a significant change and should not require hours of technical review.

[BBezaire]

Concept NACK

My node isn't your personal hard drive. It's for financial transactions, period.

We're already forced to store ordinals for free, while miners profit greatly off priority fees. Such a change will only make things worse for node operators, and destroy Bitcoin's credibility as a crypto currency.

This is a slap in the face to everything Satoshi worked towards.

[ghost]

This is a slap in the face to everything Satoshi worked towards.

I agree Satoshi would never use witness or bare multisig or OP_RETURN. They preferred coinbase. Unfortunately only miners can use it.

[itme-brain]

Concept ACK

The limit is nothing more than feel-good policy, people have been circumventing it forever.

Maybe if OP_RETURN never had an arbitrary 80 byte limit we would've never gotten JPEGs in tapscripts?

[1ma]

Concept ACK

The limit is nothing more than feel-good policy, people have been circumventing it forever.

Maybe if OP_RETURN never had an arbitrary 80 byte limit we would've never gotten JPEGs in tapscripts?

If datacarriersize is truly not effective in practice then surely there is no need for this PR at all, and it can be closed.

[HenrikJannsen]

Allows for including signature by a third party other than that signing the keys controlling the inputs. E.g., service provider.

🚩🚩🚩 Sounds like maybe this is tied to some kind of KYC nonsense?

There shouldn't be a "service provider" to begin with. (And if you really need such a signature, you could put it INSIDE the data which is being hashed.)

Would be interesting to hear the concrete use cases those who propose that have in mind.
Sounds a bit like a feature request from those who want to get "travel rule" on Bitcoin.

[vicariousdrama]

@whycorn

have you considered looking at this: #28217

Its under consideration but independent of this PR and feature.

@luke-jr

🚩🚩🚩 Sounds like maybe this is tied to some kind of KYC nonsense?

There shouldn't be a "service provider" to begin with. (And if you really need such a signature, you could put it INSIDE the data which is being hashed.)

Your emotion appears to get the better of you at times with use of the 🚩🚩🚩. This has nothing to do with KYC. Furthermore, service providers/intermediaries are welcome to use the Bitcoin network along with any other entity. If someone who has created a data set that is stored off chain wants to sign it for attestation and produce a hash and expose both signature and hash or a url to said data without associating to their own Bitcoin for privacy reasons, or prefer paying out of band in some other asset or even god forbid fiat to an intermediary, then they should be able to. Are you wanting to define specific constraints on a protocol within the data of an OP_RETURN, or just OP_RETURN itself? The arbitrary 80 character limit pushes towards centralization. Are you pro centralization?

[ayesaac]

If someone who has created a data set that is stored off chain wants to sign it for attestation and produce a hash and expose both signature and hash or a url to said data without associating to their own Bitcoin for privacy reasons, or prefer paying out of band in some other asset or even god forbid fiat to an intermediary, then they should be able to. Are you wanting to define specific constraints on a protocol within the data of an OP_RETURN, or just OP_RETURN itself? The arbitrary 80 character limit pushes towards centralization. Are you pro centralization?

Can you explain what benefit you get from storing the signature on chain? What is the benefit of storing a signature for some data on chain, and the data off chain, over storing just hash(concat(signature, data)) on chain, and the signature and data off chain?

You don't need blockchain for signatures. You don't need blockchain to share data. The only benefit you get from utilizing Bitcoin's blockchain in your protocol is the ability to reliably timestamp chunks of (signed) data. To do that, all you need is a hash.

[vicariousdrama]

If someone who has created a data set that is stored off chain wants to sign it for attestation and produce a hash and expose both signature and hash or a url to said data without associating to their own Bitcoin for privacy reasons, or prefer paying out of band in some other asset or even god forbid fiat to an intermediary, then they should be able to. Are you wanting to define specific constraints on a protocol within the data of an OP_RETURN, or just OP_RETURN itself? The arbitrary 80 character limit pushes towards centralization. Are you pro centralization?

Care to explain why the arbitrary-key signature cannot be part of the data itself, with the 32 byte hash in the op return merely timestamping that signature and the data it signs, with both the signature and the data off chain?

You don't need blockchain for signatures. You don't need blockchain to share data. The only benefit you get from utilizing Bitcoin's blockchain in your protocol is the ability to reliably timestamp chunks of (signed) data. To do that, all you need is a hash.

An intermediary wants to create an OP_RETURN that includes

1. URI to a resource
2. hash of the target resource
3. some other metadata which could be a date of validity or signature of the creator of the resource

The metadata, time stamps, signatures in number 3 are independent of the signature of the private keys used for spending the inputs to make the transaction. That is the service providers keys, not necessarily the creator of the URI. I concur with you. The arbitrary key signature (number 3 above) can be part of the data at the target URI in this case.

The size of OP_RETURN should ideally be larger to accommodate both a hash, and a reference to something outside of Bitcoin as the smallest way to make that reference / consume the least space.

[ayesaac]

The size of OP_RETURN should ideally be larger to accommodate both a hash, and a reference to something outside of Bitcoin as the smallest way to make that reference / consume the least space.

You can store 80 bytes in an op return. A SHA256 hash is 32 bytes.

Why do you need more than 48 bytes for a reference?

[vicariousdrama]

You can store 80 bytes in an op return. A SHA256 hash is 32 bytes.

Why do you need more than 48 bytes for a reference?

Varies depending on the actual URI

Assuming a SHA-1 hash of the following, but stored in binary form of 20 bytes
0e7364296be76d9ce5bc53dee6a15024d139760c

How should something like this be stored in 60 bytes
https://github.com/PoodleLabs/PoodleLabs.BitcoinBinaryVerificationScripts

[luke-jr]

It shouldn't be. Transaction keys are not URIs.

[RicYashiroLee]

The size of OP_RETURN should ideally be larger to accommodate both a hash, and a reference to something outside of Bitcoin as the smallest way to make that reference / consume the least space.

You can store 80 bytes in an op return. A SHA256 hash is 32 bytes.

Why do you need more than 48 bytes for a reference?

Mostly, Bitcoin's TINY SCOPE has no space for unrelated non-primary garbage data, desincentivizes bitcoiners of running full nodes and implicitly hurt Bitcoin's true decentralization through the only decentralizers, the node operators=bitcoiners=the clients of Bitcoin. IMV Core Devs and Miners are service providers only, their goal is often contrary to its decentralization, mostly they make money out of providing a service to bitcoiners and related companies profiting from Bitcoin.

[ayesaac]

You can store 80 bytes in an op return. A SHA256 hash is 32 bytes.
Why do you need more than 48 bytes for a reference?

Varies depending on the actual URI

Assuming a SHA-1 hash of the following, but stored in binary form of 20 bytes 0e7364296be76d9ce5bc53dee6a15024d139760c

How should something like this be stored in 60 bytes https://github.com/PoodleLabs/PoodleLabs.BitcoinBinaryVerificationScripts

Storing such a URL which you don't control in an immutable ledger is extremely fragile, wasteful, and generally a bad idea.

The hash itself can act as a resource identifier, and with a 20 byte hash, the following format:

[2 byte type/tld][1 byte length][76*6 bit characters / 10 byte raw onion link] would (more than) cover every possible clearnet domain and every possible onion 'domain'. You could also reference, say, every single npub at no additional cost; there's a lot of wasted space in that format.

Even ignoring the argument about whether such identifiers should be stored on-chain (and I would argue not), I do not see any convincing technical reason you need more than the space already available to you.

[printer-jam]

The individuals opposing this PR, believing they protect Bitcoin from centralization, ironically desire to dictate to all other people how they should and should not use Bitcoin.

Why create the PR to change Bitcoin? You are not stopped from using it within the current rules, yet you want to dictate to node runners that they must now store more arbitrary data, and if they don't agree they are the ones 'dictating' by opposing this PRs demands? Classic projection.

[gzuuus]

Concept NACK. It is easy to see that bitcoin is neither an efficient nor convenient way to store data in the "cloud", so why support the use of a feature that does not benefit and goes against of bitcoin's value transaction and is inconvenient for node operators and users in general? Strongly agree with the above answers NACK

[garlonicon]

Rational: there's lots of ways for people to publish data in the Bitcoin chain, and lots of data has been published.

When it comes to witness data, I can easily prune all of them by downgrading my node to be a non-witness node. How to achieve the same thing for large OP_RETURNs?

There's no reason for us to place artificial limits on this particular method.

The reason is you will encourage people to downgrade the way they publish data. Witness limit is 4 MB for a reason: it is easy to prune (just disable witness), does not affect legacy nodes, and in fact makes their life easier (see what is the size of the infamous block 774628 from the perspective of a non-witness node).

Current storage methods need two txs and many deviant methods were used in the past to store things in bitcoin, as we saw recently also

Just use commitments. All typical transactions contain at least one signature. If you have a single OP_CHECKSIG anywhere, you need a signature. If you use P2TR, you have a signature in your spend-by-key path, or you could use OP_CHECKSIG inside TapScript.

So, if you have a signature, it contains two parts: r-value, and s-value. You want to store something? Then just tweak your r-value in your signature, and you will get the same result as if you would when putting 256-bit hash inside OP_RETURN. With no additional on-chain bytes, and no traces that could be used to censor your transaction. For all address types that support OP_CHECKSIG or equivalent. What is the reason to not use commitments?

[luke-jr]

non-witness node

This isn't a thing. Witness data is no more prunable than other data, to a full node.

[elocremarc]

I hate to break it to you all. Bitcoin is primary a data layer and secondary a monetary layer. The data of transactions is worth spending to bribe miners for inclusion. However if some arbitrary data you disagree with gets inclusion and attestation and still follows the rules of consensus its a valid transaction. To censor any transaction that follows consensus is censorship.

Bitcoin is about uncensorable speech be it a monetary transaction or arbitrary data for someone willing to spend for inclusion. If the block utilization is low enough to allow spam then that will happen regardless of your opinion. If the block utilization is high enough from financial transactions then spam will be priced out. This is all about incentives of data inclusion nothing more.

"Information does not just want to be free,
it longs to be free.  Information expands to fill the available
storage space."
-- Cypherpunk Manifesto

[Psifour]

Bitcoin is purely a layer for storage of data that, typically, represents financial transactions. We have witnessed for years that there is no reasonable way to prevent the storage of abitrary data (any system that allows user input of 'random' values should accept the inevitability of data storage, "the signal is the noise").

The internet became the ubiquitous system that it is today not because some weak-willed soul decided for the users that they needed to be protected by limiting the power of their software stack, but instead by the raw memetic power of trolling on BBSs, porn, and attempting to subvert the control of others over our own lives.
It is messy and dirty, but by trying to fight this we have held bitcoin back repeatedly as it inevitably results in users searching for (and finding) other means of accomplishing their objectives that are worse for the network overall. I'm happy to play cat-and-mouse games and pay my non-penalized witness fee rate if I can't use OP_RETURN.

Let's take the non-consensus shackles off and let bitcoin do what it does best using the mechanisms that were purpose built to solve these problems, or let's keep pretending that endless 'spam filter' bandaids are at all a viable solution while we perpetually lean further from the permissionless cypherpunk Mecca that bitcoin can be.

[Nuropunk]

Cypherpunks write code. We know that someone has to write software to defend privacy, and since we can't get privacy unless we all do, we're going to write it. We publish our code so that our fellow Cypherpunks may practice and play with it. Our code is free for all to use, worldwide. We don't much care if you don't approve of the software we write. We know that software can't be destroyed and that a widely dispersed system can't be shut down.

[dangershony]

Concept ACK

We're building a specialized wallet and often struggle on where to keep pubkeys to recreate tapscripts, using opreturn makes it much simpler

[ChrisMartl]

I hate to break it to you all. Bitcoin is primary a data layer and secondary a monetary layer. The data of transactions is worth spending to bribe miners for inclusion. However if some arbitrary data you disagree with gets inclusion and attestation and still follows the rules of consensus its a valid transaction. To censor any transaction that follows consensus is censorship.

Bitcoin is about uncensorable speech be it a monetary transaction or arbitrary data for someone willing to spend for inclusion. If the block utilization is low enough to allow spam then that will happen regardless of your opinion. If the block utilization is high enough from financial transactions then spam will be priced out. This is all about incentives of data inclusion nothing more.

"Information does not just want to be free,
it longs to be free.  Information expands to fill the available
storage space."
-- Cypherpunk Manifesto

First, Bitcoin is a network which realize a monetary layer through decentralization; and processes data in order to pursue that layer.

Since it is a network, there are psychical instances which make possible this network to work. Those instances are provide by individuals who have the legit right of private property. As such, private property owners have the right to control what is expressed or communicated on their property. If a private entity chooses to moderate content on its physical instance, it does so as an exercise of its property rights.

Nobody prevents you or force you to not use
an alternative network which satisfies your wishes; the free market offers already options for you.

If Bitcoin participants should value something more than other, then it shall be “decentralization”; and decentralization of regular Bitcoin nodes. Decentralization provides more control to individuals over their properties and reduce the power of centralized entities in censoring information; like mining-node-entities.

It is clear, your view for Bitcoin is one which the decentralization of regular Bitcoin nodes is diminished towards more control and thus network centralization realized by mining-node-entities.

[RicYashiroLee]

I hate to break it to you all. Bitcoin is primary a data layer and secondary a monetary layer. The data of transactions is worth spending to bribe miners for inclusion. However if some arbitrary data you disagree with gets inclusion and attestation and still follows the rules of consensus its a valid transaction. To censor any transaction that follows consensus is censorship.

Bitcoin is about uncensorable speech be it a monetary transaction or arbitrary data for someone willing to spend for inclusion. If the block utilization is low enough to allow spam then that will happen regardless of your opinion. If the block utilization is high enough from financial transactions then spam will be priced out. This is all about incentives of data inclusion nothing more.

"Information does not just want to be free,
it longs to be free.  Information expands to fill the available
storage space."
-- Cypherpunk Manifesto

The altcoiner in Bitcoin throws the confusing argument of out of the point 'uncensorable' word, to claim Bitcoin is an all purpose data dumping platform, when it is not. Its scope is very TINY, based on principles, the decentralization trilemma (the case here) and the auditability/transparency vs privacy on-chain. When those principles are not followed or understood, you are suffering from altcoining mindset (your case, throwing confusing unrelated words like 'uncensorable speech', no different from altcoiner's 'inventions/solutions' on-chain (instead of off-chain where true ingenious innovation belongs, for Bitcoin). Bitcoin is not an altcoin because it MUST tyrannically (Satoshi's word) seek continuously more decentralization, not less as you are proposing. Bitcoin is getting more and more centralized due to all the BIP mania already introduced and reason why all current weaknesses (Segwit and Taproot are examples) exist, and it is not by adding more of such altcoining mindset weaknesses that the scenario improves, it is instead about allowing node operators more (UN)support OPTIONS in FULL (CORE) NODE SOFTWARE (not less), so that what is CORE=CENTRAL does get decided ONLY by them, and NOT by CORE DEVS who are in obvious conflict of interests as service providers in Bitcoin, NOT the true clients of Bitcoin. HASH-ANCHORING (=stamping, inscriptions,etc) in Bitcoin is possible the easier scripts are allowed by its introduced 'features', but instead Satoshi would definitely wanted it to continuously become as CUMBERSOME as possible in all ingenious ways possible with minimum scripts allowed, I am sure he would figure out eventually, to keep Bitcoin's SCOPE TYRANICALLY TINY as it was his intention (the only reason why Bitcoin has value and altcoins NOT). So, NOT make it even easier as you are proposing, throwing misplaced out-of-the-point ilogical intentionally confusing argumentation, against Bitcoin's basic principles and core value proposition.

[RicYashiroLee]

Bitcoin is purely a layer for storage of data that, typically, represents financial transactions. We have witnessed for years that there is no reasonable way to prevent the storage of abitrary data (any system that allows user input of 'random' values should accept the inevitability of data storage, "the signal is the noise").

The internet became the ubiquitous system that it is today not because some weak-willed soul decided for the users that they needed to be protected by limiting the power of their software stack, but instead by the raw memetic power of trolling on BBSs, porn, and attempting to subvert the control of others over our own lives. It is messy and dirty, but by trying to fight this we have held bitcoin back repeatedly as it inevitably results in users searching for (and finding) other means of accomplishing their objectives that are worse for the network overall. I'm happy to play cat-and-mouse games and pay my non-penalized witness fee rate if I can't use OP_RETURN.

Let's take the non-consensus shackles off and let bitcoin do what it does best using the mechanisms that were purpose built to solve these problems, or let's keep pretending that endless 'spam filter' bandaids are at all a viable solution while we perpetually lean further from the permissionless cypherpunk Mecca that bitcoin can be.

Bitcoin is NOT an all purpose DB. The reason why it has any value is exactly because it is NOT. CORE DEVS centralization of decision is a cancer in Bitcoin, as they are by nature in constant conflict of interests as service providers, they are NOT the clients of Bitcoin. The mechanisms of Bitcoin decentralization are corrupted the more power Core Devs have, the less power node operators have, which unfortunately has been the case, and IS SET BY CORE DEVS themselves in FULL NODE (CORE) SOFTWARE. Node operators are the only true decentralizers of Bitcoin, and the ones to decide what 'features' to VOLUNTARILY (UN)support in their full nodes (core software), NOT Core Devs with their non-stop BIP mania which only adds unwanted code complexity on-chain against Bitcoin basic principles and supposedly TINY SCOPE of its sui generis DB.

[Retropex]

Bitcoin: A Peer-to-Peer Electronic Database Cash System

[ayesaac]

Bitcoin is primarily for data storage

No. This is an absurd take.

Bitcoin is an absolutely horrible solution for storing arbitrary data in a permissionless, uncensorable manner.

If you want to store data as such, use something like a DHT torrent. It's a better solution, and doesn't force others to unwillingly store your arbitrary data.

Bitcoin is a monetary network first, but it does also happen to be the best decentralised option we have for timestamping too. But to make use of that secondary function, you do not need to store arbitrary data on Bitcoin's blockchain, as already discussed at length above.

[fanquake]

I locked this thread, because the discussion here was clearly not achieving anything (other than spamming the thousands of subscribers to this repository).

I'd suggest that anyone who's interested in continuing this discussion, take their thoughts to the mailing list: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-August/021840.html.

In the mean time, I think it's unlikely that Bitcoin Core is going to make any changes in regards to OP_RETURN. For now, I'm going to close this PR. We can revisit this in future, depending on the results of further discussion.