Change the default maximum OP_RETURN size to 80 bytes

[Flavien]

The maximum size for OP_RETURN outputs used to be 80 bytes, then got changed to 40 bytes to be on the safe side. We have now been running with 40 bytes for about 9 months, and nothing catastrophic happened to the Blockchain, so I am proposing to increase it back to 80 bytes.

Also, that value is now configurable through the datacarriersize option, so miners who want to stay on 40 bytes (or any other value) can easily do so.

[petertodd]

There's unit tests that you have to modify as well; try running src/test/test_bitcoin and you'll see them fail.

[Flavien]

Thanks Peter, I will fix the unit test.

[luke-jr]

Don't-care-ACK once tests are fixed.

[luke-jr]

Minor idea: Perhaps we should set the default to a random value between 0-80 at startup, to incentivise miners making their own decision without actually forcing it explicitly.

[TheBlueMatt]

This is semantically similar to #5075, and, as discussed there, this is not something to be discussed in a pull request, please propose this on the ML first, and then come back here.

[dexX7]

Let me play devil's advocat and ask: why should one use OP_RETURN at all? As long as alternatives are cheaper (and faster at this point), there is not much inceive to do so.

I took a look at cost as transaction size in relation to payload sizes and it turns out 40 byte OP_RETURN isn't really a great choice.

80 byte however, assuming one pay-to-pubkey change output, beats efficient bare multisig encoding (that is: spending multisig outputs!).

[luke-jr]

Well, that sounds like an argument for #5231 since bare multisig is not for data storage.

[dexX7]

I agree that bare multisig encoding should be discouraged as data carrier, if this PR is accepted.

Though it should be evaluated, if there is still need for #5231 then, or if it would just hurt the remaining other "legitimate" use cases.

[jgarzik]

OP_RETURN is pruneable

[laanwj]

As said on the mailing list, I'm OK with this. It makes OP_RETURN more attractive to use in favor of messier, unprunable approaches.

[gavinandresen]

utACK

[jgarzik]

ut ACK -- will close my own PR in preference to this one, after this merges, as this seems to have greater consensus.

[sipa]

I'm in the middle here. If this is because the alternative is people storing data in other non-prunable ways anyway, then it's obviously a win. For some use cases, there is no way around that.

But for many there is, and this is just providing them with an easier path that may result in an ecosystem with higher costs for everyone. I don't like encouraging that, and I don't think it's in the best interest of Bitcoin nodes to help people do so.

[gmaxwell]

I think OP_RETURN has shown itself to be seriously problematic; and we continue to have problems with people beleving that storing non-bitcoin related data in the chain (as opposed, e.g. to simple commitments or things like ECDH nonces) is an approved, correct, non-antisocial use of the system. We have people selling insane data storage services, etc. It's a bad place to be.

Meanwhile, many of the externalized cost creating services which could use op_return (e.g. data they're encoding is small) still don't bother. Rather than a success, I think it's hard to say that it hasn't been much of a success. Though: at least a few things that were going to encode data did switch to hashes, so I think the limitations were at least somewhat informative.

That said, there are some cases where I think more data can make sense and be useful. For example, I think of the ECDH address negoiation stuff isn't a terrible use... a bit inefficient compared to using an external system, but not unreasonable... and that application wants the ability to push in additional payment identifiers and the like (see petertodd's latest update to the stealth address additions).

So, I'd prefer to make any increase take the form of allowing more pushes, e.g. you can have up to {limit} bytes, in the form of pushes up to 40 bytes. Beyond furthering the message that this isn't an endorcement for externalizing your storage and transmission costs to the increasingly shrinking public network that our decenteralization depends on; this option also is permissive enough that there isn't a {well you can use checkmultsig instead} that is more permissive.

If people who would support this would also support that, I'll submit a PR.

[jgarzik]

@gmaxwell That's what #5075 does

[luke-jr]

@jgarzik No, #5075 allows multiple outputs. What @gmaxwell is suggesting is still a single output, but with multiple pushdata opcodes.

@gmaxwell Don't-care-ACK I gave this applies to your version as well.

[petertodd]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

@jgarzik No, #5075 allows multiple outputs. What @gmaxwell is
suggesting is still a single output, but with multiple pushdata
opcodes.

I have a pull-req open to allow that.

As for this pull-req, utACK
-----BEGIN PGP SIGNATURE-----
Version: APG v1.1.1

iQFQBAEBCAA6BQJUgM3EMxxQZXRlciBUb2RkIChsb3cgc2VjdXJpdHkga2V5KSA8
cGV0ZUBwZXRlcnRvZGQub3JnPgAKCRAZnIM7qOfwhexHB/4xpXq3P3+AC3wUsJID
YgI/oLJIOS74mVYobmfi3qjF8TWtsi6IHpHKAPRS1JCakSSFRa8wviLfwvrrglYv
wW3ZusJXxjHnMsJblbhMMJl+2mF1W2RIMoYi2jiIhz4nMMFsrU6rl8vvVERXKRec
WslksFr23Tx441FCVk4K9LpmzjkeCqr5NxUGKaudVKTM9RqSId+frsGF+Teo2uNx
QVtprvTzvxqHShSEKGL+y+cMlaWZ2p8HH3nXtN1tkoEqxdqb0LEauV8Hk0MSkRz2
6kXZnFTtxa+alwVzfsEdR6mU/5cl9Sg/RmpbQraUYUAw6SEtytVeUSAwY2NfO87V
xJsb
=ioIv
-----END PGP SIGNATURE-----

[gavinandresen]

untested ACK.

So we've got three different suggestions for how to do this, and two "maybe it is a bad idea to do it at all."

Of the three suggestions: I like this one best, it is simplest conceptually.

And I still think the benefits of a prunable 80-byte OP_RETURN (and 40-byte OP_RETURN) outweigh the costs.

[petertodd]

Removing the single PUSHDATA restriction is orthogonal to how many bytes are allowed, and mostly orthogonal to how many OP_RETURN outputs are allowed.

On 4 December 2014 21:35:19 GMT+00:00, Gavin Andresen notifications@github.com wrote:

untested ACK.

So we've got three different suggestions for how to do this, and two
"maybe it is a bad idea to do it at all."

Of the three suggestions: I like this one best, it is simplest
conceptually.

And I still think the benefits of a prunable 80-byte OP_RETURN (and
40-byte OP_RETURN) outweigh the costs.

---

Reply to this email directly or view it on GitHub:
#5286 (comment)

[petertodd]

@gmaxwell

this option also is permissive enough that there isn't a that is more permissive option.

Creating unspendable outputs is another option that is more permissive. I'm of course supporting that in the coloured coin standard I'm working on, an approach being taken by many projects. (that particular application has no show-stopper issues even with P2SH²)

[laanwj]

I think OP_RETURN has shown itself to be seriously problematic; and we continue to have problems with people beleving that storing non-bitcoin related data in the chain (as opposed, e.g. to simple commitments or things like ECDH nonces) is an approved, correct, non-antisocial use of the system

I fully agree. However, practically, until the time that we have a solution for spam in unspendable outputs (cluttering the UTXO set which is so much worse), at least encouraging the least antisocial method for people that insist on doing it is advantageous, I'd suppose...

[dexX7]

cluttering the UTXO set which is so much worse

I believe an increase in OP_RETURN's default size would mainly serve as inceive to move away from bare multisig encoding which is currently "cheaper". Getting the attention of (ab-) users who create unspendable outputs, such as cryptograffiti.info, appears to be another topic and an increase to 80 byte probably has no effect in that context.

[gmaxwell]

Fake h160 and checkmultisig stuffing all also are limited to storing <32 bytes per push, which is why I pointed out that the limited OP_RETURN pushes would still be "prefered", but still lilmiting the people promoting misuse of the system from claiming that arbritary data storage is a endorsed, supported, and maintained activity.

[petertodd]

@gmaxwell The relevant metric is cost per byte published; I'd suggest you get the recent P2SH IsStandard() rule removal reverted if you are concerned about sending such a "message" and wish to be self-consistent; see https://github.com/petertodd/uniquebits specifically https://github.com/petertodd/uniquebits/blob/master/uniquebits/_unibits.py#L85

@dexX7 How was that graph calculated?

[dexX7]

Hey @petertodd: the chart was created based on the following assumptions:

• no unspendable outputs
• after each message, the state should be similar (means in pracise there might be change)
• all public keys are compressed
• if public keys are used to embed, then two bytes are subtracted to form fully valid public keys
• if the payload size surpasses capacity, then multiple transactions are created
• pay-to-pubkey is cheaper than pay-to-pubkeyhash

Please let me know, if there are flaws or more efficient ways to transport data, but I came up with the following scripts:

OP_RETURN: up to 40 byte (80 byte, 200 byte)

scriptSig: [signature]
scriptPubKey 0: [op_return] [payload]
scriptPubKey 1: [public_key] [op_checksig]

Bare multisig: up to 62 byte, embedded as public key, 31 byte each, with n = [2, 3]

scriptSig: [op_0] [signature]
scriptPubKey: [op_1] [public_key] [payload] ( [payload] ) [op_n] [op_checkmultisig]

P2SH: up to 434 byte, embedded as public key, 31 byte each

Transaction 1 (commit):
scriptSig: [signature]
scriptPubKey: [op_hash160] [script_hash] [op_equal]

Transaction 2 (reveal) with n = [2, 15]:
scriptSig: [op_0] [signature] [op_1] [public_key] [payload] ( ... ) [op_n] [op_checkmultisig]
scriptPubKey: [public_key] [op_checksig]

[petertodd]

no unspendable outputs

Better to determine what the cost for creating them is due to coins lost to the dust limit. Also calculate w/o dust limit, as lots of hashing power doesn't follow it.

if the payload size surpasses capacity, then multiple transactions are created

Why multiple transactions? Why not have multiple outputs instead?

[bharrisau]

Why multiple transactions? Why not have multiple outputs instead?

Only one OP_RETURN output per transaction?

[dexX7]

Also calculate w/o dust limit, ...

I have the impression we're drifting a bit off topic, but since this is about defining new defaults, I think other defaults should be respected, such as the dust rule.

Why multiple transactions? Why not have multiple outputs instead?

You're right. It was mentioned in particular for OP_RETURN and this might be misleading. In the case of bare multisig or script hash transactions, using multiple outputs is prefered of course. Still worth to note: the overhead of creating a new transaction is actually only 10 byte, assuming outputs are redeemed, because each additional output requires an additional input to reclaim sooner or later.

[petertodd]

I have the impression we're drifting a bit off topic, but since this is about defining new defaults, I think other defaults should be respected, such as the dust rule.

I'm interested in what the economics are of people doing proof-of-publication, not about "respecting" any particular rule.

Similarly assuming outputs are redeemed is not a good assumption; what's the underlying economics?

[dexX7]

@petertodd: don't get me wrong, I fully agree and think the economics and inceive aspects should dictate the decision whether the size of OP_RETURN should be increased or not. This is also the reason why I created the chart - which might not be 100 % accurate, but close to. One can start to make assumptions about miners who don't "respect" the dust rule, but this is difficult to messure and there are a few other aspects as well, e.g.:

• miners who block bare multisig
• miners who don't mine OP_RETURN

My previous tests showed (with sample sizes of less than thousand transactions) the acceptance of OP_RETURN increased significantly over the year, but bare multisig is still faster.

The bottom line:

Those additional factors support an increase in size, whether it's based on the assumption that it's more profitable to ignore the dust rule, to create unspent outputs or to include the fact that not every miner mines data transactions.

It can further be determined that the size should be at least 80 byte to stay competitive against other data embedding mechanisms.

[jtimon]

I think it's more important to move this check to policy.o (something similar to luke-jr/bitcoin@581ffd7) than to discuss the "perfect" defaults when they will always really be arbitrary. There's no guarantee that the people defending 80 won't be defending 120 in the future...

But it is true that people using bare multisig for data store is even worse, so I won't oppose this PR (nor #5231, even though that's another thing that should be only used in policy.o).

[Flavien]

Has a decision been made regarding this PR? There seems to be some level of agreement on it.

[petertodd]

@Flavien I'd be very surprised if a decision gets made that's anything other than "status quo"

As I keep saying, design protocols with adaptive encoding.

[genjix]

If devs are going to make artificial policies to encumber OP_RETURN outputs in a transaction, then eventually people will just go directly to the miners. I personally don't want to see a protocol which is increasingly shaped by influence and money. Therefore it is in your interest to allow multiple OP_RETURN outputs in a transaction each with a limit of 80 bytes. If people feel your authority is undermining their products, eventually they will subvert that and it only takes 1 miner to disagree. And eventually you'll end up playing politics trying to enforce limitations on the network which not everyone agrees with. Policy should be shaped by need and use of the blockchain, not personal views about how it should be used.

[luke-jr]

Going directly to miners is exactly what people should be doing. Mining policy is not something developers should be responsible for.

[genjix]

I agree but devs also want to shape the process rather than allowing miners to vote. And as stands, they do have some tenuous way of shaping what behaviour is allowed. I'm simply stating they will lose that power if they're too inflexible.

[luke-jr]

If being too inflexible gets miners to take responsibility for their own policies, that sounds like a good idea...

[jtimon]

In my opinion the underlying problem is that the standard policy is far too important because users and miners don't implement their own policies. It is certainly not easy at this point.
In my opinion using bitcoind with a custom policy should be as simple as extending the standard policy class and calling bitcoind -policy=custom as shown in jtimon/bitcoin@f71f691 (related to #5595 and #5696 among other PRs).
Hopefully that would reduce these endless policy discussions...

[sipa]

There already is -maxdatacarriersize. All this does is change the default.

[Flavien]

The problem here is that most miners don't care and are fine with whatever the default policy is as long as it doesn't increase their orphan rate.

We all understand miners should be good citizens and vote, but they just can't be bothered (just like with "real" democracy).

[jtimon]

Sure, in the case of -datacarriersize it's specially surprising that the discussions continue.
So some people don't want the standard policy to be less important, but rather than it keeps being as important but it is the way they like it. This is very annoying and doesn't scale to many different opinions.

[Flavien]

If you want -datacarriersize to have any practical relevance, you'll have to make it mandatory.

[gmaxwell]

There is also a matter of driving competent design rather than lazy first thing that works. E.g. In stealth addresses the early proposals use highly inefficient single ECDH point per output instead of simply pooling them. Network behavior is one of the few bits of friction driving good technical design rather than "move fast, break things, and force everyone else onto my way of doing thing rather than discussing the design in public". No one wants to be an outright gatekeeper, but the network is a shared resource and it's perfectly reasonable node behavior to be stingy about the perpetual storage impact of the transactions they're willing to process, especially when it comes to neutral technical criteria like the amount of network irrelevant data stuffed in transactions.

There is also a very clear pattern we've seen in the past where people take anything the system lets them do as strong evidence that they have a irrevocable right to use the system in that way, and that their only responsibility-- and if their usage harms the system it's the responsibility of the system to not permit it. This is especially concerning since external data that isn't necessary for the operation of the system comes with a liability that people may try ordering nodes or miners to censor the data (e.g. when it's being used as a botnet control channel acting as a high bandwidth channel for "illegal information"). For mitigating these risks it's optimal if transactions seem as uniform and indistinguishable as reasonably possible.

[dexX7]

... surprising that the discussions continues.

It's similar to other policy decisions such as default fees or permitting bare multisig. maxdatacarriersize is certainly a step in the right direction, but as long as there is a relation between default values and the actual behavior of the network, it continues to have relevance and any decision or non-decision in this context has consequences.

There is also a matter of driving competent design rather than lazy first thing that works.

The intend to push innovation, so to speak, is great and I like it.

There is also a very clear pattern we've seen in the past where people take anything the system lets them do as strong evidence that they have a irrevocable right to use the system in that way, and that their only responsibility-- and if their usage harms the system it's the responsibility of the system to not permit it.

True, this is the case, but this is related to the point I was trying to make earlier: OP_RETURN in it's current form is not really competitive against other data carrying mechanisms such as using bare multisig, and this abuse probably won't stop unless OP_RETURN gains an edge. The alternative, making bare multisig non-standard as proposed by Luke, may lead to even more harmful results, namely switching back to the creation of non-redeemable outputs, which is also cheaper in terms of effective transaction size/payload size for larger payloads.

[genjix]

gmaxwell, your idea doesn't work for coinjoin. txs want multiple stealth outputs so they can be coinjoined.

[gmaxwell]

@genjix They can be, they just have to share the ephemeral key. An important point is that a lot of the privacy of a coinjoin comes from not being able to tell if its a coinjoin or not and if so how many outputs it has, so it's really preferable to not break that if at all possible. (See also, http://fc15.ifca.ai/preproceedings/bitcoin/paper_5.pdf )

[genjix]

That doesn't work. Especially for prefixes or payment IDs. For usability, there's no way getting around needing multiple outputs.

[jtimon]

@dexX7 those are good points but imagine we do both: make bare multisig non-standard and change MAX_OP_RETURN_RELAY from 40 to 80 or 999999. Maybe we can also have different default constants for relay and mining.
Will that make standard policy discussions end?
I don't think so. That's why I prefer to focus on making policy code behavior more easily configurable rather than dedicate an unbound amount of time to get to the "objectively best policy", which we know doesn't exist.
People have to be aware that the standard policy is conservative, probably over-conservative; and deal with it by having their own policies. Having a standard policy being too permissive kind of defeats the purpose of having a standard policy in the first place.
So I think I agree with @luke-jr there (but I will rephrase it): if having parts of the standard policy that people who care about policy don't like, and that causes people who care about policies implement or configure their own policies....maybe it is good to have "unpopular" things on the standard policy.

@genjix I don't know about @gmaxwell's solution but the maximum number of OP_RETURN outputs in the standard policy can be made configurable too.

I'd just rather not create more policy-specific command-line options until the policy code is isolated.
I wish people had less time for discussing the "correct" standard policy and more time to review and ack a trivial PRs like #5696 (sorry for linking that again, but it just comes to mind every time I comment on anything related to policy).

[gmaxwell]

@genjix PeterTodd specced out an alternative that can just put many prefixes in a single output. In any case it isn't a 'free move' adding external non-system related data to transactions has serious (externalized) costs that have be considered thoughtfully, even if some usages are simple and boring.

[genjix]

I will just use normal Bitcoin addresses and avoid all this headache. And I'll be sure to include false positives so they can't be blacklisted or detected with certainty.

All I hear is blaa blaa about shaping Bitcoin according to how you think it should be used rather than how people are using it driven by their needs. What kind of arbitrary limit is one OP_RETURN per tx for?

Bitcoin is not and never will be a payments system. It is money, or store of value and the blockchain is a multi-use tool for many applications and it should be embraced totally in that vein rather than having small minds and trying to keep it pulled down in some metaphorical stone age. Unencumber the protocol and let people be free to create.

You can create a decentralised web, advanced financial instruments, stocks/shares/futures/options, distributed markets, publishing systems and much more... if you're that concerned about resources then you should vehemently oppose increasing/removing the block size limit. But many people are too driven by a view of what they already know trying to force Bitcoin to be that, rather than what it could be. Bitcoin is not credit cards and will always be a rubbish payments system, so long as it remains decentralised.

[dexX7]

@jtimon: I fully agree, moving towards user chosen policies is highly valuable, and I also support a more restrictive approach in general, but it doesn't seem to be an either-or choice. While new default values very likely won't stop similar discussions, I do believe it can encourage or discourage certain behavior.

The reduction of OP_RETURN to 40 byte was based on the assumption that:

80, as the standard line length, is almost asking for "insert your graffiti message here".

As it turned out, this backfired with services such as cryptograffiti.info or roughly 210k out of 260k bare multisig outputs that can be linked to data storage, whereby the number of all OP_RETURN uses on mainnet is still less than 20k. It's probably only a matter of time until more libraries such as blockpop or Stone appear, which abstract data storage and decouple data from the underlying transport layer, which is likely going to be chosen based on metrics such as transaction costs and miner adoption.

I have very mixed feelings about it in general, but getting the inceives right and minimize disadvantages of cost per payload and a slightly higher confirmation delay of using OP_RETURN compared to alternatives seems reasonable.

[laanwj]

80, as the standard line length, is almost asking for "insert your graffiti message here".

Yes, that's a quote from me. I was afraid that OP_RETURN would be used en-masse to send messages. It has been standard for quite a while now and that hasn't happened, so I'm okay with increasing the size to 80 (as I've already posted earlier above).

Edit: Also this has enough ACKs, no clue why this is still open.

[petertodd]

On Mon, Feb 02, 2015 at 08:43:08PM -0800, Gregory Maxwell wrote:

@genjix PeterTodd specced out an alternative that can just put many prefixes in a single output.

That alternative compromises on privacy for no good reason; I specced it
only because you asked me too and said you'd do a pull-req with the
necessary op-ret changes to make it work. You haven't done that, so I'm
assuming you're not interested. In the meantime, I'd much rather use the
version that doesn't have silly compromises.

[btcdrak]

Can this be backported to 0.10?

[petertodd]

@btcdrak You can submit a pull-req directly to the v0.10 branch actually. I could do that in another day or two; time-sensitive so probably better if you do.

[gidgreen]

Some empirical data from the coinsecrets.org database to contribute to the discussion. So far there have been 16154 OP_RETURNs on mainnet, as of block 341912.

Breakdown by protocol that we can identify:
4877 Open Assets
3015 Proof of Existence
1933 CoinSpark
600 BlockSign
5729 (other)

So for now the two main identifiable use cases are colored coins and hashing a document to timestamp its existence. There are still far fewer OP_RETURNs than multisigs used by Counterparty.

Average OP_RETURNs per block (grouped by 10,000 blocks):
20xxxx 0.0000
21xxxx 0.0000
22xxxx 0.0002
23xxxx 0.0000
24xxxx 0.0029
25xxxx 0.0127
26xxxx 0.0134
27xxxx 0.0195
28xxxx 0.0274
29xxxx 0.0332
30xxxx 0.1520
31xxxx 0.2667
32xxxx 0.4258
33xxxx 0.5170
34xxxx 0.7559

So there is clear growth but it's not showing signs of being exponential.

[laanwj]

@gidgreen Interesting numbers, thanks for the research!

[jtimon]

Proof of existence doesn't need op_return, seehttps://github.com/Blockstream/contracthashtool . In fact, there's a form of hard to censor colored coins that uses a similar technique instead of putting data in the chain (I believe the most used name is "committed assets".

If this gets into 0.10 I will be jealous. Why this can go in but not my commit removing the miner's hashmeter, for example?

[petertodd]

On 4 February 2015 07:40:13 GMT-08:00, "Jorge Timón" notifications@github.com wrote:

Proof of existence doesn't need op_return,
seehttps://github.com/Blockstream/contracthashtool .

That technique is quite dangerous to use, as it creates coins unrecoverable by your wallet seed.

If this gets into 0.10 I will be jealous. Why this can go in but not my
commit removing the miner's hashmeter, for example?

Because more people cared about this.

[btcdrak]

@jtimon I think the reasoning for adding this patch to 0.10 is much more straight forward: it limits "damage" being done by meta protocols that currently use more wasteful and bloaty encoding. Allowing them to move to OP_RETURN encoding sooner is better for everyone.

Having this go into 0.10 now would mean the eco system moves to relaying 80 opreturns faster than if we wait for 0.11 which might be 6 months away. Let's face it, wide deployment of a new version takes considerable time after release, so sooner is better for everyone. Given that the change is uncontroversial technically speaking, I personally think it should go into 0.10 now.

@petertodd I didn't create a PR for this because it's a straightforward cherry-pick, and since all merges are done manually anyhow, it doesn't create any more or less work for the maintainers to pull.

[jtimon]

I just said I will be jealous...

@petertodd when I said "proof of publication" I really meant proof of existence/timestamping (the use case @gidgreen was referring to). I don't understand the danger, you can pay to yourself to do this and you have all the information you require to claim your coins once again.

[gmaxwell]

@petertodd The contract hash tool works just as well on DSA nonces.

[gmaxwell]

@btcdrak One of the things requests is actual example applications; which have been in short supply. It's hard to not get the impression that people are not keeping their applications secret in order to game the process here.

[jgarzik]

This uses proof of publication: https://en.bitcoin.it/wiki/Identity_protocol_v1 Admittedly the proof part is not really used at all today, for limit related reasons.

[petertodd]

On 4 February 2015 10:44:55 GMT-08:00, Gregory Maxwell notifications@github.com wrote:

@petertodd The contract hash tool works just as well on DSA nonces.

Mind uploading an implementation then? Also last I checked with you that crypto was uncertain; a concern repeated by some non-Bitcoin cryptographers I ran the idea past.