Policy: Decouple Solver() from nMaxDatacarrierBytes (for free) #6424

jtimon · 2015-07-12T14:13:16Z

This is a rebased and modified version of #5079. If @petertodd accepts my suggestions (specially the one about taking to opportunity of decoupling Solver() from nMaxDatacarrierBytes), I can close this PR.
If he doesn't and #5079 gets merged first, this can be rebased to do the decoupling.

dexX7 · 2015-07-12T14:44:42Z

src/policy/policy.cpp

@@ -49,7 +49,9 @@ bool IsStandard(const CScript& scriptPubKey, txnouttype& whichType)
            return false;
        if (m < 1 || m > n)
            return false;
-    }
+    } else if (whichType == TX_NULL_DATA &&
+               (!GetBoolArg("-datacarrier", true) || scriptPubKey.size() > nMaxDatacarrierBytes))


If nMaxDatacarrierBytes is 80, wouldn't the effective size only be 79 77 byte now, due to the op code, which is part of scriptPubKey?

It is clearer if you read commit by commit:
jtimon@dbc241d
jtimon@904d593
jtimon@984f7b4

The simplest way to maintain the current "effective size" of 80 is to simply add 3 to MAX_OP_RETURN_RELAY.
But the 3 is quite arbitrary: the number of pushes depends on the size of the data. So I think it's simpler to count the op_return and the pushes as part of the data: simply reduce the "effective size" from 80 to 77 and change the tests instead of the constant (as in the last commit).

Thanks, I understand your reasoning, and even though it's a bit arbitrary, this may have an impact on applications. For example, a block header doesn't fit in nicely anymore. (I don't really care about those few bytes, but I recall reading about this use case somewhere..)

Yes, I know the last commit will probably be discarded (that's why I put it the last), but I just want it to be considered as an option. I'm totally fine with 83 in the constant.

It is a bit disappointing that GetBoolArg() use from prior code is continued. Need a state-and-options struct.

It still needs to be a global. Later it can be an attribute of a global object, like in 4c412c9 but that global object can be "moved up" as a paramter (see ddcb6a1 ).

laanwj · 2015-07-13T07:06:02Z

Concept ACK - I've always thought policy doesn't belong in the Solver, which would be expected to determine what kind of script something is, then the policy to judge based on its output.

petertodd · 2015-07-17T15:25:20Z

utACK

petertodd · 2015-07-17T15:27:35Z

If you want to go ahead and rebase, that'd be great - I don't have the time/funding to do anything with this.

jtimon · 2015-07-17T19:35:30Z

Squashed everything but the last commit.

@petertodd Thanks for the review. As long as there are reviewers with time to re-review, I'm happy to re-rebase.

Thoughts on the last commit again?

petertodd · 2015-07-18T17:15:19Z

@jtimon I would strongly recommend you change the constant rather than the unit tests - people end up designing stuff based on things like "most miners will mine 80 byte op_return" so changing it can be highly disruptive.

jtimon · 2015-08-21T02:21:20Z

No input from anyone else besides @petertodd for a while, so I've just removed the last commit "fixup! Better yet, change the tests instead of the default constant" like he suggested.

dcousens · 2015-09-16T14:36:36Z

src/test/transaction_tests.cpp

@@ -350,12 +350,29 @@ BOOST_AUTO_TEST_CASE(test_IsStandard)
    t.vout[0].scriptPubKey = CScript() << OP_1;
    BOOST_CHECK(!IsStandardTx(t, reason));

-    // 80-byte TX_NULL_DATA (standard)
+    // MAX_OP_RETURN_RELAY-byte TX_NULL_DATA (standard)


MAX_OP_RETURN_RELAY length bytes, the hyphen seems odd here

dcousens · 2015-09-16T14:38:43Z

concept/utACK

jtimon · 2015-09-22T12:47:59Z

ping

sipa · 2015-09-22T16:34:56Z

Untested ACK.

laanwj · 2015-10-01T16:04:32Z

utACK, needs rebase

Previously unlike other transaction types the TX_SCRIPTHASH would not clear vSolutionsRet, which means that unlike other transaction types if it was called twice in a row you would get the result of the previous invocation as well.

Allows IsPushOnly() to be applied to just part of the script for OP_RETURN outputs.

Previously only one PUSHDATA was allowed, needlessly limiting applications such as matching OP_RETURN contents with bloom filters that operate on a per-PUSHDATA level. Now any combination that passes IsPushOnly() is allowed, so long as the total size of the scriptPubKey is less than 42 bytes. (unchanged modulo non-minimal PUSHDATA encodings) Also, this fixes the odd bug where previously the PUSHDATA could be replaced by any single opcode, even sigops consuming opcodes such as CHECKMULTISIG. (20 sigops!)

jtimon · 2015-10-01T16:42:27Z

Rebased

da894ab Accept any sequence of PUSHDATAs in OP_RETURN outputs (Peter Todd) 5d8709c Add IsPushOnly(const_iterator pc) (Peter Todd) 6a07eb6 Make TX_SCRIPTHASH clear vSolutionsRet first (Peter Todd)

laanwj · 2015-10-02T09:45:25Z

Forgot this, but @petertodd sending a tweet remindede me of it: this needs mention in doc/release-notes.md.

petertodd · 2015-10-03T13:00:14Z

@laanwj Fixed: #6751

Mention now allowed sequence of pushdatas in OP_RETURN outputs in release notes.

9204930 Document pull-req #6424 in release-notes (Peter Todd)

Mention now allowed sequence of pushdatas in OP_RETURN outputs in release notes.

NicolasDorier · 2015-11-09T14:10:12Z

What was the rational about allowing any push data in the OP_RETURN instead of just allowing any OP ? Why not accepting any operation ? After all, since the rest is never executed, we can save bytes by allowing any OP to be used, even invalid one. (Invalid OPs are standard if they are not executed)

jtimon · 2015-11-10T13:54:29Z

@NicolasDorier That's a very good question. In the original PR #5079 @petertodd says "Previously only one PUSHDATA was allowed, needlessly limiting applications such as matching OP_RETURN contents with bloom filters that operate on a per-PUSHDATA level."
So it seems that this additional limitation of in the standard policy of allowing only unspendable outputs that consist only on pushdata instructions (previously only one) seems to be related to bloom filters, presumable used in some protocol based on "proof of publication". But I personally don't know how these bloom filters are used or even for what.
Maybe @petertodd can explain more.

NicolasDorier · 2015-11-10T14:16:44Z

bloom filter is a way to get notified about content in OP_RETURN by SPV wallets with configurable compromise on privacy.

So this is not the reason, accepting all OP do not prevent you to use PushData in your OP_RETURN instead of garbage if you really need SPV capabilities.

jtimon · 2015-11-10T14:27:23Z

Thanks for the clarification, I'm thinking that I should have moved that check to policy::IsStandard() as well. But I don't really understand what is for, so I would like to hear an answer to your question as well.

petertodd · 2015-11-10T19:58:33Z

What was the rational about allowing any push data in the OP_RETURN instead of just allowing any OP ? Why not accepting any operation ? After all, since the rest is never executed, we can save bytes by allowing any OP to be used, even invalid one.

That's an clever observation, and I had the same idea too!

But unfortunately sigops are counted in outputs, which makes it tricky to be sure we're not opening up a some kind of DoS attack by allowing anything in OP_RETURN. Easiest was just to restrict this to only pushdata's, which is pretty much just as good anyway.

NicolasDorier · 2015-11-11T05:56:23Z

Ok, did not thought about sigops :(
Will the SIG OP rule will change one day ? Something like a limit count per script ?
I guess we can already be DoS by that now arbitrary redeem script are standard, and, most likely, miners does not even check the number of sig in a particular redeem.

sipa · 2015-11-11T06:13:04Z

@NicolasDorier That would be hard fork. I don't think anyone here has to authority to say whether or not that will change.

NicolasDorier · 2015-11-11T06:15:55Z

So what I mean is more, is there proposal being written to fix the issue right now ? (or even some args that the user/miner can turn on if we get spammed attacked with that so it is possible to refuse script with too much sig, we'll get some mitigation)

Bitcoin script PRs 1 Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#6335 - bitcoin/bitcoin#6424 - bitcoin/bitcoin#11058 - bitcoin/bitcoin#12460 - bitcoin/bitcoin#13194 Part of #2074.

5d96c91 add missing policy.cpp file to cmake (furszy) 2d0783d Standard Solver(): Include cold staking template matching. (furszy) b7fe4c0 Remove template matching and pseudo opcodes (furszy) fe31b31 Assert CPubKey::ValidLength to the pubkey's header-relevent size (Ben Woosley) 2f6b249 Make TX_SCRIPTHASH clear vSolutionsRet first (Peter Todd) 45e5102 Policy: MOVEONLY: 3 functions to policy (furszy) 2c7da7f Policy: MOVEONLY: Create policy/policy.h with some constants (furszy) Pull request description: Coming from the following PRs (with small customizations for our features): * bitcoin#6335 * bitcoin#6424 * bitcoin#12460 * bitcoin#13194 ACKs for top commit: random-zebra: Looking good. ACK 5d96c91 Fuzzbawls: ACK 5d96c91 Tree-SHA512: b3e7479864e16942682b9bc19f432b2a657561647eeadf0ef1285651feb05875d295ff530582ebbc8edacaf3c7d3888249513b4a93d871c2d6c2e1b157ace8c8

jtimon mentioned this pull request Jul 12, 2015

Accept any sequence of PUSHDATAs in OP_RETURN outputs #5079

Closed

dexX7 reviewed Jul 12, 2015
View reviewed changes

jtimon force-pushed the policy-datacarriersize-0.11.99 branch from df69832 to 984f7b4 Compare July 12, 2015 18:05

laanwj added the Refactoring label Jul 13, 2015

petertodd mentioned this pull request Jul 17, 2015

Allow OP_RETURN with two PUSHDATA #6396

Closed

jtimon force-pushed the policy-datacarriersize-0.11.99 branch from 984f7b4 to 0b21b27 Compare July 17, 2015 19:32

jtimon force-pushed the policy-datacarriersize-0.11.99 branch from 0b21b27 to 12f4303 Compare July 21, 2015 09:17

jtimon force-pushed the policy-datacarriersize-0.11.99 branch from 12f4303 to 935a0c0 Compare August 21, 2015 02:21

jtimon force-pushed the policy-datacarriersize-0.11.99 branch from 935a0c0 to 5ca12ee Compare August 25, 2015 22:50

dcousens reviewed Sep 16, 2015
View reviewed changes

laanwj added the TX fees and policy label Oct 1, 2015

petertodd added 3 commits October 1, 2015 18:27

Make TX_SCRIPTHASH clear vSolutionsRet first

6a07eb6

Previously unlike other transaction types the TX_SCRIPTHASH would not clear vSolutionsRet, which means that unlike other transaction types if it was called twice in a row you would get the result of the previous invocation as well.

Add IsPushOnly(const_iterator pc)

5d8709c

Allows IsPushOnly() to be applied to just part of the script for OP_RETURN outputs.

jtimon force-pushed the policy-datacarriersize-0.11.99 branch from 5ca12ee to da894ab Compare October 1, 2015 16:39

laanwj merged commit da894ab into bitcoin:master Oct 1, 2015

petertodd added a commit to petertodd/bitcoin that referenced this pull request Oct 5, 2015

Document pull-req bitcoin#6424 in release-notes

9204930

Mention now allowed sequence of pushdatas in OP_RETURN outputs in release notes.

laanwj added a commit that referenced this pull request Oct 6, 2015

Merge pull request #6751

b6f3a4e

9204930 Document pull-req #6424 in release-notes (Peter Todd)

luke-jr mentioned this pull request Oct 14, 2015

Backport bugfixes to 0.11 (2015-10-22 / f2c869a) #6825

Merged

luke-jr pushed a commit to luke-jr/bitcoin that referenced this pull request Oct 15, 2015

Document pull-req bitcoin#6424 in release-notes

420e93c

Mention now allowed sequence of pushdatas in OP_RETURN outputs in release notes.

dexX7 mentioned this pull request Aug 5, 2017

Update default datacarriersize to 220 byte Bitcoin-ABC/bitcoin-abc#41

Closed

dexX7 mentioned this pull request Feb 3, 2018

Add May 2018 hardfork specification bitcoincashorg/spec#53

Merged

dagurval mentioned this pull request Mar 16, 2018

Policy: Decouple Solver() from nMaxDatacarrierBytes (for free) bitcoinxt/bitcoinxt#359

Merged

str4d mentioned this pull request Nov 14, 2019

Bitcoin script PRs 1 zcash/zcash#4227

Merged

furszy mentioned this pull request Jun 18, 2020

Upstream scripts back ports [Step 1] PIVX-Project/PIVX#1693

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Policy: Decouple Solver() from nMaxDatacarrierBytes (for free) #6424

Policy: Decouple Solver() from nMaxDatacarrierBytes (for free) #6424

jtimon commented Jul 12, 2015

dexX7 Jul 12, 2015

jtimon Jul 12, 2015

dexX7 Jul 12, 2015

jtimon Jul 12, 2015

jgarzik Sep 16, 2015

jtimon Sep 17, 2015

laanwj commented Jul 13, 2015

petertodd commented Jul 17, 2015

petertodd commented Jul 17, 2015

jtimon commented Jul 17, 2015

petertodd commented Jul 18, 2015

jtimon commented Aug 21, 2015

dcousens Sep 16, 2015

dcousens commented Sep 16, 2015

jtimon commented Sep 22, 2015

sipa commented Sep 22, 2015

laanwj commented Oct 1, 2015

jtimon commented Oct 1, 2015

laanwj commented Oct 2, 2015

petertodd commented Oct 3, 2015

NicolasDorier commented Nov 9, 2015

jtimon commented Nov 10, 2015

NicolasDorier commented Nov 10, 2015

jtimon commented Nov 10, 2015

petertodd commented Nov 10, 2015

NicolasDorier commented Nov 11, 2015

sipa commented Nov 11, 2015

NicolasDorier commented Nov 11, 2015

Policy: Decouple Solver() from nMaxDatacarrierBytes (for free) #6424

Policy: Decouple Solver() from nMaxDatacarrierBytes (for free) #6424

Conversation

jtimon commented Jul 12, 2015

dexX7 Jul 12, 2015

Choose a reason for hiding this comment

jtimon Jul 12, 2015

Choose a reason for hiding this comment

dexX7 Jul 12, 2015

Choose a reason for hiding this comment

jtimon Jul 12, 2015

Choose a reason for hiding this comment

jgarzik Sep 16, 2015

Choose a reason for hiding this comment

jtimon Sep 17, 2015

Choose a reason for hiding this comment

laanwj commented Jul 13, 2015

petertodd commented Jul 17, 2015

petertodd commented Jul 17, 2015

jtimon commented Jul 17, 2015

petertodd commented Jul 18, 2015

jtimon commented Aug 21, 2015

dcousens Sep 16, 2015

Choose a reason for hiding this comment

dcousens commented Sep 16, 2015

jtimon commented Sep 22, 2015

sipa commented Sep 22, 2015

laanwj commented Oct 1, 2015

jtimon commented Oct 1, 2015

laanwj commented Oct 2, 2015

petertodd commented Oct 3, 2015

NicolasDorier commented Nov 9, 2015

jtimon commented Nov 10, 2015

NicolasDorier commented Nov 10, 2015

jtimon commented Nov 10, 2015

petertodd commented Nov 10, 2015

NicolasDorier commented Nov 11, 2015

sipa commented Nov 11, 2015

NicolasDorier commented Nov 11, 2015