CHECKLOCKTIMEVERIFY (BIP65) IsSuperMajority() soft-fork #6351

petertodd · 2015-06-28T19:17:32Z

Final step towards CLTV deployment on mainnet.

I've copied the logic and tests from the previous BIP66 (DERSIG) soft-fork line-by-line for ease of review; any code review applicable to BIP66 should be applicable to BIP65.

Once merged I'll prepare a backport of the soft-fork logic for the v0.10.x branch as well.

jonasschnelli · 2015-06-29T11:06:12Z

IMO bip65-cltv-p2p.py should be included in extended list in pull-tester/rpc-tests.sh. Adding it in the "normal" not extended list makes not much sense even if the script execution time is acceptable (<1min on a decent system).

Tested ACK.

jtimon · 2015-06-29T12:24:19Z

I didn't reviewed the python tests, just the c++ part, but utACK on that part.

btcdrak · 2015-06-29T12:31:14Z

utACK

petertodd · 2015-06-29T23:33:43Z

@jonasschnelli Added the tests to the extended list. I didn't actually know about the pull-tester test list before, thanks!

mruddy · 2015-07-01T00:25:12Z

qa/rpc-tests/bip65-cltv.py

+        self.nodes[2].generate(1)
+        self.sync_all()
+        if (self.nodes[0].getblockcount() != cnt + 1051):
+            raise AssertionError("Failed to mine a version=3 block")


Line 70 should be: Failed to mine a version=4 block

mruddy · 2015-07-01T00:55:35Z

Looks pretty good. Minor todo's with the python test scripts.

Needs some release notes, but you might have been waiting on those until later in the process. When you get to them, perhaps add a disclaimer that we may change the OP_NOP2 name decode to OP_CHECKLOCKTIMEVERIFY in script "asm" outputs in the future. That way we'll have the flexibility to make that change if/when we get to it.

petertodd · 2015-07-01T21:35:20Z

So, @mruddy brought up two issues that are really the same as with the BIP66 tests: out of date copyright headers and TODO items that are out of date. The latter issue is pretty simple: bipdersig-p2p.py covers the missing test cases marked as "TODO" in bipdersig.py

I wanted to show I've replicated the exact same testing procedure done on BIP66 given that the code changes itself are also replicated exactly; if I'm going to fix these two nits I think it makes sense to fix them in the BIP66 tests first, in a separate commit that'll go first in this pull-req, so the two sets of tests can still be diffed directly to show exact equivalence. Makes sense?

@mruddy Yup, I'll add release notes later, and similarly, will look at what changing the NOP2 name will entail later. Right now I'm focused on making the minimal possible change that gets the job done so backporting the changes to v0.10.x (and probably v0.11.x) will be easy and clearly correct. v0.12.x can do the more invasive stuff of changing opcode names and what-not.

mruddy · 2015-07-02T12:13:01Z

@petertodd I see what you're saying about trying to keep the diff compares as close and easy as possible. But, the diffs on the tests are necessarily going to be somewhat different. I wouldn't make updated BIP66 test cases part of this change set. My choice would be to make the minor cleanups to these tests so that they are cleaner when reviewed in isolation (and in the future, not within the context of how this change set compares to that of BIP66). I can see how line-only diffs vs some block or structural diffs might have some value to reduce dissonance though. So, however you want to handle it is OK with me.

About the decode name update, I agree, leave that until later. I was just saying that if you mention the upcoming change in the notes now, then it makes actually pushing it through later, easier. It's more of a busy work change anyways, so leave it for someone newer to the project than yourself to work on. It's good for someone newer to work on because of how it may affect various parts of the code base.

petertodd · 2015-07-04T06:16:53Z

@mruddy Thanks! It'd be awesome for you to do that, and it'd be a good opportunity to have more eyes on the code too of course.

Anyway, I'll see what the other reviewers think re: the tests.

mruddy · 2015-07-04T18:14:48Z

I probably will, it's on my todo list.

mruddy · 2015-08-02T09:48:03Z

What do we need to do now to move this forward? Is this waiting on coordination with some other work, in need of more testing, or just baking and giving time for more review (it's been about a month since last update)?

jtimon · 2015-08-04T11:00:41Z

src/main.cpp

    if (block.nVersion >= 3 && IsSuperMajority(3, pindex->pprev, chainparams.GetConsensus().nMajorityEnforceBlockUpgrade, chainparams.GetConsensus())) {
        flags |= SCRIPT_VERIFY_DERSIG;
    }

+    // Start enforcing CHECKLOCKTIMEVERIFY, (BIP65) for block.nVersion=4
+    // blocks, when 75% of the network has upgraded:
+    if (block.nVersion >= 4 && IsSuperMajority(4, pindex->pprev, chainparams.GetConsensus().nMajorityEnforceBlockUpgrade, chainparams.GetConsensus())) {


I still don't know why we wait for 75% to start using a policy rule.
I would do it from the start but only enforce it as a consensus rule after 95%.
I don't want to slow this down since it is a general softfork question (even if I'm right it can be solved after this PR), but I haven't been able to find an answer anywhere.

This is not a policy rule.

While individual miners apply it but still not enforce it as a consensus rule (ie reject blocks that don't apply it), it is just mining policy.
What am I missing?

Seriously, what am I missing here? Why wait for 75% for enforcing the new rule in your own blocks (which by the way you are marking as being version 4)?

This code is executed against all blocks, not just blocks you created. If another miner creates a nVersion=4 block with an invalid use of CHECKLOCKTIMEVERIFY if you reject it without a majority of hashing power also rejecting it you'll get forked.

In fact, it might be a good idea to go the other way and remove the 75% rule, accepting invalid nVersion=4 blocks until 95%

Thanks, for some reason I was fixated on this 75% threshold only affecting your own blocks, but this is ConnectBlock so it obviously affects all blocks. Yes, I wouldn't oppose to wait for the 95% here too. But with versionbits this disappears so is probably not worth to discuss this much.

Yeah, I probably would have changed it to remove 75% a year ago... but at this stage I think it's harmless to leave in.

jtimon · 2015-08-04T11:04:07Z

@mruddy I would say that, as usual, we're just waiting for more review, but IMO the harder-to-review parts have already been merged. I would really like that we move forward on this.

petertodd · 2015-08-04T11:06:40Z

I think what's holding this up is a lack of political will to do yet another non-nVersion bits soft-fork upgrade; @gmaxwell has indicated a strong preference for never using IsSuperMajority() again.

On 4 August 2015 07:04:50 GMT-04:00, "Jorge Timón" notifications@github.com wrote:

@mruddy I would say that, as usual, we're just waiting for more review,
but IMO the harder-to-review parts have already been merged. I would
really like that we move forward on this.

Reply to this email directly or view it on GitHub:
#6351 (comment)

btcdrak · 2015-08-04T16:20:44Z

It would seem a better plan to deploy this softfork now rather than waiting for nVersionbits code which isnt done yet. We know this method works and it doesnt get in the way of nVersionbits deployment later. Many players in the ecosystem are waiting for CLTV, I dont see why we should wait for perfection when we can deploy a much anticipated feature now. @sipa @gmaxwell @laanwj

jl2012 · 2015-08-04T17:39:19Z

Do we have any softfork ready to deploy in the coming 4 months? If no I don't think we need to wait for nVerionbits. I guess BIP62 is not ready yet.

Is there any plan to prevent a fork due to reckless SPV mining?

By the way, if BIP101 is implemented in Bitcoin XT, blocks will have a version number with bits 1,2,3 and 14 set (0x20000007 in hex). In that case, it is not possible to support BIP101 without also supporting BIP65. @gavinandresen @mikehearn

btcdrak · 2015-08-04T22:44:15Z

Just for the record, @sipa ACK'd for IsSuperMajority() on the ML

dcousens · 2015-08-05T01:29:13Z

@gmaxwell has indicated a strong preference for never using IsSuperMajority() again.

@gmaxwell can you confirm this?

maaku · 2015-08-05T16:45:09Z

@dcousens this pull request shouldn't be held up waiting for a response from someone who has never participated in it.

@jl2012 there is BIP68, and the as-yet unnumbered CHECKSEQUENCEVERIFY and median-lock-time-past BIPs, all of which are related to CHECKTIMELOCKVERIFY and could be rolled out together...

jl2012 · 2015-08-06T13:02:02Z

@maaku I think we could go with IsSuperMajority(). If BIP68 or other related BIPs are ready before BIP65 is completed, they could be considered as "addon" of BIP65. People will use nVersion=5 to support BIP65 and those addon.

btcdrak · 2015-08-06T22:41:24Z

@maaku I was thinking the same thing recently that rolling out CLTV and CSV opcodes as one upgrade would be ideal and most efficient but when are you planning to make the PR for CSV? Seems like the code is there in your fork...

If this is going to take you a while then in my opinion we should not hold up CLTV, but I think it would be a wasted opportunity not to bundle the features you list as one soft-fork.

maaku · 2015-08-06T22:50:58Z

The code is done, ready, tested, deployed to alpha testnet and known to
work.

What remains to be done is writing a BIP... Assistance is welcome if
someone wants to help speed this along. I'm being pulled in 10 different
directions and having trouble allocating the time necessary.

On Thu, Aug 6, 2015 at 3:42 PM, ฿tcDrak notifications@github.com wrote:

@maaku https://github.com/maaku I was thinking the same thing recently
that rolling out CLTV and CSV opcodes as one upgrade would be ideal and
most efficient but when are you planning to make the PR for CSV? Seems like
the code is there in your fork...

If this is going to take you a while then in my opinion we should not hold
up CLTV, but I think it would be a wasted opportunity not to bundle the
features you list as one soft-fork.

—
Reply to this email directly or view it on GitHub
#6351 (comment).

jtimon · 2015-08-06T22:58:10Z

+1 to @btcdrak 's thoughts.

TheBlueMatt · 2015-08-08T16:36:25Z

Concept ACK and +1 to @btcdrak. I'd suggest holding off merging this until either CSV is ready or a week or two before a new release is going to freeze.

dcousens · 2015-10-07T12:57:59Z

It also has to reject everything the old code rejected.

How would that be tested?

petertodd · 2015-10-07T13:03:32Z

@dcousens With great difficulty.... That's one of the biggest challenges in changing consensus critical code.

CodeShark · 2015-10-07T14:59:12Z

@petertodd I created a new PR #6774 and rebased #6747 atop it to make the relevant code changes as explicit as possible.I also added the mechanism for your PR to it without activating it yet so that we're ready to go with BIP65.

In any case, I don't want this to hold up deploying BIP65 and I understand you patterned it after BIP66. My intention was to make it even easier to compare the two (or any other soft fork deployments we make in the future, for that matter).

Consensus-critical code does indeed present challenges, but I hope the relatively few code movements in these PRs are very simple to follow, now. Checking its correctness should basically amount to ensuring there were no copy/paste errors.

I ran the comparison tool locally and it passed. It also passed Travis' barrage of tests.

Based on the earlier BIP66 soft-fork logic implemented by Pieter Wuille's 5a47811

bip65-cltv.py is based on the earlier BIP66 soft-fork RPC test implemented by Pieter Wuille's 819bcf9 bip65-cltv-p2p.py is based on the earlier BIP66 P2P test by Suhas Daftuar's d76412b

petertodd · 2015-10-08T15:38:33Z

Rebased and added BIP65 to getblockchaininfo softforks list.

CodeShark · 2015-10-09T02:03:13Z

@petertodd I created a branch demonstrating how we will deploy soft forks using #6747. The deployment logic for BIP65 can be conveniently compared side-by-side with that for BIP66, and future soft forks will also be possible to compare in this fashion regardless of whether we use IsSuperMajority or versionbits.

grep for "DEPLOY BIP65" to see the the hooks.

btcdrak · 2015-10-22T20:45:38Z

ACK

65ef372 Add BIP65 to getblockchaininfo softforks list (Peter Todd) cde7ab2 Add RPC tests for the CHECKLOCKTIMEVERIFY (BIP65) soft-fork (Peter Todd) 287f54f Add CHECKLOCKTIMEVERIFY (BIP65) soft-fork logic (Peter Todd)

NicolasDorier · 2015-10-24T09:43:02Z

By making my test, I noticed something odd...

// Now that we know we're comparing apples-to-apples, the
// comparison is a simple numeric one.
if(nLockTime > (long)txTo.LockTime)
    return false;

This means that if nLockTime == txTo.LockTime, then the script evaluate to true, which made me believe that I can spend "1000 OP_CLTV" from exactly block 1000.

However, such a transaction would not be final as IsFinalTx show :

    if ((int64_t)tx.nLockTime < ((int64_t)tx.nLockTime < LOCKTIME_THRESHOLD ? (int64_t)nBlockHeight : nBlockTime))
        return true;

We can see that if such transaction is propagated at block 1000 it will be rejected as NonFinal !
This mean that 1000 OP_CLTV can only be spent from block 1001, even if the script pass at block 1000.

Is this the expected behavior ? This seems kind of wierd...
I would have changed to

// Now that we know we're comparing apples-to-apples, the
// comparison is a simple numeric one.
if(nLockTime > (long)txTo.LockTime - 1)
    return false;

So the script "1000 OP_CLTV" only pass from block 1001.

jl2012 · 2015-10-24T15:42:02Z

@NicolasDorier

So the script "1000 OP_CLTV" only pass from block 1001.

It is consistent with the meaning of nLockTime, i.e. a tx is final only after (not at or before) the nLockTime. "1000 OP_CLTV" means the output is spendable after block 1000.

1000 OP_CLTV can only be spent from block 1001, even if the script pass at block 1000.

It is always possible to create a valid script at block X, but is not final at block X. For example if the current height is 380000, a script of "390000 OP_CLTV" with nLockTime = 391000 is still valid but not final

NicolasDorier · 2015-10-24T17:25:13Z

It is consistent with the meaning of nLockTime, i.e. a tx is final only after (not at or before) the nLockTime. "1000 OP_CLTV" means the output is spendable after block 1000.

This is what is not consistent ! "1000 OP_CLTV" is currently true if nLockTime is 1000. (1000 included)

Whereas a nLocktime of 1000 at block 1000 is not final as you rightly said " tx is final only after (not at or before)"

A consistent behavior would be "1000 OP_CLTV" false with a nLockTime of 1000.

jl2012 · 2015-10-24T17:37:07Z

A consistent behavior would be "1000 OP_CLTV" false with a nLockTime of 1000.

In that case you need a nLockTime of 1001 to spend "1000 OP_CLTV", which means unspendable until block 1002.

Anyway, by "consistent" I mean:

With "X OP_CLTV", an output is spendable after block X.
With nLockTime = X, the tx is valid after block X

NicolasDorier · 2015-10-25T04:28:31Z

With "X OP_CLTV", an output is spendable after block X.

No, with "X OP_CLTV" the output is spendable AT block X.

With nLockTime = X, the tx is valid after block X

Yes, in other words nLockTime=X the tx is valid at block X+1.

This is the inconsistency.
Because of 2., it means that X OP_CLTV is spendable at X+1 even if the script returns true at X. This is what is not very coherent.

gmaxwell · 2015-10-25T04:59:57Z

@NicolasDorier it sounds like you are misunderstanding the design of OP_CLTV. There is no "return true" in it. It is a VERIFY op_code. It checks that the nlocktime field in the transaction agrees with the value on the stack, if it does-- it does nothing. If it does not, it terminates execution. It has no clue about the height of the blockchain, and if it did it would make script execution no long a pure function of the transaction, violate the layering between transactions and the blockchain, make it possible for a script to become invalid that was valid before, etc.

NicolasDorier · 2015-10-25T06:44:04Z

@gmaxwell I understand that.
What I am saying is that from a script perpective "X OP_CLTV" means "can spend AT and AFTER block X".

While nLockTime=X means "can broadcast AT and AFTER block X+1".

Both of these means that "X OP_CLTV" will always been able to me spent at X+1, even if the X OP_CLTV will pass with nLockTime of X.

Well, I don't mind though, if it is the expected behavior, I have nothing to complain, sounds weird for me but this seems to be just a personal perspective rather than a mistake in the code. So I'm fine with it. I was just worried it was a mistake.

gmaxwell · 2015-10-25T06:57:02Z

@NicolasDorier CLTV has nothing to do with blocks, X OP_CLTV means is "this transaction is invalid if X > the transaction's nLocktime field". What effect the nlocktime field has is largely a mystery to OP_CLTV.

NicolasDorier · 2015-10-25T16:24:00Z

Ok now by thinking about it like you said,
If I want to say "Can spend this output at block X" then I need to use :
"X-1 OP_CTLV" and
at least nLockTime = X-1 when spending which is coherent.

I was indeed making things more complicated than they needed to be, I'm fine now !

petertodd · 2015-10-25T17:36:55Z

Great!

If you think there's a way to better explain CLTV in the BIP, pull-reqs welcome.

On 25 October 2015 12:24:41 GMT-04:00, Nicolas Dorier notifications@github.com wrote:

Ok now by thinking about it like you said,
If I want to say "Can spend this output at block X" then I need to use
:
"X-1 OP_CTLV" and
at least nLockTime = X-1 when spending which is coherent indeed.

I was indeed making things more complicated than they needed to be, I'm
fine now !

Reply to this email directly or view it on GitHub:
#6351 (comment)

petertodd force-pushed the cltv-is-super-majority-soft-fork branch from 4053ac5 to 1706aa0 Compare June 29, 2015 23:23

mruddy reviewed Jul 1, 2015
View reviewed changes

petertodd force-pushed the cltv-is-super-majority-soft-fork branch from 1706aa0 to 3082578 Compare July 1, 2015 21:35

laanwj added the Validation label Jul 3, 2015

petertodd mentioned this pull request Jul 17, 2015

Restore minimum feerate to 10000 satoshis #6201

Closed

jtimon reviewed Aug 4, 2015
View reviewed changes

petertodd added 2 commits October 8, 2015 17:23

Add CHECKLOCKTIMEVERIFY (BIP65) soft-fork logic

287f54f

Based on the earlier BIP66 soft-fork logic implemented by Pieter Wuille's 5a47811

Add RPC tests for the CHECKLOCKTIMEVERIFY (BIP65) soft-fork

cde7ab2

bip65-cltv.py is based on the earlier BIP66 soft-fork RPC test implemented by Pieter Wuille's 819bcf9 bip65-cltv-p2p.py is based on the earlier BIP66 P2P test by Suhas Daftuar's d76412b

petertodd force-pushed the cltv-is-super-majority-soft-fork branch from 3082578 to 1d6365b Compare October 8, 2015 15:36

Add BIP65 to getblockchaininfo softforks list

65ef372

petertodd force-pushed the cltv-is-super-majority-soft-fork branch from 1d6365b to 65ef372 Compare October 8, 2015 15:38

laanwj merged commit 65ef372 into bitcoin:master Oct 23, 2015

btcdrak mentioned this pull request Oct 23, 2015

doc: mention BIP65 softfork in bips.md #6879

Merged

ryanxcharles mentioned this pull request Oct 23, 2015

BIP65 - OP_CHECKLOCKTIMEVERIFY moneybutton/yours-bitcoin#16

Closed

NicolasDorier added a commit to MetacoSA/NBitcoin that referenced this pull request Oct 24, 2015

OP_CHECKLOCKTIMEVERIFY is standard bitcoin/bitcoin#6351

55a820d

dcousens mentioned this pull request Oct 31, 2015

BIP65: Change to accepted status bitcoin/bips#233

Merged

btcdrak mentioned this pull request Nov 17, 2015

BIP-68: Mempool-only sequence number constraint verification #6312

Closed

RandyMcMillan mentioned this pull request Feb 2, 2021

doc/bips: Add BIPs 43, 44, 49, and 84 #21028

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

CHECKLOCKTIMEVERIFY (BIP65) IsSuperMajority() soft-fork #6351

CHECKLOCKTIMEVERIFY (BIP65) IsSuperMajority() soft-fork #6351

Conversation

petertodd commented Jun 28, 2015

jonasschnelli commented Jun 29, 2015

jtimon commented Jun 29, 2015

btcdrak commented Jun 29, 2015

petertodd commented Jun 29, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mruddy commented Jul 1, 2015

petertodd commented Jul 1, 2015

mruddy commented Jul 2, 2015

petertodd commented Jul 4, 2015

mruddy commented Jul 4, 2015

mruddy commented Aug 2, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtimon commented Aug 4, 2015

petertodd commented Aug 4, 2015

btcdrak commented Aug 4, 2015

jl2012 commented Aug 4, 2015

btcdrak commented Aug 4, 2015

dcousens commented Aug 5, 2015

maaku commented Aug 5, 2015

jl2012 commented Aug 6, 2015

btcdrak commented Aug 6, 2015

maaku commented Aug 6, 2015

jtimon commented Aug 6, 2015

TheBlueMatt commented Aug 8, 2015

dcousens commented Oct 7, 2015

petertodd commented Oct 7, 2015

CodeShark commented Oct 7, 2015

petertodd commented Oct 8, 2015

CodeShark commented Oct 9, 2015

btcdrak commented Oct 22, 2015

NicolasDorier commented Oct 24, 2015

jl2012 commented Oct 24, 2015

NicolasDorier commented Oct 24, 2015

jl2012 commented Oct 24, 2015

NicolasDorier commented Oct 25, 2015

gmaxwell commented Oct 25, 2015

NicolasDorier commented Oct 25, 2015

gmaxwell commented Oct 25, 2015

NicolasDorier commented Oct 25, 2015

petertodd commented Oct 25, 2015