Add capability to extend expiration time of pre fork claims #137

kaykurokawa · 2018-05-10T23:14:44Z

Details of hard fork for all:

The fork of the mainnet will be on block 400155 (will be around noon EST 7/9/18)

The fork of the testnet will be on block 280831 (will be around noon EST 5/21/18)

Claims made on and after the target hardfork blocks will expire after 2102400 blocks (approximately 10 years assuming block time 2.5 minutes).

Claims that were made before the hardfork and have not expired will have their expiration extended by the difference between the new expiration time and the previous expiration time (456 days).

Claims that already expired will remain expired.

Details for reviewers

This is further work off of : #115 in order to add the extension of claim expiration to already existing claims ( in #115 claims made before the hard fork would still expire at the old expiration time)

While working on this, I made two adjustments to #115. A) the first was that the new expiration time took effect 1 block after the target hard fork block, and not on the target hard fork block. It would be more intuitive if the new expiration time took effect on the target hard fork block. B) I fixed the logic of calling setExpirationTime() to only be called when we are at the target hard fork block, instead of on every block. This is more intuitive and in line with the soft-fork logic that is already present for Bitcoin. This means we need to also called setExpirationTime when we load the claimtrie from disk so I did that. The combination of A) and B) fixes a bug in #115 where if you loaded the claimtrie from disk (restarted lbrycrdd) than you expiration time for the first block after restart would be the old expiration time instead of the new one even after the hard fork.

The above work is on 2df3bb1

While working on extending the expiration of existing claims, I realized that #110 had the problem where it would not properly be able to remove expirations from the expiration queue properly. This is because the function removeFromExpirationQueue() : https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp#L1675 was not adjusted to consider the hard fork change in expiration time hence this line: https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp#L1677 could calculate the wrong expiration time for a claim that was created before the hard fork an expires after the hard fork.

The way we extend the claim is fairly straight forward. When we encounter the hard fork block, we set the new expiration time, and we look at all the entries in the expiration queue either in the dirty queue (where entries that are not saved to disk resides) or the disk database. The entries are adjusted to have their expiration time extended by the difference between the new expiration time and the old expiration time. Effectively, this makes it look like all the claims share the same new expiration time so the issue I described is solved.

The above work on extending the expiraiton of existing claims is on
b3aa62c

For the unit tests, I added a series of tests to see if supports behaved properly, and also added tests to see if the claimtrie behaved properly on the hard fork if I simulated a restart (write claimtrie to disk, clear it in memory, and than read it from disk). 6621f50

The other commits should be straight forward.

lbrynaut · 2018-05-15T17:34:44Z

src/chainparams.cpp

@@ -131,6 +131,9 @@ class CMainParams : public CChainParams {
        consensus.powLimit = uint256S("0000ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff");
        consensus.nPowTargetTimespan = 150; //retarget every block
        consensus.nPowTargetSpacing = 150;
+        consensus.nOriginalClaimExpirationTime = 262974;
+        consensus.nExtendedClaimExpirationTime = 2102400;
+        consensus.nExtendedClaimExpirationForkHeight = 400155; // FIXME: pick a real height


Remove FIXME comment

lbrynaut · 2018-05-15T17:34:54Z

src/chainparams.cpp

@@ -292,6 +298,9 @@ class CRegTestParams : public CChainParams {
        consensus.powLimit = uint256S("7fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff");
        consensus.nPowTargetTimespan = 1;//14 * 24 * 60 * 60; // two weeks
        consensus.nPowTargetSpacing = 1;
+        consensus.nOriginalClaimExpirationTime = 500;
+        consensus.nExtendedClaimExpirationTime = 600;
+        consensus.nExtendedClaimExpirationForkHeight = 800; // FIXME: pick a real height


Remove FIXME comment

lbrynaut · 2018-05-15T17:37:15Z

src/claimtrie.cpp

+    }
+
+
+    //look through db for expiration queues, if we haven't already found it in dirty expiraiton queue


expiration is spelled incorrectly in several places, maybe do a quick search for them?

lbrynaut · 2018-05-15T17:39:51Z

src/main.cpp

                        claimId = ClaimIdHash(hash, i);
+                        LogPrintf("--- %s[%lu]: OP_CLAIM_NAME \"%s\" = \"%s\" with claimId %s and tx prevout %s at index %d\n",
+                                  __func__, view.AccessCoins(hash)->nHeight, name, SanitizeString(value),


I realize I initially used the view height, but we could replace with pindex->nHeight (in all places it's used in debug)

lbrynaut · 2018-05-15T17:41:02Z

src/main.cpp

@@ -2220,6 +2232,13 @@ bool DisconnectBlock(const CBlock& block, CValidationState& state, const CBlockI
    assert(trieCache.finalizeDecrement());
    trieCache.setBestBlock(pindex->pprev->GetBlockHash());
    assert(trieCache.getMerkleHash() == pindex->pprev->hashClaimTrie);
+    if (pindex->nHeight == Params().GetConsensus().nExtendedClaimExpirationForkHeight)
+    {
+        LogPrintf("Decremented past the fork v13 fork height");


What is v13 fork? Why not call it the claim expiration fork or something more intuitive?

lbrynaut · 2018-05-15T17:41:28Z

src/test/claimtriebranching_tests.cpp

@@ -18,12 +18,12 @@
 #include <iostream>
 #include "test/test_bitcoin.h"

+


lbrynaut · 2018-05-15T17:41:55Z

src/test/claimtriebranching_tests.cpp

    {
        fRequireStandard = false;
        BOOST_CHECK(pclaimTrie->nCurrentHeight == chainActive.Height() + 1);
        LOCK(cs_main);
        num_txs_for_next_block = 0;
        num_txs = 0; 
        coinbase_txs_used = 0;
+
+
+
        // generate coinbases to spend 


Seems 1 line is enough?

lbrynaut · 2018-05-15T17:42:34Z

src/test/claimtriebranching_tests.cpp

+    void WriteClearReadClaimTrie()
+    {
+        // this will simulate restart of lbrycrdd by writing the claimtrie to disk,
+        // clearing the-in memory claimtrie, and then reading it the saved claimtrie


s/reading it/reading/

lbrynaut · 2018-05-15T17:43:25Z

src/test/claimtriebranching_tests.cpp

+        pclaimTrie->WriteToDisk();
+        pclaimTrie->clear();
+        pclaimTrie->ReadFromDisk(true);
+    }


This also doesn't do exactly what's in the comment since the Cache isn't modified/updated/rebuilt.

What do you mean here exactly ? You may have misssed where WriteToDisk clears out the cache entries https://github.com/lbryio/lbrycrd/blob/master/src/claimtrie.cpp#L1047 here

I was referring to the state of the TrieCache.

kaykurokawa · 2018-05-16T16:02:38Z

It was discovered that the testnet blocks haven't been progressing (mining was turned off) so I'll need to set a new fork height for the testnet.

kaykurokawa · 2018-05-16T19:28:31Z

made fixes as reviewed by lbrynaut

shyba · 2018-05-17T17:16:20Z

src/main.cpp

                        claimId = ClaimIdHash(hash, i);
+                        LogPrintf("--- %s[%lu]: OP_CLAIM_NAME \"%s\" = \"%s\" with claimId %s and tx prevout %s at index %d\n",


maybe log this only if in debug mode? otherwise getnameproof can become verbose as it currently disconnects and reconnects blocks for generating previous block proofs.

is there a debugging log?

shyba · 2018-05-17T17:25:06Z

src/test/claimtriebranching_tests.cpp

+    fixture.IncrementBlocks(fixture.expirationForkHeight - chainActive.Height());
+    BOOST_CHECK(is_best_claim("test2", u2));
+    BOOST_CHECK(best_claim_effective_amount_equals("test2",2));
+    // increment to original expiraiton, should not be expired


s/expiraiton/expiration/

oh, I see @lbrynaut already found that, sorry for the flood.

shyba · 2018-05-17T17:25:21Z

src/test/claimtriebranching_tests.cpp

+    BOOST_CHECK_EQUAL(chainActive.Height(), fixture.expirationForkHeight);
+    BOOST_CHECK_EQUAL(pclaimTrie->nExpirationTime, fixture.extendedExpiration);
+    // make sure decrementing to before the fork height will apppropriately set back the
+    // expiration time to the original expiraiton time


s/expiraiton/expiration/

kaykurokawa · 2018-05-22T01:46:01Z

Made fix as suggested by syba,
Adjusted fork height on testnet to 278160 (in about an hour from now) , and I will run this on testnet.
Maybe for debug logging comment by shyba, we can do another PR to perhaps improve/clean up different log messages?

shyba · 2018-05-23T03:54:26Z

@kaykurokawa yes, I agree. Created an issue for logging here.

… tests for extended claim expiration.

…t test for the extended expiration hardfork.

kaykurokawa · 2018-05-23T19:07:46Z

Rebased to two commits, rebased to master

// lbryio/lbrycrd#137

lbry-bot assigned kaykurokawa May 10, 2018

kaykurokawa force-pushed the claim-expiration-fixes branch from bc0caa6 to 6621f50 Compare May 15, 2018 02:28

kaykurokawa changed the title ~~[WIP] add capability to extend expiration time of pre fork claims~~ Add capability to extend expiration time of pre fork claims May 15, 2018

kaykurokawa requested review from shyba and lbrynaut May 15, 2018 16:59

lbry-bot assigned lbrynaut and shyba and unassigned kaykurokawa May 15, 2018

lbrynaut reviewed May 15, 2018

View reviewed changes

lbry-bot unassigned lbrynaut May 15, 2018

lbrynaut reviewed May 15, 2018

View reviewed changes

src/test/claimtriebranching_tests.cpp Outdated

@@ -18,12 +18,12 @@

#include <iostream>

#include "test/test_bitcoin.h"

Copy link

Contributor

lbrynaut May 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

lbrynaut reviewed May 15, 2018

View reviewed changes

lbrynaut approved these changes May 15, 2018

View reviewed changes

lbrynaut approved these changes May 16, 2018

View reviewed changes

lbry-bot unassigned shyba May 17, 2018

shyba approved these changes May 17, 2018

View reviewed changes

lbry-bot assigned kaykurokawa May 17, 2018

tiger5226 mentioned this pull request May 20, 2018

Hardfork rework - Block Expiration OdyseeTeam/chainquery#27

Closed

lbry-bot assigned kaykurokawa and unassigned kaykurokawa May 22, 2018

kaykurokawa force-pushed the claim-expiration-fixes branch from 5057389 to 01a222a Compare May 22, 2018 01:54

shyba mentioned this pull request May 23, 2018

Improve logging by separating debug from information #144

Closed

kaykurokawa force-pushed the claim-expiration-fixes branch from 01a222a to 338e7a3 Compare May 23, 2018 19:05

lbrynaut and others added 2 commits May 23, 2018 12:06

Implement a hard fork for extended claim expiration times. Added unit…

c44bd0a

… tests for extended claim expiration.

Extend expiration of pre fork claims and supports. Adjust and add uni…

99669e2

…t test for the extended expiration hardfork.

kaykurokawa force-pushed the claim-expiration-fixes branch from 338e7a3 to 99669e2 Compare May 23, 2018 19:07

kaykurokawa merged commit 4e147cb into master May 24, 2018

kaykurokawa mentioned this pull request May 28, 2018

Implement capability for a hard fork for extended/infinite claim expiration times #115

Closed

lyoshenka deleted the claim-expiration-fixes branch May 30, 2018 17:36

tiger5226 added a commit to OdyseeTeam/chainquery that referenced this pull request Jul 9, 2018

Added hardfork logic for expiration changes @ lbryio/lbrycrd#137

49afb00

tiger5226 added a commit to OdyseeTeam/chainquery that referenced this pull request Jul 9, 2018

Added hardfork logic for expiration changes @ lbryio/lbrycrd#137

10d359b

tiger5226 added a commit to lbryio/lbry.tech that referenced this pull request Jul 18, 2018

Update lbry-claimtrie.md

8850ea8

// lbryio/lbrycrd#137

tiger5226 mentioned this pull request Jul 18, 2018

Update lbry-claimtrie.md lbryio/lbry.tech#129

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add capability to extend expiration time of pre fork claims #137

Add capability to extend expiration time of pre fork claims #137

kaykurokawa commented May 10, 2018 •

edited

Loading

lbrynaut May 15, 2018

lbrynaut May 15, 2018

lbrynaut May 15, 2018

lbrynaut May 15, 2018

lbrynaut May 15, 2018

lbrynaut May 15, 2018

lbrynaut May 15, 2018

lbrynaut May 15, 2018

lbrynaut May 15, 2018

kaykurokawa May 16, 2018

lbrynaut May 16, 2018

kaykurokawa commented May 16, 2018

kaykurokawa commented May 16, 2018

shyba May 17, 2018

kaykurokawa May 22, 2018

shyba May 17, 2018

shyba May 17, 2018

shyba May 17, 2018

kaykurokawa May 22, 2018

kaykurokawa commented May 22, 2018 •

edited

Loading

shyba commented May 23, 2018

kaykurokawa commented May 23, 2018

		}


		//look through db for expiration queues, if we haven't already found it in dirty expiraiton queue

		@@ -18,12 +18,12 @@
		#include <iostream>
		#include "test/test_bitcoin.h"

		claimId = ClaimIdHash(hash, i);
		LogPrintf("--- %s[%lu]: OP_CLAIM_NAME \"%s\" = \"%s\" with claimId %s and tx prevout %s at index %d\n",

Add capability to extend expiration time of pre fork claims #137

Add capability to extend expiration time of pre fork claims #137

Conversation

kaykurokawa commented May 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaykurokawa commented May 16, 2018

kaykurokawa commented May 16, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaykurokawa commented May 22, 2018 • edited Loading

shyba commented May 23, 2018

kaykurokawa commented May 23, 2018

kaykurokawa commented May 10, 2018 •

edited

Loading

kaykurokawa commented May 22, 2018 •

edited

Loading