Skip to content
This repository

Relay blocks as a "preview" before checking the transactions in them #1586

Closed
wants to merge 2 commits into from

3 participants

Luke-Jr Jeff Garzik BitcoinPullTester
Luke-Jr

DRAFT status, please help test! Trying to coordinate testing on IRC, possibly using NTP and https://gist.github.com/3094657

Jeff Garzik
Collaborator

@gmaxwell and I are definitely interested in this, at least. Will keep this pull request open.

A rebase would be nice, if you have time.

Luke-Jr

Unfortunately, it seems p2p connections are handled synchronously, so everything this does to rush out the relaying is for naught. The only simple way I can see to refactor this to be asynchronous would be either to use coroutines (which Boost won't have until 2013) or one thread per command/connection. Either one is probably too late for 0.7, as would be an even larger refactor, I presume?

BitcoinPullTester

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/4e54ea804ccdd2223e622497f0d46cceb27b9d22 for binaries and test log.

BitcoinPullTester

Automatic sanity-testing: FAILED MERGE, see http://jenkins.bluematt.me/pull-tester/4e54ea804ccdd2223e622497f0d46cceb27b9d22 for test log.

This pull does not merge cleanly onto current master
This test script verifies pulls every time they are updated. It, however, dies sometimes and fails to test properly. If you are waiting on a test, please check timestamps to verify that the test.log is moving at http://jenkins.bluematt.me/pull-tester/current/
Contact BlueMatt on freenode if something looks broken.

Jeff Garzik
Collaborator

Closing - timeout. Interesting proposal, needs more work and dev momentum.

Jeff Garzik jgarzik closed this August 24, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 2 unique commits by 1 author.

Aug 12, 2012
Luke-Jr Relay blocks as a "preview" before checking the transactions in them 0fc6cf6
Aug 13, 2012
Luke-Jr Begin relaying blocks immediately after doing validity checks on the …
…header, even before our own download has completed
4e54ea8
This page is out of date. Refresh to see the latest.
167  src/main.cpp
@@ -1741,34 +1741,42 @@ bool CBlock::AddToBlockIndex(unsigned int nFile, unsigned int nBlockPos)
1741 1741
 
1742 1742
 
1743 1743
 
1744  
-bool CBlock::CheckBlock(bool fCheckPOW, bool fCheckMerkleRoot) const
  1744
+bool CBlock::CheckBlockHeader(bool fCheckPOW) const
1745 1745
 {
1746  
-    // These are checks that are independent of context
1747  
-    // that can be verified before saving an orphan block.
1748  
-
1749  
-    // Size limits
1750  
-    if (vtx.empty() || vtx.size() > MAX_BLOCK_SIZE || ::GetSerializeSize(*this, SER_NETWORK, PROTOCOL_VERSION) > MAX_BLOCK_SIZE)
1751  
-        return DoS(100, error("CheckBlock() : size limits failed"));
  1746
+    // These are checks that are independent of transaction count/content
  1747
+    // that can be verified with just the block header
1752 1748
 
1753 1749
     // Check proof of work matches claimed amount
1754 1750
     if (fCheckPOW && !CheckProofOfWork(GetHash(), nBits))
1755  
-        return DoS(50, error("CheckBlock() : proof of work failed"));
  1751
+        return DoS(50, error("CheckBlockHeader() : proof of work failed"));
1756 1752
 
1757 1753
     // Check timestamp
1758 1754
     if (GetBlockTime() > GetAdjustedTime() + 2 * 60 * 60)
1759  
-        return error("CheckBlock() : block timestamp too far in the future");
  1755
+        return error("CheckBlockHeader() : block timestamp too far in the future");
  1756
+
  1757
+    return true;
  1758
+}
  1759
+
  1760
+bool CBlock::CheckBlockBody(bool fCheckMerkleRoot) const
  1761
+{
  1762
+    // These are checks that are independent of context
  1763
+    // that can be verified before saving an orphan block.
  1764
+
  1765
+    // Size limits
  1766
+    if (vtx.empty() || vtx.size() > MAX_BLOCK_SIZE || ::GetSerializeSize(*this, SER_NETWORK, PROTOCOL_VERSION) > MAX_BLOCK_SIZE)
  1767
+        return DoS(100, error("CheckBlockBody() : size limits failed"));
1760 1768
 
1761 1769
     // First transaction must be coinbase, the rest must not be
1762 1770
     if (vtx.empty() || !vtx[0].IsCoinBase())
1763  
-        return DoS(100, error("CheckBlock() : first tx is not coinbase"));
  1771
+        return DoS(100, error("CheckBlockBody() : first tx is not coinbase"));
1764 1772
     for (unsigned int i = 1; i < vtx.size(); i++)
1765 1773
         if (vtx[i].IsCoinBase())
1766  
-            return DoS(100, error("CheckBlock() : more than one coinbase"));
  1774
+            return DoS(100, error("CheckBlockBody() : more than one coinbase"));
1767 1775
 
1768 1776
     // Check transactions
1769 1777
     BOOST_FOREACH(const CTransaction& tx, vtx)
1770 1778
         if (!tx.CheckTransaction())
1771  
-            return DoS(tx.nDoS, error("CheckBlock() : CheckTransaction failed"));
  1779
+            return DoS(tx.nDoS, error("CheckBlockBody() : CheckTransaction failed"));
1772 1780
 
1773 1781
     // Check for duplicate txids. This is caught by ConnectInputs(),
1774 1782
     // but catching it earlier avoids a potential DoS attack:
@@ -1778,7 +1786,7 @@ bool CBlock::CheckBlock(bool fCheckPOW, bool fCheckMerkleRoot) const
1778 1786
         uniqueTx.insert(tx.GetHash());
1779 1787
     }
1780 1788
     if (uniqueTx.size() != vtx.size())
1781  
-        return DoS(100, error("CheckBlock() : duplicate transaction"));
  1789
+        return DoS(100, error("CheckBlockBody() : duplicate transaction"));
1782 1790
 
1783 1791
     unsigned int nSigOps = 0;
1784 1792
     BOOST_FOREACH(const CTransaction& tx, vtx)
@@ -1786,15 +1794,20 @@ bool CBlock::CheckBlock(bool fCheckPOW, bool fCheckMerkleRoot) const
1786 1794
         nSigOps += tx.GetLegacySigOpCount();
1787 1795
     }
1788 1796
     if (nSigOps > MAX_BLOCK_SIGOPS)
1789  
-        return DoS(100, error("CheckBlock() : out-of-bounds SigOpCount"));
  1797
+        return DoS(100, error("CheckBlockBody() : out-of-bounds SigOpCount"));
1790 1798
 
1791 1799
     // Check merkle root
1792 1800
     if (fCheckMerkleRoot && hashMerkleRoot != BuildMerkleTree())
1793  
-        return DoS(100, error("CheckBlock() : hashMerkleRoot mismatch"));
  1801
+        return DoS(100, error("CheckBlockBody() : hashMerkleRoot mismatch"));
1794 1802
 
1795 1803
     return true;
1796 1804
 }
1797 1805
 
  1806
+bool CBlock::CheckBlock(bool fCheckPOW, bool fCheckMerkleRoot) const
  1807
+{
  1808
+    return CheckBlockHeader(fCheckPOW) && CheckBlockBody(fCheckMerkleRoot);
  1809
+}
  1810
+
1798 1811
 bool CBlock::AcceptBlock()
1799 1812
 {
1800 1813
     // Check for duplicate
@@ -1849,18 +1862,17 @@ bool CBlock::AcceptBlock()
1849 1862
     return true;
1850 1863
 }
1851 1864
 
1852  
-bool ProcessBlock(CNode* pfrom, CBlock* pblock)
  1865
+bool ProcessBlockHeader(CNode* pfrom, CBlock* pblock, const uint256& hash, bool& fEnablePreview)
1853 1866
 {
1854 1867
     // Check for duplicate
1855  
-    uint256 hash = pblock->GetHash();
1856 1868
     if (mapBlockIndex.count(hash))
1857  
-        return error("ProcessBlock() : already have block %d %s", mapBlockIndex[hash]->nHeight, hash.ToString().substr(0,20).c_str());
  1869
+        return error("ProcessBlockHeader() : already have block %d %s", mapBlockIndex[hash]->nHeight, hash.ToString().substr(0,20).c_str());
1858 1870
     if (mapOrphanBlocks.count(hash))
1859  
-        return error("ProcessBlock() : already have block (orphan) %s", hash.ToString().substr(0,20).c_str());
  1871
+        return error("ProcessBlockHeader() : already have block (orphan) %s", hash.ToString().substr(0,20).c_str());
1860 1872
 
1861 1873
     // Preliminary checks
1862  
-    if (!pblock->CheckBlock())
1863  
-        return error("ProcessBlock() : CheckBlock FAILED");
  1874
+    if (!pblock->CheckBlockHeader(true))
  1875
+        return error("ProcessBlockHeader() : CheckBlockHeader FAILED");
1864 1876
 
1865 1877
     CBlockIndex* pcheckpoint = Checkpoints::GetLastCheckpoint(mapBlockIndex);
1866 1878
     if (pcheckpoint && pblock->hashPrevBlock != hashBestChain)
@@ -1871,7 +1883,7 @@ bool ProcessBlock(CNode* pfrom, CBlock* pblock)
1871 1883
         {
1872 1884
             if (pfrom)
1873 1885
                 pfrom->Misbehaving(100);
1874  
-            return error("ProcessBlock() : block with timestamp before last checkpoint");
  1886
+            return error("ProcessBlockHeader() : block with timestamp before last checkpoint");
1875 1887
         }
1876 1888
         CBigNum bnNewBlock;
1877 1889
         bnNewBlock.SetCompact(pblock->nBits);
@@ -1881,15 +1893,48 @@ bool ProcessBlock(CNode* pfrom, CBlock* pblock)
1881 1893
         {
1882 1894
             if (pfrom)
1883 1895
                 pfrom->Misbehaving(100);
1884  
-            return error("ProcessBlock() : block with too little proof-of-work");
  1896
+            return error("ProcessBlockHeader() : block with too little proof-of-work");
  1897
+        }
  1898
+    }
  1899
+
  1900
+    // Only enable block preview broadcasts when the difficulty in making them is high enough that it's not worth DoSing with them
  1901
+    {
  1902
+        double dDiff = (double)0x0000ffff / (double)(pblock->nBits & 0x00ffffff);
  1903
+        for (int nShift = pblock->nBits >> 24; nShift < 29; ++nShift)
  1904
+            dDiff *= 0x100;
  1905
+        fEnablePreview = dDiff >= 0x20000;
  1906
+    }
  1907
+
  1908
+    if (pblock->hashPrevBlock == pindexBest->GetBlockHash())
  1909
+    {
  1910
+        // We can check the immediate next block's time and bits headers earlier
  1911
+        if (pblock->GetBlockTime() <= pindexBest->GetMedianTimePast())
  1912
+            return error("ProcessBlockHeader() : block's timestamp is too early");
  1913
+
  1914
+        if (pblock->nBits != GetNextWorkRequired(pindexBest, pblock))
  1915
+            return pblock->DoS(100, error("ProcessBlockHeader() : incorrect proof of work"));
  1916
+
  1917
+        if (fEnablePreview)
  1918
+        {
  1919
+            // Relay blocks after only checking their header, so we don't bias them against including transactions too much
  1920
+            pblock->csBlockDownload = new std::pair<boost::mutex, boost::condition_variable>();
  1921
+            pblock->fBlockDownloading = true;
  1922
+            RelayMessage(CInv(MSG_BLOCK_PREVIEW, hash), *pblock);
1885 1923
         }
1886 1924
     }
1887 1925
 
  1926
+    return true;
  1927
+}
  1928
+
  1929
+bool ProcessBlockBody(CNode* pfrom, CBlock* pblock, const uint256& hash)
  1930
+{
  1931
+    if (!pblock->CheckBlockBody(true))
  1932
+        return error("ProcessBlockBody() : CheckBlockBody FAILED");
1888 1933
 
1889 1934
     // If don't already have its previous block, shunt it off to holding area until we get it
1890 1935
     if (!mapBlockIndex.count(pblock->hashPrevBlock))
1891 1936
     {
1892  
-        printf("ProcessBlock: ORPHAN BLOCK, prev=%s\n", pblock->hashPrevBlock.ToString().substr(0,20).c_str());
  1937
+        printf("ProcessBlockBody: ORPHAN BLOCK, prev=%s\n", pblock->hashPrevBlock.ToString().substr(0,20).c_str());
1893 1938
         CBlock* pblock2 = new CBlock(*pblock);
1894 1939
         mapOrphanBlocks.insert(make_pair(hash, pblock2));
1895 1940
         mapOrphanBlocksByPrev.insert(make_pair(pblock2->hashPrevBlock, pblock2));
@@ -1902,7 +1947,7 @@ bool ProcessBlock(CNode* pfrom, CBlock* pblock)
1902 1947
 
1903 1948
     // Store to disk
1904 1949
     if (!pblock->AcceptBlock())
1905  
-        return error("ProcessBlock() : AcceptBlock FAILED");
  1950
+        return error("ProcessBlockBody() : AcceptBlock FAILED");
1906 1951
 
1907 1952
     // Recursively process any orphan blocks that depended on this one
1908 1953
     vector<uint256> vWorkQueue;
@@ -1923,7 +1968,22 @@ bool ProcessBlock(CNode* pfrom, CBlock* pblock)
1923 1968
         mapOrphanBlocksByPrev.erase(hashPrev);
1924 1969
     }
1925 1970
 
1926  
-    printf("ProcessBlock: ACCEPTED\n");
  1971
+    printf("ProcessBlockBody: ACCEPTED\n");
  1972
+    return true;
  1973
+}
  1974
+
  1975
+bool ProcessBlock(CNode* pfrom, CBlock* pblock)
  1976
+{
  1977
+    uint256 hash = pblock->GetHash();
  1978
+    bool fEnablePreview;
  1979
+    if (!ProcessBlockHeader(pfrom, pblock, hash, fEnablePreview))
  1980
+        return false;
  1981
+    if (!ProcessBlockBody(pfrom, pblock, hash))
  1982
+    {
  1983
+        if (fEnablePreview)
  1984
+            pblock->nDoS = 0;
  1985
+        return false;
  1986
+    }
1927 1987
     return true;
1928 1988
 }
1929 1989
 
@@ -2355,6 +2415,7 @@ bool static AlreadyHave(CTxDB& txdb, const CInv& inv)
2355 2415
         }
2356 2416
 
2357 2417
     case MSG_BLOCK:
  2418
+    case MSG_BLOCK_PREVIEW:
2358 2419
         return mapBlockIndex.count(inv.hash) ||
2359 2420
                mapOrphanBlocks.count(inv.hash);
2360 2421
     }
@@ -2590,7 +2651,8 @@ bool static ProcessMessage(CNode* pfrom, string strCommand, CDataStream& vRecv)
2590 2651
         // find last block in inv vector
2591 2652
         unsigned int nLastBlock = (unsigned int)(-1);
2592 2653
         for (unsigned int nInv = 0; nInv < vInv.size(); nInv++) {
2593  
-            if (vInv[vInv.size() - 1 - nInv].type == MSG_BLOCK) {
  2654
+            int& type = vInv[vInv.size() - 1 - nInv].type;
  2655
+            if (type == MSG_BLOCK || type == MSG_BLOCK_PREVIEW) {
2594 2656
                 nLastBlock = vInv.size() - 1 - nInv;
2595 2657
                 break;
2596 2658
             }
@@ -2829,18 +2891,55 @@ bool static ProcessMessage(CNode* pfrom, string strCommand, CDataStream& vRecv)
2829 2891
 
2830 2892
     else if (strCommand == "block")
2831 2893
     {
  2894
+        bool fFullBlock = vRecv.nType & SER_BLOCKHEADERONLY;
  2895
+        if (!fFullBlock)
  2896
+            // FIXME: Do we need to support this?
  2897
+            printf("received block with SER_BLOCKHEADERONLY already set; this is not supported!\n");
  2898
+
2832 2899
         CBlock block;
  2900
+        vRecv.nType |= SER_BLOCKHEADERONLY;
2833 2901
         vRecv >> block;
  2902
+        if (fFullBlock)
  2903
+            vRecv.nType &= ~SER_BLOCKHEADERONLY;
  2904
+        uint256 hashBlock = block.GetHash();
  2905
+        printf("received block header %s\n", hashBlock.ToString().substr(0,20).c_str());
  2906
+        bool fEnablePreview;
  2907
+        // NOTE: ProcessBlockHeader potentially begins this block relaying as a preview.
  2908
+        if (!ProcessBlockHeader(pfrom, &block, hashBlock, fEnablePreview))
  2909
+        {
  2910
+            // We don't want this block. Can we skip the download? TODO
  2911
+            if (block.nDoS) pfrom->Misbehaving(block.nDoS);
  2912
+            {
  2913
+                boost::lock_guard<boost::mutex> lock(block.csBlockDownload->first);
  2914
+                block.fBlockDownloading = false;
  2915
+            }
  2916
+            block.csBlockDownload->second.notify_all();
  2917
+        }
  2918
+        else
  2919
+        {
  2920
+            // Download the block contents
  2921
+            vRecv >> block.vtx;
  2922
+            {
  2923
+                boost::lock_guard<boost::mutex> lock(block.csBlockDownload->first);
  2924
+                block.fBlockDownloading = false;
  2925
+            }
  2926
+            block.csBlockDownload->second.notify_all();
  2927
+            printf("received block %s\n", block.GetHash().ToString().substr(0,20).c_str());
2834 2928
 
2835  
-        printf("received block %s\n", block.GetHash().ToString().substr(0,20).c_str());
2836  
-        // block.print();
  2929
+            CInv inv(MSG_BLOCK, block.GetHash());
  2930
+            pfrom->AddInventoryKnown(inv);
2837 2931
 
2838  
-        CInv inv(MSG_BLOCK, block.GetHash());
2839  
-        pfrom->AddInventoryKnown(inv);
  2932
+            if (ProcessBlockBody(pfrom, &block, hashBlock))
  2933
+                mapAlreadyAskedFor.erase(inv);
2840 2934
 
2841  
-        if (ProcessBlock(pfrom, &block))
2842  
-            mapAlreadyAskedFor.erase(inv);
2843  
-        if (block.nDoS) pfrom->Misbehaving(block.nDoS);
  2935
+            /* Ignore DoS if block previewing is enabled:
  2936
+             * - We might have been relayed the body before or without
  2937
+             *   checking it.
  2938
+             * - Previewing is only enabled when the block difficulty is
  2939
+             *   sufficiently high to make it costly for a DoS.
  2940
+             */
  2941
+            if (block.nDoS && !fEnablePreview) pfrom->Misbehaving(block.nDoS);
  2942
+        }
2844 2943
     }
2845 2944
 
2846 2945
 
18  src/main.h
@@ -12,6 +12,9 @@
12 12
 
13 13
 #include <list>
14 14
 
  15
+#include <boost/thread/condition_variable.hpp>
  16
+#include <boost/thread/mutex.hpp>
  17
+
15 18
 class CWallet;
16 19
 class CBlock;
17 20
 class CBlockIndex;
@@ -837,6 +840,10 @@ class CBlock
837 840
     mutable int nDoS;
838 841
     bool DoS(int nDoSIn, bool fIn) const { nDoS += nDoSIn; return fIn; }
839 842
 
  843
+    // Fast block relay
  844
+    bool fBlockDownloading;
  845
+    std::pair<boost::mutex, boost::condition_variable> *csBlockDownload;
  846
+
840 847
     CBlock()
841 848
     {
842 849
         SetNull();
@@ -854,7 +861,15 @@ class CBlock
854 861
 
855 862
         // ConnectBlock depends on vtx being last so it can calculate offset
856 863
         if (!(nType & (SER_GETHASH|SER_BLOCKHEADERONLY)))
  864
+        {
  865
+            if (fBlockDownloading)
  866
+            {
  867
+                boost::unique_lock<boost::mutex> lock(csBlockDownload->first);
  868
+                while (fBlockDownloading)
  869
+                    csBlockDownload->second.wait(lock);
  870
+            }
857 871
             READWRITE(vtx);
  872
+        }
858 873
         else if (fRead)
859 874
             const_cast<CBlock*>(this)->vtx.clear();
860 875
     )
@@ -870,6 +885,7 @@ class CBlock
870 885
         vtx.clear();
871 886
         vMerkleTree.clear();
872 887
         nDoS = 0;
  888
+        fBlockDownloading = false;
873 889
     }
874 890
 
875 891
     bool IsNull() const
@@ -1021,6 +1037,8 @@ class CBlock
1021 1037
     bool ReadFromDisk(const CBlockIndex* pindex, bool fReadTransactions=true);
1022 1038
     bool SetBestChain(CTxDB& txdb, CBlockIndex* pindexNew);
1023 1039
     bool AddToBlockIndex(unsigned int nFile, unsigned int nBlockPos);
  1040
+    bool CheckBlockHeader(bool fCheckPOW) const;
  1041
+    bool CheckBlockBody(bool fCheckMerkleRoot) const;
1024 1042
     bool CheckBlock(bool fCheckPOW=true, bool fCheckMerkleRoot=true) const;
1025 1043
     bool AcceptBlock();
1026 1044
 
1  src/net.h
@@ -72,6 +72,7 @@ enum
72 72
 {
73 73
     MSG_TX = 1,
74 74
     MSG_BLOCK,
  75
+    MSG_BLOCK_PREVIEW,
75 76
 };
76 77
 
77 78
 class CRequestTracker
1  src/protocol.cpp
@@ -16,6 +16,7 @@ static const char* ppszTypeName[] =
16 16
     "ERROR",
17 17
     "tx",
18 18
     "block",
  19
+    "block",
19 20
 };
20 21
 
21 22
 CMessageHeader::CMessageHeader()
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.