Add missing cs_main locks when accessing chainActive

[practicalswift]

Add missing cs_main locks when accessing chainActive.

The variable chainActive is guarded by the mutex cs_main.

[sdaftuar]

I think we can just add an AssertLockHeld(cs_main) to the top of this function -- this function only has two callers, both of which lock cs_main before calling.

EDIT: actually one of the callers does AssertLockHeld(cs_main).

[morcos]

I think we should just get rid of this assert. We can't guarantee chainActive height will not go down.

[practicalswift]

@sdaftuar @morcos Thanks for reviewing. Suggested changes incorporated! :-)

[promag]

utACK.

[promag]

Remove, all callers have cs_main locked, assert lock is held instead. Ping @ryanofsky as he says the goal is to removed recursive locks.

[practicalswift]

@promag Thanks! Now fixed.

[practicalswift]

@promag I'm getting an assertion failure when running test/functional/test_runner.py with your suggested patch applied. Does it pass for you? Reverting to previous version for now.

[promag]

I'll check.

[promag]

@practicalswift following #11596 (comment), I believe a7eb21a is the fix.

[practicalswift]

@promag What is the reasoning behind the patch (a7eb21a) – what are the underlying locking requirements (exact variables being guarded)? Your patch is likely correct, but I'm trying to understand exactly why :-)

[TheBlueMatt]

nit: care to do the serialization of bitmap/outs outside of cs_main (same below)?

[TheBlueMatt]

Ugh, can we not add more cs_main requirements - if the goal is to resolve the chainActive.Height() issue only then please add the cs_main around just that, its not gonna hurt anything to not hold cs_main for the duration of this function, and we really should be trying to remove cs_main from this function, not go the other way.

[practicalswift]

@TheBlueMatt Thanks for reviewing. Feedback addressed.

[TheBlueMatt]

Please no. Can we instead add the locking in CreateTransaction itself? We need to be pushing locks down, especially when it comes to cs_main, not putting more locks in RPC functions.

[practicalswift]

Reverting the patch suggested by @promag. The locking is now made down in CreateTransaction as suggested by @TheBlueMatt.

[TheBlueMatt]

utACK 2e441c9
Thanks!

[luke-jr]

This isn't sufficient I think. We don't want to allow chainActive to change from when we get the results, to when we identify the chain they're for. So extract this info above in the first lock...

[promag]

Which first lock?

[practicalswift]

@luke-jr Thanks for reviewing. Do you mean like this?

diff --git a/src/rest.cpp b/src/rest.cpp
index 0ed11e6..649059e 100644
--- a/src/rest.cpp
+++ b/src/rest.cpp
@@ -479,6 +479,8 @@ static bool rest_getutxos(HTTPRequest* req, const std::string& strURIPart)
     std::string bitmapStringRepresentation;
     std::vector<bool> hits;
     bitmap.resize((vOutPoints.size() + 7) / 8);
+    int chainActiveHeight;
+    uint256 chainActiveTipBlockHash;
     {
         LOCK2(cs_main, mempool.cs);

@@ -503,6 +505,9 @@ static bool rest_getutxos(HTTPRequest* req, const std::string& strURIPart)
             bitmapStringRepresentation.append(hit ? "1" : "0"); // form a binary string representation (human-readable for json output)
             bitmap[i / 8] |= ((uint8_t)hit) << (i % 8);
         }
+
+        chainActiveHeight = chainActive.Height();
+        chainActiveTipBlockHash = chainActive.Tip()->GetBlockHash();
     }

     switch (rf) {
@@ -510,11 +515,7 @@ static bool rest_getutxos(HTTPRequest* req, const std::string& strURIPart)
         // serialize data
         // use exact same output as mentioned in Bip64
         CDataStream ssGetUTXOResponse(SER_NETWORK, PROTOCOL_VERSION);
-        {
-            LOCK(cs_main);
-            ssGetUTXOResponse << chainActive.Height() << chainActive.Tip()->GetBlockHash();
-        }
-        ssGetUTXOResponse << bitmap << outs;
+        ssGetUTXOResponse << chainActiveHeight << chainActiveTipBlockHash << bitmap << outs;
         std::string ssGetUTXOResponseString = ssGetUTXOResponse.str();

         req->WriteHeader("Content-Type", "application/octet-stream");
@@ -524,11 +525,7 @@ static bool rest_getutxos(HTTPRequest* req, const std::string& strURIPart)

     case RF_HEX: {
         CDataStream ssGetUTXOResponse(SER_NETWORK, PROTOCOL_VERSION);
-        {
-            LOCK(cs_main);
-            ssGetUTXOResponse << chainActive.Height() << chainActive.Tip()->GetBlockHash();
-        }
-        ssGetUTXOResponse << bitmap << outs;
+        ssGetUTXOResponse << chainActiveHeight << chainActiveTipBlockHash << bitmap << outs;
         std::string strHex = HexStr(ssGetUTXOResponse.begin(), ssGetUTXOResponse.end()) + "\n";

         req->WriteHeader("Content-Type", "text/plain");
@@ -541,11 +538,8 @@ static bool rest_getutxos(HTTPRequest* req, const std::string& strURIPart)

         // pack in some essentials
         // use more or less the same output as mentioned in Bip64
-        {
-            LOCK(cs_main);
-            objGetUTXOResponse.push_back(Pair("chainHeight", chainActive.Height()));
-            objGetUTXOResponse.push_back(Pair("chaintipHash", chainActive.Tip()->GetBlockHash().GetHex()));
-        }
+        objGetUTXOResponse.push_back(Pair("chainHeight", chainActiveHeight));
+        objGetUTXOResponse.push_back(Pair("chaintipHash", chainActiveTipBlockHash.GetHex()));
         objGetUTXOResponse.push_back(Pair("bitmap", bitmapStringRepresentation));

         UniValue utxos(UniValue::VARR);

[luke-jr]

Same

[luke-jr]

Same.

[practicalswift]

Added commit 9b3d094 addressing @luke-jr:s feedback.

Please re-review :-)

[TheBlueMatt]

@practicalswift why did you rebase this? It makes reviewing something that was previously reviewed much more difficult. Mind squashing the two commits? There seems to be no reason to have them separate.

[luke-jr]

utACK

[practicalswift]

Now squashed into one commit. Please re-review! :-)

@TheBlueMatt The reason for the rebase was that I wanted to get rebased on top of 76ea17c (IIRC) in order to get the build to pass – I always compile with -Werror=thread-safety-analysis :-)

The reason for the two commits was to allow separate review for the latter commit in the case that I had misunderstood @luke-jr:s suggestion and needed to revert.

[practicalswift]

Updated:

• Added a few more missing cs_main locks.
• Added a commit with the Clang thread safety analysis annotation chainActive GUARDED_BY(cs_main) and the corresponding EXCLUSIVE_LOCKS_REQUIRED(…) annotations that follow from that.

Please review :-)