Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent concurrent savemempool #12842

Merged
merged 1 commit into from Oct 24, 2018
Merged

Conversation

@promag
Copy link
Member

@promag promag commented Mar 30, 2018

Follow up of #12172, this change prevents calling savemempool RPC concurrently.

@promag
Copy link
Member Author

@promag promag commented Mar 30, 2018

In order to test the new check and error add:

--- a/src/rpc/blockchain.cpp
+++ b/src/rpc/blockchain.cpp
@@ -1619,6 +1619,8 @@ UniValue savemempool(const JSONRPCRequest& request)
         throw JSONRPCError(RPC_MISC_ERROR, "Currently dumping the mempool");
     }

+    MilliSleep(10000);
+
     if (!g_is_mempool_loaded) {
         throw JSONRPCError(RPC_MISC_ERROR, "The mempool was not loaded yet");
     }

Then run

bitcoin-cli -regtest savemempool & bitcoin-cli -regtest savemempool

The second should wait for the first to terminate.

@@ -1614,6 +1614,11 @@ UniValue savemempool(const JSONRPCRequest& request)
);
}

TxMempoolDumpReserver reserver(mempool);
if (!reserver.reserve()) {
throw JSONRPCError(RPC_MISC_ERROR, "Currently dumping the mempool");
Copy link
Contributor

@conscott conscott Apr 2, 2018

Nit: maybe instead,

"savemempool already in progress"

Copy link
Member Author

@promag promag Apr 2, 2018

Sounds better.

Copy link
Contributor

@conscott conscott left a comment

Tested ACK 7f722a1f0c82e1cc8dd9f6af9159bf7d7b8f2626

@laanwj
Copy link
Member

@laanwj laanwj commented Apr 3, 2018

After #12863 I'm even more convinced that this might be the wrong approach. Protecting access using a mutex instead of returning these kind of 'busy errors' would avoid having to implement retry-poll-loops client-side (including in the tests).

If you want to stick with this you'd at least need to define a new error code that means 'transient error', similar to RPC_IN_WARMUP. But I think pushing this responsibility to the client is unnecessary.

@promag
Copy link
Member Author

@promag promag commented Apr 3, 2018

@laanwj indeed it's very arguable. I think it's preferable to have the client retrying than reserving resources on the server side. IMO having the client waiting is also not desirable because the client can wait indefinitely and also timeout (but then he can raise the timeout) and we would process the request anyway (?). Getting an error instead of waiting is also more informative and allows the client to explicitly log and retry when he feels like.

@laanwj
Copy link
Member

@laanwj laanwj commented Apr 3, 2018

OTOH

  • Then the API wouldn't be changed. Users of savemempool can rely on the call working, or fatally failing.
  • Hanging indefinitely is a bug no matter what. If it takes too long, something is wrong.
  • Interrupt semantics usually require less resources than polling. In this case: sleeping on a mutex requires virtually no resources. Whereas repetitive RPC requests take up CPU cycles and I/O.

@MarcoFalke
Copy link
Member

@MarcoFalke MarcoFalke commented Apr 3, 2018

I think we had reports that loading the mempool took 1.5 hours or so, just noting without further comment.

@promag
Copy link
Member Author

@promag promag commented Apr 3, 2018

Then the API wouldn't be changed. Users of savemempool can rely on the call working, or fatally failing.

The caller should always have error handling, and this new error would fall in his "unkown error".

Hanging indefinitely is a bug no matter what. If it takes too long, something is wrong.

We have calls that can hang a lot of time, for instance when the wallet is really big (as in lots of keys, lots of transactions).

Interrupt semantics usually require less resources than polling.

Right, but at least a thread and a socket is on hold. And when the mutex is acquired, it might do something that is no longer relevant - in this case dumping the mempool again..

We have this semantic in rescanblockchain, do you think we should change that too?

And regarding #12863, having the client blindingly wait for 1.5 hours should not be an option (?)

@sipa
Copy link
Member

@sipa sipa commented Apr 3, 2018

I'm also more a fan of blocking until the existing save has finished. Given that no modifications in the mempool can happen in the mean time anyway (I think?), we could even just wait until the existing save operation finishes and then return from both RPC calls (without an additional save).

@promag
Copy link
Member Author

@promag promag commented Apr 3, 2018

Given that no modifications in the mempool can happen in the mean time anyway

Actually it can, DumpMempool takes a copy of the mempool contents and then releases the lock to then write to file. So, while writing to disk, the mempool can change.

@promag
Copy link
Member Author

@promag promag commented Jun 7, 2018

What should be done here? I've to address @conscott comment above, but I guess I should wait for more concept ACKs.

@luke-jr
Copy link
Member

@luke-jr luke-jr commented Jun 12, 2018

I agree with @laanwj - it should just hold a lock on the file to avoid problems, but otherwise succeed.

@promag promag force-pushed the 2018-03-concurrent-savemempool branch 3 times, most recently from 0b68854 to 535be5b Jun 14, 2018
@promag
Copy link
Member Author

@promag promag commented Jun 14, 2018

Rebased and updated to block/wait while there is another dump.

@promag promag force-pushed the 2018-03-concurrent-savemempool branch from 535be5b to 35bdaf7 Jun 14, 2018
@promag
Copy link
Member Author

@promag promag commented Jun 15, 2018

@luke do you think the current code is enough?

@MarcoFalke
Copy link
Member

@MarcoFalke MarcoFalke commented Jun 15, 2018

utACK 35bdaf7489cf33d120b3c66f87067c2698ed2e8c

@DrahtBot
Copy link
Contributor

@DrahtBot DrahtBot commented Aug 10, 2018

The last travis run for this pull request was 57 days ago and is thus outdated. To trigger a fresh travis build, this pull request should be closed and re-opened.

@DrahtBot DrahtBot closed this Aug 10, 2018
@DrahtBot DrahtBot reopened this Aug 10, 2018
src/txmempool.h Outdated
@@ -485,6 +485,7 @@ class CTxMemPool

mutable CCriticalSection cs;
indexed_transaction_set mapTx GUARDED_BY(cs);
std::mutex m_dump_mutex;
Copy link
Contributor

@ajtowns ajtowns Sep 18, 2018

(a) Shouldn't this be a global mutex, since it's specific to the "mempool.dat.new" file whose path is based on the (global) GetDataDir()?
(b) Why a std::mutex instead of a CCriticalSection?

Copy link
Member

@MarcoFalke MarcoFalke Sep 18, 2018

Why a CCriticalSection instead of a Mutex?

Copy link
Member Author

@promag promag Sep 18, 2018

a) Not sure why that matters. To reduce exposure the mutex could be a static variable in DumpMempool() function;
b) we don't need a recursive mutex.

Copy link
Contributor

@ajtowns ajtowns Sep 19, 2018

a) Yeah, a static variable in/next to the DumpMempool() function was what I was thinking. I don't think it matters in practice though.
b) More about trying to just use one set of locking primitives; Mutex/LOCK per MarcoFalke's suggestion would make more sense to me.

@@ -4777,6 +4777,8 @@ bool DumpMempool(void)
std::map<uint256, CAmount> mapDeltas;
std::vector<TxMempoolInfo> vinfo;

std::lock_guard<std::mutex> lock(mempool.m_dump_mutex);
Copy link
Member

@MarcoFalke MarcoFalke Sep 19, 2018

Could rebase on master, make the type Mutex and use LOCK(...) here?

@MarcoFalke MarcoFalke added this to the 0.18.0 milestone Sep 19, 2018
@MarcoFalke MarcoFalke removed this from the 0.18.0 milestone Sep 19, 2018
@promag promag force-pushed the 2018-03-concurrent-savemempool branch from 35bdaf7 to 585b47c Oct 20, 2018
@promag
Copy link
Member Author

@promag promag commented Oct 20, 2018

@MarcoFalke @achow101 rebased on master and changed to Mutex/LOCK and static variable in DumpMempool.

@MarcoFalke
Copy link
Member

@MarcoFalke MarcoFalke commented Oct 24, 2018

re-utACK 585b47c

@laanwj
Copy link
Member

@laanwj laanwj commented Oct 24, 2018

I really like this clean and simple solution

utACK 585b47c

@laanwj laanwj merged commit 585b47c into bitcoin:master Oct 24, 2018
2 checks passed
laanwj added a commit that referenced this issue Oct 24, 2018
585b47c rpc: Prevent concurrent savemempool (João Barbosa)

Pull request description:

  Follow up of #12172, this change prevents calling `savemempool` RPC concurrently.

Tree-SHA512: 4759a7107658a9794f5c6ab7e3e3002276abadd64996010be67a2791914d284db6fe0377c071a8d6a42387bfb0178f219b73aeec077ce5c4fe5c634a30b3e081
@laanwj laanwj added this to the 0.17.1 milestone Oct 26, 2018
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jun 27, 2021
585b47c rpc: Prevent concurrent savemempool (João Barbosa)

Pull request description:

  Follow up of bitcoin#12172, this change prevents calling `savemempool` RPC concurrently.

Tree-SHA512: 4759a7107658a9794f5c6ab7e3e3002276abadd64996010be67a2791914d284db6fe0377c071a8d6a42387bfb0178f219b73aeec077ce5c4fe5c634a30b3e081
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jun 28, 2021
585b47c rpc: Prevent concurrent savemempool (João Barbosa)

Pull request description:

  Follow up of bitcoin#12172, this change prevents calling `savemempool` RPC concurrently.

Tree-SHA512: 4759a7107658a9794f5c6ab7e3e3002276abadd64996010be67a2791914d284db6fe0377c071a8d6a42387bfb0178f219b73aeec077ce5c4fe5c634a30b3e081
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jun 29, 2021
585b47c rpc: Prevent concurrent savemempool (João Barbosa)

Pull request description:

  Follow up of bitcoin#12172, this change prevents calling `savemempool` RPC concurrently.

Tree-SHA512: 4759a7107658a9794f5c6ab7e3e3002276abadd64996010be67a2791914d284db6fe0377c071a8d6a42387bfb0178f219b73aeec077ce5c4fe5c634a30b3e081
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jul 1, 2021
585b47c rpc: Prevent concurrent savemempool (João Barbosa)

Pull request description:

  Follow up of bitcoin#12172, this change prevents calling `savemempool` RPC concurrently.

Tree-SHA512: 4759a7107658a9794f5c6ab7e3e3002276abadd64996010be67a2791914d284db6fe0377c071a8d6a42387bfb0178f219b73aeec077ce5c4fe5c634a30b3e081
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jul 1, 2021
585b47c rpc: Prevent concurrent savemempool (João Barbosa)

Pull request description:

  Follow up of bitcoin#12172, this change prevents calling `savemempool` RPC concurrently.

Tree-SHA512: 4759a7107658a9794f5c6ab7e3e3002276abadd64996010be67a2791914d284db6fe0377c071a8d6a42387bfb0178f219b73aeec077ce5c4fe5c634a30b3e081
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jul 1, 2021
585b47c rpc: Prevent concurrent savemempool (João Barbosa)

Pull request description:

  Follow up of bitcoin#12172, this change prevents calling `savemempool` RPC concurrently.

Tree-SHA512: 4759a7107658a9794f5c6ab7e3e3002276abadd64996010be67a2791914d284db6fe0377c071a8d6a42387bfb0178f219b73aeec077ce5c4fe5c634a30b3e081
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jul 3, 2021
585b47c rpc: Prevent concurrent savemempool (João Barbosa)

Pull request description:

  Follow up of bitcoin#12172, this change prevents calling `savemempool` RPC concurrently.

Tree-SHA512: 4759a7107658a9794f5c6ab7e3e3002276abadd64996010be67a2791914d284db6fe0377c071a8d6a42387bfb0178f219b73aeec077ce5c4fe5c634a30b3e081
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

9 participants