New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RPC][mempool]: Add savemempool RPC #11099

Merged
merged 2 commits into from Sep 6, 2017

Conversation

Projects
None yet
9 participants
@greenaddress
Contributor

greenaddress commented Aug 20, 2017

Adds a simple parameterless rpc command to dump the mempool.

Rationale:

Sometimes there can be a crash for whatever reason (bug, power loss, etc) causing the mempool.dat file to not be saved.

This change allows to script/cron the rpc call to have more regular saves to the file as well as cli/ad-hoc.

This should solve issue #11086

@promag

Please add a test.

The function can be changed to know if it failed or not.

Show outdated Hide outdated src/rpc/blockchain.cpp
Show outdated Hide outdated src/rpc/blockchain.cpp
@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Aug 20, 2017

Member

Suggestion, reword Add dumpmempool RPC.

Member

promag commented Aug 20, 2017

Suggestion, reword Add dumpmempool RPC.

@greenaddress greenaddress changed the title from [RPC][mempool]: add rpc command to dump the mempool to disk to [RPC][mempool]: Add dumpmempool RPC Aug 21, 2017

@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Aug 21, 2017

Contributor

@promag thanks, updated as for feedback and squashed

Contributor

greenaddress commented Aug 21, 2017

@promag thanks, updated as for feedback and squashed

@promag

I wonder if it should throw RPC_INTERNAL_ERROR, since DumpMempool handles it's errors.

Please split in two commits:

  • Add return value to DumpMempool
  • Add dumpmempool RPC.
Show outdated Hide outdated src/rpc/blockchain.cpp
Show outdated Hide outdated src/rpc/blockchain.cpp
Show outdated Hide outdated src/rpc/blockchain.cpp
@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Aug 21, 2017

Contributor

@promag thanks, updated as for feedback and split in two commits

re: RPC_INTERNAL_ERROR - didn't immediately found something better, very open to suggestions

Anything else you would add to the test?

Contributor

greenaddress commented Aug 21, 2017

@promag thanks, updated as for feedback and split in two commits

re: RPC_INTERNAL_ERROR - didn't immediately found something better, very open to suggestions

Anything else you would add to the test?

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Aug 21, 2017

Member

I meant maybe not throw, and have the bool in the response.

Member

promag commented Aug 21, 2017

I meant maybe not throw, and have the bool in the response.

@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Aug 21, 2017

Contributor

I think the condition is exceptional and deserve a throw of an exception & consistent with the other rpcs (though I don't know if there's a more appropriate exception for it)

Contributor

greenaddress commented Aug 21, 2017

I think the condition is exceptional and deserve a throw of an exception & consistent with the other rpcs (though I don't know if there's a more appropriate exception for it)

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Aug 21, 2017

Member

I almost agree with that if it wasn't the fact that the function doesn't throw.. but I have no strong opinion on that.

If the throw stays, then maybe RPC_MISC_ERROR. See what causes RPC_INTERNAL_ERROR, sound like really bad excpetions.

Member

promag commented Aug 21, 2017

I almost agree with that if it wasn't the fact that the function doesn't throw.. but I have no strong opinion on that.

If the throw stays, then maybe RPC_MISC_ERROR. See what causes RPC_INTERNAL_ERROR, sound like really bad excpetions.

@jonasschnelli

Concept ACK

Show outdated Hide outdated src/rpc/blockchain.cpp
Show outdated Hide outdated src/rpc/blockchain.cpp
@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Aug 22, 2017

Contributor

thanks @jonasschnelli - good points

added a commit (which i think should be squashed at some point together with "Add dumpmempool RPC")

Contributor

greenaddress commented Aug 22, 2017

thanks @jonasschnelli - good points

added a commit (which i think should be squashed at some point together with "Add dumpmempool RPC")

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Aug 23, 2017

Member

utACK cd0ea48

Maybe add an rational (possible use-cases) to the PR description.

Member

jonasschnelli commented Aug 23, 2017

utACK cd0ea48

Maybe add an rational (possible use-cases) to the PR description.

@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Aug 23, 2017

Contributor

@promag updated the exception, thanks

@jonasschnelli updated the description. Please note your utACK points to a commit from @TheBlueMatt. GitHub also tells me there's some changes required, should I ignore that?

Contributor

greenaddress commented Aug 23, 2017

@promag updated the exception, thanks

@jonasschnelli updated the description. Please note your utACK points to a commit from @TheBlueMatt. GitHub also tells me there's some changes required, should I ignore that?

@MarcoFalke

Concept ACK, but I don't like the tests.

Show outdated Hide outdated test/functional/mempool_dump.py
Show outdated Hide outdated test/functional/mempool_dump.py
Show outdated Hide outdated test/functional/mempool_dump.py
Show outdated Hide outdated test/functional/mempool_dump.py
@MarcoFalke

This comment has been minimized.

Show comment
Hide comment
@MarcoFalke

MarcoFalke Aug 23, 2017

Member

You might even combine the tests with the existing mempool persist test.

Member

MarcoFalke commented Aug 23, 2017

You might even combine the tests with the existing mempool persist test.

@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Aug 23, 2017

Contributor

@MarcoFalke I tried with permissions and didn't work. I couldn't find another way to trigger the error.

Will address the other feedback now

Contributor

greenaddress commented Aug 23, 2017

@MarcoFalke I tried with permissions and didn't work. I couldn't find another way to trigger the error.

Will address the other feedback now

Show outdated Hide outdated src/validation.cpp
Show outdated Hide outdated test/functional/mempool_dump.py
Show outdated Hide outdated test/functional/mempool_dump.py
@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Aug 23, 2017

Member

If the throw stays, then maybe RPC_MISC_ERROR. See what causes RPC_INTERNAL_ERROR, sound like really bad excpetions.

I think either is fine. bitcoind should always be able to write to its data directory, so I would say failure to do so counts as an internal error.

Concept ACK, but I don't like the tests.

I also don't like the test framework messing with bitcoind's data directory. I don't think the failure testcase is necessary. If bitcoind can't write to its datadirectory, there are more pressing issues than the RPC raising the correct error.

Agree with @MarcoFalke that it might be a good idea to combine this with the mempool_persist.py test (if it can be done tidily).

Member

jnewbery commented Aug 23, 2017

If the throw stays, then maybe RPC_MISC_ERROR. See what causes RPC_INTERNAL_ERROR, sound like really bad excpetions.

I think either is fine. bitcoind should always be able to write to its data directory, so I would say failure to do so counts as an internal error.

Concept ACK, but I don't like the tests.

I also don't like the test framework messing with bitcoind's data directory. I don't think the failure testcase is necessary. If bitcoind can't write to its datadirectory, there are more pressing issues than the RPC raising the correct error.

Agree with @MarcoFalke that it might be a good idea to combine this with the mempool_persist.py test (if it can be done tidily).

@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Aug 23, 2017

Contributor

@jnewbery thanks, addressed some feedback.

Re: testing I also don't like messing with the datadir but unless someone has better suggestions the options are keeping it or removing it.

Contributor

greenaddress commented Aug 23, 2017

@jnewbery thanks, addressed some feedback.

Re: testing I also don't like messing with the datadir but unless someone has better suggestions the options are keeping it or removing it.

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Aug 23, 2017

Member

the options are keeping it or removing it.

I vote for removing it

Member

jnewbery commented Aug 23, 2017

the options are keeping it or removing it.

I vote for removing it

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Aug 23, 2017

Member

bitcoind should always be able to write to its data directory

@jnewbery DumpMempool() ignores failures, treats them irrelevant, hence my initial point of removing the throw in the RPC.

Member

promag commented Aug 23, 2017

bitcoind should always be able to write to its data directory

@jnewbery DumpMempool() ignores failures, treats them irrelevant, hence my initial point of removing the throw in the RPC.

@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Aug 24, 2017

Contributor

@MarcoFalke I've merged the tests and added your suggestion "copy the dumped mempool from node0 to node1 and check that it can be loaded. (The nodes are not connected, but share the same blockchain)"

I left in the exception test for now.

Contributor

greenaddress commented Aug 24, 2017

@MarcoFalke I've merged the tests and added your suggestion "copy the dumped mempool from node0 to node1 and check that it can be loaded. (The nodes are not connected, but share the same blockchain)"

I left in the exception test for now.

@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Aug 27, 2017

Contributor

Updated slightly the tests only: removed the ugly rename of the node directory in favor of using permissions to make it fail in the test

@MarcoFalke Permissions initially didn't work because there was a rename involved from a tmp file but it works if I use permissions on the tmp file that gets renamed ('mempool.dat.new') so switched to that

I also removed from the test some redundant asserts re: os.remove doing its job (it already throws if it can't find a file or fails to remove it)

Contributor

greenaddress commented Aug 27, 2017

Updated slightly the tests only: removed the ugly rename of the node directory in favor of using permissions to make it fail in the test

@MarcoFalke Permissions initially didn't work because there was a rename involved from a tmp file but it works if I use permissions on the tmp file that gets renamed ('mempool.dat.new') so switched to that

I also removed from the test some redundant asserts re: os.remove doing its job (it already throws if it can't find a file or fails to remove it)

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Aug 28, 2017

Member

utACK d0bfa6a

@jonasschnelli Have your concerns been addressed?

Member

sipa commented Aug 28, 2017

utACK d0bfa6a

@jonasschnelli Have your concerns been addressed?

@jnewbery

I much, much prefer using file permissions on the mempool.dat file instead of deleting directories.

A few nits in the testcase, but otherwise looks great. Thanks for indulging all of my review comments :)

Show outdated Hide outdated test/functional/mempool_persist.py
Show outdated Hide outdated test/functional/mempool_persist.py
Show outdated Hide outdated test/functional/mempool_persist.py
@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Aug 29, 2017

Contributor

@jnewbery thanks! Keep in mind we are not doing permissions on the mempool.dat file as that doesn't work (the dump gets done on a tmp file which then replace the mempool.dat and ignores permissions so we are acting on the tmp file instead)

Contributor

greenaddress commented Aug 29, 2017

@jnewbery thanks! Keep in mind we are not doing permissions on the mempool.dat file as that doesn't work (the dump gets done on a tmp file which then replace the mempool.dat and ignores permissions so we are acting on the tmp file instead)

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Aug 29, 2017

Member

Keep in mind we are not doing permissions on the mempool.dat file as that doesn't work (the dump gets done on a tmp file which then replace the mempool.dat and ignores permissions so we are acting on the tmp file instead)

Ah. Very crafty! Can you add a comment to the test to explain that (mempool.dat.new is an implementation detail - it could potentially be changed in future, breaking this test, so a comment would be helpful)

Member

jnewbery commented Aug 29, 2017

Keep in mind we are not doing permissions on the mempool.dat file as that doesn't work (the dump gets done on a tmp file which then replace the mempool.dat and ignores permissions so we are acting on the tmp file instead)

Ah. Very crafty! Can you add a comment to the test to explain that (mempool.dat.new is an implementation detail - it could potentially be changed in future, breaking this test, so a comment would be helpful)

@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Aug 30, 2017

Contributor

@jnewbery I added the comment and removed the sleep.
I didn't change the logging debug vs info as that imho should be done in a separate PR for all the log statements in the file (i.e. better to be consistent for now)

Contributor

greenaddress commented Aug 30, 2017

@jnewbery I added the comment and removed the sleep.
I didn't change the logging debug vs info as that imho should be done in a separate PR for all the log statements in the file (i.e. better to be consistent for now)

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Aug 31, 2017

ghost commented Aug 31, 2017

@jnewbery

Looks really good. Just a couple more nits (sorry!)

Show outdated Hide outdated test/functional/mempool_persist.py
Show outdated Hide outdated test/functional/mempool_persist.py
@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Sep 1, 2017

Member

Needed slight rebase changes after #11121 (apologies - I keep making changes to the test_framework API).

I've done that and pushed here: https://github.com/jnewbery/bitcoin/tree/pr11099 . Feel free to reset your branch onto that commit.

Member

jnewbery commented Sep 1, 2017

Needed slight rebase changes after #11121 (apologies - I keep making changes to the test_framework API).

I've done that and pushed here: https://github.com/jnewbery/bitcoin/tree/pr11099 . Feel free to reset your branch onto that commit.

@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Sep 1, 2017

Contributor

@jnewbery thank you! I've updated the PR with your rebase, no need to apologize especially as you did all the work :)

Contributor

greenaddress commented Sep 1, 2017

@jnewbery thank you! I've updated the PR with your rebase, no need to apologize especially as you did all the work :)

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Sep 1, 2017

Member

Tested ACK 67d307f. Looks great. Thanks for sticking with this.

Expect Travis to fail (for this and all other PRs) until #11215 is merged.

Member

jnewbery commented Sep 1, 2017

Tested ACK 67d307f. Looks great. Thanks for sticking with this.

Expect Travis to fail (for this and all other PRs) until #11215 is merged.

@promag

There is only one test for dumpmempool failure, which is the can't write to mempool.dat.new. However there is another return false; inside the catch (which I can't understand why) that should be tested.

Also, RenameOver() doesn't throw, and IMO we should fail if the mempool.dat can't be updated.

Show outdated Hide outdated src/rpc/blockchain.cpp
@MeshCollider

This comment has been minimized.

Show comment
Hide comment
@MeshCollider
Member

MeshCollider commented Sep 1, 2017

utACK 67d307f

@luke-jr

This comment has been minimized.

Show comment
Hide comment
@luke-jr

luke-jr Sep 2, 2017

Member

Suggest renaming it to savemempool or something.

The name dumpmempool to me suggested the same outcome as getrawmempool...

Member

luke-jr commented Sep 2, 2017

Suggest renaming it to savemempool or something.

The name dumpmempool to me suggested the same outcome as getrawmempool...

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Sep 2, 2017

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Sep 2, 2017

Add dumpmempool RPC
Github-Pull: #11099
Rebased-From: 67d307f
@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Sep 2, 2017

Contributor

@luke-jr changed the name, savemempool sounds better to me too.

@promag added brackets.

didn't squash and rename one commit yet but happy to when people are happy with the changes

Contributor

greenaddress commented Sep 2, 2017

@luke-jr changed the name, savemempool sounds better to me too.

@promag added brackets.

didn't squash and rename one commit yet but happy to when people are happy with the changes

@greenaddress greenaddress changed the title from [RPC][mempool]: Add dumpmempool RPC to [RPC][mempool]: Add savemempool RPC Sep 2, 2017

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Sep 3, 2017

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery
Member

jnewbery commented Sep 4, 2017

utACK 8ef87d2

@MarcoFalke

This comment has been minimized.

Show comment
Hide comment
@MarcoFalke

MarcoFalke Sep 5, 2017

Member

@greenaddress Will review after rebase

Member

MarcoFalke commented Sep 5, 2017

@greenaddress Will review after rebase

@promag

Please see my above comment.

Show outdated Hide outdated test/functional/mempool_persist.py
Show outdated Hide outdated src/rpc/blockchain.cpp

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Sep 5, 2017

Add savemempool RPC
Github-Pull: #11099
Rebased-From: 67d307f 8ef87d2

greenaddress added some commits Aug 21, 2017

@greenaddress

This comment has been minimized.

Show comment
Hide comment
@greenaddress

greenaddress Sep 6, 2017

Contributor

rebased & squashed and reworded the second commit to the new savemempool name - a lot of thanks for all feedback

@promag @jnewbery @MarcoFalke I changed the return value to NullUniValue (and accordingly updated the test)

Contributor

greenaddress commented Sep 6, 2017

rebased & squashed and reworded the second commit to the new savemempool name - a lot of thanks for all feedback

@promag @jnewbery @MarcoFalke I changed the return value to NullUniValue (and accordingly updated the test)

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Sep 6, 2017

Member

utACK 1aa97ee.

Member

promag commented Sep 6, 2017

utACK 1aa97ee.

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery
Member

jnewbery commented Sep 6, 2017

utACK 1aa97ee

@MarcoFalke

This comment has been minimized.

Show comment
Hide comment
@MarcoFalke

MarcoFalke Sep 6, 2017

Member

@greenaddress Thanks for sticking with the pull. Quite a pain to go through so much review iterations.

Member

MarcoFalke commented Sep 6, 2017

@greenaddress Thanks for sticking with the pull. Quite a pain to go through so much review iterations.

@MarcoFalke MarcoFalke merged commit 1aa97ee into bitcoin:master Sep 6, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

MarcoFalke added a commit that referenced this pull request Sep 6, 2017

Merge #11099: [RPC][mempool]: Add savemempool RPC
1aa97ee Add savemempool RPC (Lawrence Nahum)
467cbbc Add return value to DumpMempool (Lawrence Nahum)

Pull request description:

  Adds a simple parameterless rpc command to dump the mempool.

  Rationale:

  Sometimes there can be a crash for whatever reason (bug, power loss, etc) causing the mempool.dat file to not be saved.

  This change allows to script/cron the rpc call to have more regular saves to the file as well as cli/ad-hoc.

  This should solve issue #11086

Tree-SHA512: e856ae9777425a4521279c9b58e69285d8e374790bebefd3284cf91931eac0e456f86224f427a087a01bf70440bf6e439fa02c8a34940eb1046ae473e98b6aaa
@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Sep 6, 2017

Member

Quite a pain to go through so much review iterations.

You mean fun? :trollface:

Member

promag commented Sep 6, 2017

Quite a pain to go through so much review iterations.

You mean fun? :trollface:

@greenaddress greenaddress deleted the greenaddress:dump_mempool_rpc branch Jul 17, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment