Properly account for JoinSplit value when deciding if a transaction should be placed in a mined block. #1718
Conversation
…hould be placed in a mined block.
ACK code, but needs tests. (Wow, that simple?) |
on mobile: what are the other issues mentioned above? How does nfeedelta impact tx selection? Tx were included in testnet and regtest mode, but there aren't significantly more tx now that a miner picks from, right? |
I don't understand why miners shouldn't include absolutely all transactions until they hit the block limit, or some other limit that is a well-defined standard rule. In fact I thought we'd explicitly decided to turn off dust checks! |
Does this fix the whole problem? A transaction with only a JoinSplit and zero public input (possibly also zero public output) should still be promptly mined. The assumption that it's reasonable to prioritise transactions by their publicly visible value is fundamentally broken for Zcash; we simply don't have enough information to do that. I thought that's why we decided to disable the "dust" provisions entirely. It's not clear to me whether the problem is that we didn't succeed in fully disabling them, or whether there is another layer of (de)prioritisation that we missed. In any case, the purpose of fees is to prevent denial-of-service attacks and incentivize inclusion of transactions by miners who are following their own, non-default, policy. They're not there because of some ideological commitment to node costs needing to be paid for by this specific mechanism. As long as transaction volumes are low enough not to result in full blocks and the mining rewards are high, there really isn't any problem with the default client including as many transactions as it can. We should fix the immediate problem by doing that, rather than assuming that the transaction inclusion policy arrived at by Bitcoin 0.11.2 is suitable for Zcash. |
Do we want to go for a more aggressive change (removing the priority logic) for a 1.0.1 fix of this sort? |
@daira , are you saying no-fee transactions should work by design? |
ACK and tested empirically. |
@@ -9,6 +9,7 @@ zcash_gtest_SOURCES = \ | |||
gtest/test_checktransaction.cpp \ | |||
gtest/json_test_vectors.cpp \ | |||
gtest/json_test_vectors.h \ | |||
gtest/test_fillnewblock.cpp \ |
str4d
Oct 31, 2016
Contributor
This test sets Params()
, and IIRC there's an ordering thing that needs to be correct. @bitcartel?
This test sets Params()
, and IIRC there's an ordering thing that needs to be correct. @bitcartel?
bitcartel
Oct 31, 2016
Contributor
Yes, test_checktransaction has to run before the tests that set Params. Since you're updating the file worth putting a comment next to test_checktransaction to remind us not to move it from the top.
Yes, test_checktransaction has to run before the tests that set Params. Since you're updating the file worth putting a comment next to test_checktransaction to remind us not to move it from the top.
I will write tests for this on my flight! |
What happened to the commits that @str4d pushed here? I needed those :( |
@ebfull did you force-push the first commit again? I have them locally, I can push them on top again. |
@ebfull pushed to your branch |
Thanks, got em. |
@zkbot r- Sorry for the confusion. I said "ack" on slack to mean "I see a request to do a review, so I'll do a review now". |
How has this been tested? How do I recreate the test? Who has tested it? |
ACK on (~15 min) code review. |
It looks like the refactor to do dependency injection is too burdensome now. Can we create automated tests in the |
Okay, now there's an RPC test for this which fails if the patch is not applied. |
self.nodes[0].generate(1) | ||
for x in range(0,50): | ||
self.nodes[0].sendtoaddress(addrtest, 0.01); | ||
|
||
joinsplit_tx = self.nodes[0].createrawtransaction([], {}) | ||
joinsplit_result = self.nodes[0].zcrawjoinsplit(joinsplit_tx, {receive_result["note"] : zcsecretkey}, {zcaddress: 39.8}, 0, 0.1) | ||
|
bitcartel
Nov 2, 2016
Contributor
How about adding a call to check that the txid of the joinsplit tx is in the mempool, and then after the block is mined, check that the txid is no longer in the mempool and that it has 1 confirmation?
How about adding a call to check that the txid of the joinsplit tx is in the mempool, and then after the block is mined, check that the txid is no longer in the mempool and that it has 1 confirmation?
ebfull
Nov 2, 2016
Author
Contributor
We do this below, sort of. exists
becomes True when the transaction finds its way into a block.
We do this below, sort of. exists
becomes True when the transaction finds its way into a block.
str4d
Nov 2, 2016
Contributor
I'm okay with this being indirect for now.
I'm okay with this being indirect for now.
bitcartel
Nov 2, 2016
Contributor
I know but the zcraw calls are deprecated and at some point will be removed entirely. Not a blocker for this PR though.
I know but the zcraw calls are deprecated and at some point will be removed entirely. Not a blocker for this PR though.
nathan-at-least
Nov 2, 2016
Contributor
Didn't this test already pass (with or without the new nested for-loops)? Doesn't receive_result['exists'] == True`` before and after this change?
I don't understand how the presence of non-JS-only transactions matters, so I don't understand why these loops target the code in question.
Finally, how do I know this test isn't testing a behavior that's unique to regtest mode whereas mainnet will behave differently wrt fee calculations, priority, and transaction selection?
Didn't this test already pass (with or without the new nested for-loops)? Doesn't receive_result['exists'] == True`` before and after this change?
I don't understand how the presence of non-JS-only transactions matters, so I don't understand why these loops target the code in question.
Finally, how do I know this test isn't testing a behavior that's unique to regtest mode whereas mainnet will behave differently wrt fee calculations, priority, and transaction selection?
ebfull
Nov 2, 2016
Author
Contributor
This patch is blocking the release of 1.0.1.
The change that I made allows the test that I wrote to pass. It does not pass without my patch, which is what you wanted.
The reason why it behaves like this is that the bug has a side-effect which is only noticeable when the mempool has other stuff in it. The exact conditions are on line 266 in miner.cpp
. "Free" transactions are skipped and not entered into a block. Our existing tests would have caught the bug that this PR is fixing if we were doing other stuff with the mempool as well. So I've modified the test to stress that as well.
This patch is blocking the release of 1.0.1.
The change that I made allows the test that I wrote to pass. It does not pass without my patch, which is what you wanted.
The reason why it behaves like this is that the bug has a side-effect which is only noticeable when the mempool has other stuff in it. The exact conditions are on line 266 in miner.cpp
. "Free" transactions are skipped and not entered into a block. Our existing tests would have caught the bug that this PR is fixing if we were doing other stuff with the mempool as well. So I've modified the test to stress that as well.
utACK on the tests. |
See new comment above about number of txs in mempool --> https://github.com/zcash/zcash/pull/1718/files/52676958d15987019b4c4b22f4340a33690bc772#r86243821. Does not block. ACK. |
self.nodes[0].generate(1) | ||
for x in range(0,50): | ||
self.nodes[0].sendtoaddress(addrtest, 0.01); | ||
|
bitcartel
Nov 2, 2016
Contributor
I debugged the mempool at this point and, as expected, there were 50 txs due to the last iteration of the outer loop.. The mining of 10 blocks seems unnecessary. You could just have the 'for x' loop to create transactions, whilst dropping the 'for xx' outer loop.
I debugged the mempool at this point and, as expected, there were 50 txs due to the last iteration of the outer loop.. The mining of 10 blocks seems unnecessary. You could just have the 'for x' loop to create transactions, whilst dropping the 'for xx' outer loop.
Until I understand why these new loops exercise the buggy code, and that this test is not regtest-specific I can't yet ack. |
self.nodes[0].generate(1) | ||
for x in range(0,50): | ||
self.nodes[0].sendtoaddress(addrtest, 0.01); | ||
|
||
joinsplit_tx = self.nodes[0].createrawtransaction([], {}) | ||
joinsplit_result = self.nodes[0].zcrawjoinsplit(joinsplit_tx, {receive_result["note"] : zcsecretkey}, {zcaddress: 39.8}, 0, 0.1) | ||
|
nathan-at-least
Nov 2, 2016
Contributor
Didn't this test already pass (with or without the new nested for-loops)? Doesn't receive_result['exists'] == True`` before and after this change?
I don't understand how the presence of non-JS-only transactions matters, so I don't understand why these loops target the code in question.
Finally, how do I know this test isn't testing a behavior that's unique to regtest mode whereas mainnet will behave differently wrt fee calculations, priority, and transaction selection?
Didn't this test already pass (with or without the new nested for-loops)? Doesn't receive_result['exists'] == True`` before and after this change?
I don't understand how the presence of non-JS-only transactions matters, so I don't understand why these loops target the code in question.
Finally, how do I know this test isn't testing a behavior that's unique to regtest mode whereas mainnet will behave differently wrt fee calculations, priority, and transaction selection?
I stated in another comment that the bug this PR is trying to fix is only triggered when the mempool contains other transactions. The reason why our existing RPC tests didn't catch the bug (that pure joinsplits were being mined at all in our tests) is that the mempool had no other objects in it. |
One klunky way to create a negative test would be to use a runtime parameter and pass it into a new qa test (like in wallet_protectcoinbase.py):
|
Ok, I change my block to an ACK (based only on code-review / questions). I'm going to test it locally with w/out the fix to convince myself, but this doesn't block the merge or release. |
Ok, I've locally tested to produce a proper fail (w/out the fix) and a proper success (w/ the fix). ACK source review and testing. @zkbot r+ |
|
Properly account for JoinSplit value when deciding if a transaction should be placed in a mined block. Closes #1705. The transaction selection logic in miner.cpp was not updated to account for JoinSplit value. This caused issues that include, but are not limited to, miners not including pure JoinSplit transactions in their blocks.
I remain deeply skeptical that this is an adequate fix; actually I'm sure that it isn't. If I understand correctly, after this PR it sums the public inputs to JoinSplits as well as the public transparent inputs as one of the inputs to the decision of which transactions to include in a block. So if the public inputs are zero, then the PR will have no effect, and the bug will still be present. That can't be right; there is absolutely no reason for a transaction with zero JoinSplit inputs to be deprioritized. |
|
@daira the problem is that for transactions containing pure JoinSplits and no transparent inputs, the fee is paid using |
Move |
@zkbot retry |
Properly account for JoinSplit value when deciding if a transaction should be placed in a mined block. Closes #1705. The transaction selection logic in miner.cpp was not updated to account for JoinSplit value. This caused issues that include, but are not limited to, miners not including pure JoinSplit transactions in their blocks.
|
Oh, |
Closes #1705.
The transaction selection logic in miner.cpp was not updated to account for JoinSplit value. This caused issues that include, but are not limited to, miners not including pure JoinSplit transactions in their blocks.