Properly account for JoinSplit value when deciding if a transaction should be placed in a mined block. #1718

ebfull · Oct 29, 2016

Closes #1705.

The transaction selection logic in miner.cpp was not updated to account for JoinSplit value. This caused issues that include, but are not limited to, miners not including pure JoinSplit transactions in their blocks.

ACK code, but needs tests. (Wow, that simple?)

daira · Oct 29, 2016

ACK code, but needs tests. (Wow, that simple?)

on mobile: what are the other issues mentioned above? How does nfeedelta impact tx selection? Tx were included in testnet and regtest mode, but there aren't significantly more tx now that a miner picks from, right?

bitcartel · Oct 29, 2016

on mobile: what are the other issues mentioned above? How does nfeedelta impact tx selection? Tx were included in testnet and regtest mode, but there aren't significantly more tx now that a miner picks from, right?

I don't understand why miners shouldn't include absolutely all transactions until they hit the block limit, or some other limit that is a well-defined standard rule. In fact I thought we'd explicitly decided to turn off dust checks!

daira · Oct 30, 2016

I don't understand why miners shouldn't include absolutely all transactions until they hit the block limit, or some other limit that is a well-defined standard rule. In fact I thought we'd explicitly decided to turn off dust checks!

daira · Oct 30, 2016

daira reviewed Oct 30, 2016 • edited

View changes

Does this fix the whole problem? A transaction with only a JoinSplit and zero public input (possibly also zero public output) should still be promptly mined.

The assumption that it's reasonable to prioritise transactions by their publicly visible value is fundamentally broken for Zcash; we simply don't have enough information to do that. I thought that's why we decided to disable the "dust" provisions entirely. It's not clear to me whether the problem is that we didn't succeed in fully disabling them, or whether there is another layer of (de)prioritisation that we missed.

In any case, the purpose of fees is to prevent denial-of-service attacks and incentivize inclusion of transactions by miners who are following their own, non-default, policy. They're not there because of some ideological commitment to node costs needing to be paid for by this specific mechanism. As long as transaction volumes are low enough not to result in full blocks and the mining rewards are high, there really isn't any problem with the default client including as many transactions as it can. We should fix the immediate problem by doing that, rather than assuming that the transaction inclusion policy arrived at by Bitcoin 0.11.2 is suitable for Zcash.

Do we want to go for a more aggressive change (removing the priority logic) for a 1.0.1 fix of this sort?

ebfull · Oct 30, 2016

Do we want to go for a more aggressive change (removing the priority logic) for a 1.0.1 fix of this sort?

@daira , are you saying no-fee transactions should work by design?

buzztiaan · Oct 31, 2016

@daira , are you saying no-fee transactions should work by design?

ACK and tested empirically.

bitcartel · Oct 31, 2016

ACK and tested empirically.

str4d · Oct 31, 2016

str4d reviewed Oct 31, 2016

View changes

str4d · Oct 31, 2016

src/Makefile.gtest.include

@@ -9,6 +9,7 @@ zcash_gtest_SOURCES = \
 	gtest/test_checktransaction.cpp \
 	gtest/json_test_vectors.cpp \
        gtest/json_test_vectors.h \
+	gtest/test_fillnewblock.cpp \


This test sets Params(), and IIRC there's an ordering thing that needs to be correct. @bitcartel?

Yes, test_checktransaction has to run before the tests that set Params. Since you're updating the file worth putting a comment next to test_checktransaction to remind us not to move it from the top.

I will write tests for this on my flight!

ebfull · Nov 1, 2016

I will write tests for this on my flight!

What happened to the commits that @str4d pushed here? I needed those :(

ebfull · Nov 1, 2016

What happened to the commits that @str4d pushed here? I needed those :(

@ebfull did you force-push the first commit again? I have them locally, I can push them on top again.

str4d · Nov 1, 2016

@ebfull did you force-push the first commit again? I have them locally, I can push them on top again.

@ebfull pushed to your branch 😃

str4d · Nov 1, 2016

@ebfull pushed to your branch 😃

Thanks, got em.

🛫

ebfull · Nov 1, 2016

Thanks, got em.

🛫

I worked on this on the plane. Unfortunately what is being mocked out for these tests isn't even enough, after some changes and using mock CCoins, we still have things like ContextualCheckTransaction using globals. That's far too invasive...

My recommendation is to merge as-is.

ebfull · Nov 2, 2016

I worked on this on the plane. Unfortunately what is being mocked out for these tests isn't even enough, after some changes and using mock CCoins, we still have things like ContextualCheckTransaction using globals. That's far too invasive...

My recommendation is to merge as-is.

@ebfull did you intentionally force-push? The commits seem to have disappeared again.

str4d · Nov 2, 2016

@ebfull did you intentionally force-push? The commits seem to have disappeared again.

@str4d Yep, I removed them since we can't use them to test this. I kept a local copy though.

ebfull · Nov 2, 2016

@str4d Yep, I removed them since we can't use them to test this. I kept a local copy though.

ACK for not blocking this on trying to get working tests, but this needs input from @nathan-at-least.

str4d · Nov 2, 2016

ACK for not blocking this on trying to get working tests, but this needs input from @nathan-at-least.

Here was my attempt before I realized there was an endless stream of globals to wade through.

ebfull · Nov 2, 2016

Here was my attempt before I realized there was an endless stream of globals to wade through.

ACK on merging as-is. The tests should not delay this needed fix. @ebfull and myself tested this in person and the solution works.

bitcartel · Nov 2, 2016

ACK on merging as-is. The tests should not delay this needed fix. @ebfull and myself tested this in person and the solution works.

Nathan OK'd this.

@zkbot r+

ebfull · Nov 2, 2016

Nathan OK'd this.

@zkbot r+

📌 Commit 2b2bc69 has been approved by ebfull

zkbot · Nov 2, 2016

📌 Commit 2b2bc69 has been approved by ebfull

@zkbot r-

Sorry for the confusion. I said "ack" on slack to mean "I see a request to do a review, so I'll do a review now".

nathan-at-least · Nov 2, 2016

@zkbot r-

Sorry for the confusion. I said "ack" on slack to mean "I see a request to do a review, so I'll do a review now".

How has this been tested? How do I recreate the test? Who has tested it?

nathan-at-least · Nov 2, 2016

How has this been tested? How do I recreate the test? Who has tested it?

ACK on (~15 min) code review.

nathan-at-least · Nov 2, 2016

ACK on (~15 min) code review.

It looks like the refactor to do dependency injection is too burdensome now. Can we create automated tests in the ./qa/pull-tester/ framework which fail without this patch, but succeed with the patch?

nathan-at-least · Nov 2, 2016

It looks like the refactor to do dependency injection is too burdensome now. Can we create automated tests in the ./qa/pull-tester/ framework which fail without this patch, but succeed with the patch?

Okay, now there's an RPC test for this which fails if the patch is not applied.

ebfull · Nov 2, 2016

Okay, now there's an RPC test for this which fails if the patch is not applied.

bitcartel · Nov 2, 2016

bitcartel reviewed Nov 2, 2016

View changes

bitcartel · Nov 2, 2016

qa/rpc-tests/zcjoinsplit.py

+            self.nodes[0].generate(1)
+            for x in range(0,50):
+                self.nodes[0].sendtoaddress(addrtest, 0.01);
+
        joinsplit_tx = self.nodes[0].createrawtransaction([], {})
        joinsplit_result = self.nodes[0].zcrawjoinsplit(joinsplit_tx, {receive_result["note"] : zcsecretkey}, {zcaddress: 39.8}, 0, 0.1)



How about adding a call to check that the txid of the joinsplit tx is in the mempool, and then after the block is mined, check that the txid is no longer in the mempool and that it has 1 confirmation?

We do this below, sort of. exists becomes True when the transaction finds its way into a block.

I'm okay with this being indirect for now.

I know but the zcraw calls are deprecated and at some point will be removed entirely. Not a blocker for this PR though.

Didn't this test already pass (with or without the new nested for-loops)? Doesn't receive_result['exists'] == True`` before and after this change?

I don't understand how the presence of non-JS-only transactions matters, so I don't understand why these loops target the code in question.

Finally, how do I know this test isn't testing a behavior that's unique to regtest mode whereas mainnet will behave differently wrt fee calculations, priority, and transaction selection?

This patch is blocking the release of 1.0.1.

The change that I made allows the test that I wrote to pass. It does not pass without my patch, which is what you wanted.

The reason why it behaves like this is that the bug has a side-effect which is only noticeable when the mempool has other stuff in it. The exact conditions are on line 266 in miner.cpp. "Free" transactions are skipped and not entered into a block. Our existing tests would have caught the bug that this PR is fixing if we were doing other stuff with the mempool as well. So I've modified the test to stress that as well.

utACK on the tests.

str4d · Nov 2, 2016

utACK on the tests.

See new comment above about number of txs in mempool --> https://github.com/zcash/zcash/pull/1718/files/52676958d15987019b4c4b22f4340a33690bc772#r86243821.

Does not block. ACK.

bitcartel · Nov 2, 2016

See new comment above about number of txs in mempool --> https://github.com/zcash/zcash/pull/1718/files/52676958d15987019b4c4b22f4340a33690bc772#r86243821.

Does not block. ACK.

bitcartel · Nov 2, 2016

bitcartel reviewed Nov 2, 2016

View changes

bitcartel · Nov 2, 2016

qa/rpc-tests/zcjoinsplit.py

+            self.nodes[0].generate(1)
+            for x in range(0,50):
+                self.nodes[0].sendtoaddress(addrtest, 0.01);
+


I debugged the mempool at this point and, as expected, there were 50 txs due to the last iteration of the outer loop.. The mining of 10 blocks seems unnecessary. You could just have the 'for x' loop to create transactions, whilst dropping the 'for xx' outer loop.

nathan-at-least · Nov 2, 2016

nathan-at-least requested changes Nov 2, 2016

View changes

Until I understand why these new loops exercise the buggy code, and that this test is not regtest-specific I can't yet ack.

nathan-at-least · Nov 2, 2016

qa/rpc-tests/zcjoinsplit.py

+            self.nodes[0].generate(1)
+            for x in range(0,50):
+                self.nodes[0].sendtoaddress(addrtest, 0.01);
+
        joinsplit_tx = self.nodes[0].createrawtransaction([], {})
        joinsplit_result = self.nodes[0].zcrawjoinsplit(joinsplit_tx, {receive_result["note"] : zcsecretkey}, {zcaddress: 39.8}, 0, 0.1)



Didn't this test already pass (with or without the new nested for-loops)? Doesn't receive_result['exists'] == True`` before and after this change?

I don't understand how the presence of non-JS-only transactions matters, so I don't understand why these loops target the code in question.

Finally, how do I know this test isn't testing a behavior that's unique to regtest mode whereas mainnet will behave differently wrt fee calculations, priority, and transaction selection?

I stated in another comment that the bug this PR is trying to fix is only triggered when the mempool contains other transactions. The reason why our existing RPC tests didn't catch the bug (that pure joinsplits were being mined at all in our tests) is that the mempool had no other objects in it.

ebfull · Nov 2, 2016

I stated in another comment that the bug this PR is trying to fix is only triggered when the mempool contains other transactions. The reason why our existing RPC tests didn't catch the bug (that pure joinsplits were being mined at all in our tests) is that the mempool had no other objects in it.

One klunky way to create a negative test would be to use a runtime parameter and pass it into a new qa test (like in wallet_protectcoinbase.py):

if (!mapArgs.count("-fudgeparam")) {
  theonelinefix
}

bitcartel · Nov 2, 2016

One klunky way to create a negative test would be to use a runtime parameter and pass it into a new qa test (like in wallet_protectcoinbase.py):

if (!mapArgs.count("-fudgeparam")) {
  theonelinefix
}

Ok, I change my block to an ACK (based only on code-review / questions).

I'm going to test it locally with w/out the fix to convince myself, but this doesn't block the merge or release.

nathan-at-least · Nov 3, 2016

Ok, I change my block to an ACK (based only on code-review / questions).

I'm going to test it locally with w/out the fix to convince myself, but this doesn't block the merge or release.

Ok, I've locally tested to produce a proper fail (w/out the fix) and a proper success (w/ the fix).

ACK source review and testing.

@zkbot r+

nathan-at-least · Nov 3, 2016

Ok, I've locally tested to produce a proper fail (w/out the fix) and a proper success (w/ the fix).

ACK source review and testing.

@zkbot r+

📌 Commit 5267695 has been approved by nathan-at-least

zkbot · Nov 3, 2016

📌 Commit 5267695 has been approved by nathan-at-least

⌛️ Testing commit 5267695 with merge 1430573...

zkbot · Nov 3, 2016

⌛️ Testing commit 5267695 with merge 1430573...

I remain deeply skeptical that this is an adequate fix; actually I'm sure that it isn't. If I understand correctly, after this PR it sums the public inputs to JoinSplits as well as the public transparent inputs as one of the inputs to the decision of which transactions to include in a block. So if the public inputs are zero, then the PR will have no effect, and the bug will still be present. That can't be right; there is absolutely no reason for a transaction with zero JoinSplit inputs to be deprioritized.

daira · Nov 3, 2016

I remain deeply skeptical that this is an adequate fix; actually I'm sure that it isn't. If I understand correctly, after this PR it sums the public inputs to JoinSplits as well as the public transparent inputs as one of the inputs to the decision of which transactions to include in a block. So if the public inputs are zero, then the PR will have no effect, and the bug will still be present. That can't be right; there is absolutely no reason for a transaction with zero JoinSplit inputs to be deprioritized.

💔 Test failed - zcash

zkbot · Nov 3, 2016

💔 Test failed - zcash

@daira the problem is that for transactions containing pure JoinSplits and no transparent inputs, the fee is paid using vpub_new, which is a transparent input to the value pool that was not being accounted for.

str4d · Nov 3, 2016

@daira the problem is that for transactions containing pure JoinSplits and no transparent inputs, the fee is paid using vpub_new, which is a transparent input to the value pool that was not being accounted for.

Move gtest/test_checkblock.cpp to the end of the list of tests in Makefile.gtest.include. Fix the whitespace/tab issue on the line gtest/json_test_vectors.h. Locally build and test zcash-gtest. It should no longer fail. (Same issue with #1735 for miner metrics, so perhaps make a PR to fix the zcash-gtest issue and then rebase both on master)

bitcartel · Nov 3, 2016

Move gtest/test_checkblock.cpp to the end of the list of tests in Makefile.gtest.include. Fix the whitespace/tab issue on the line gtest/json_test_vectors.h. Locally build and test zcash-gtest. It should no longer fail. (Same issue with #1735 for miner metrics, so perhaps make a PR to fix the zcash-gtest issue and then rebase both on master)

@zkbot retry

ebfull · Nov 3, 2016

@zkbot retry

⌛️ Testing commit 5267695 with merge 9eb852e...

zkbot · Nov 3, 2016

⌛️ Testing commit 5267695 with merge 9eb852e...

☀️ Test successful - zcash

zkbot · Nov 3, 2016

☀️ Test successful - zcash

Oh, GetJoinSplitValueIn sums the outputs of JoinSplits. That function name makes no sense to me (it should be something like GetTotalInputToTransparentPoolFromJoinSplits), but I see how the fix works now.

daira · Nov 3, 2016

Oh, GetJoinSplitValueIn sums the outputs of JoinSplits. That function name makes no sense to me (it should be something like GetTotalInputToTransparentPoolFromJoinSplits), but I see how the fix works now.

Properly account for joinsplit value when deciding if a transaction s…
…hould be placed in a mined block.

2b2bc69

feeleep75 referenced this pull request Oct 29, 2016
Closed
unconfirmed transactions #1705

str4d modified the milestone: 1.0.1 Oct 31, 2016

nathan-at-least modified the milestones: 1.0.1, 1.0.2 Oct 31, 2016

nathan-at-least assigned str4d Oct 31, 2016

str4d assigned ebfull and unassigned str4d Nov 1, 2016

Test that a pure joinsplit will mine if other transactions are in the…
… mempool.

5267695

nathan-at-least referenced this pull request Nov 3, 2016
Open
Refactor `CreateNewBlock` to facilitate mock-based unit-testing. #1761

str4d referenced this pull request Nov 3, 2016
Merged
Metrics screen improvements #1735

zkbot merged commit 5267695 into zcash:master Nov 3, 2016
1 check passed

1 check passed

homu Test successful
Details

zcash/zcash

Join GitHub today

Properly account for JoinSplit value when deciding if a transaction should be placed in a mined block. #1718

Conversation

ebfull commented Oct 29, 2016

This comment has been minimized.

daira commented Oct 29, 2016

This comment has been minimized.

bitcartel commented Oct 29, 2016

feeleep75 referenced this pull request Oct 29, 2016

This comment has been minimized.

daira commented Oct 30, 2016 • edited Edited 1 time daira edited Oct 30, 2016 (most recent)

daira reviewed Oct 30, 2016 • edited View changes

This comment has been minimized.

ebfull commented Oct 30, 2016

str4d modified the milestone: 1.0.1 Oct 31, 2016

This comment has been minimized.

buzztiaan commented Oct 31, 2016

This comment has been minimized.

bitcartel commented Oct 31, 2016 • edited Edited 1 time bitcartel edited Oct 31, 2016 (most recent)

nathan-at-least modified the milestones: 1.0.1, 1.0.2 Oct 31, 2016

nathan-at-least assigned str4d Oct 31, 2016

str4d reviewed Oct 31, 2016 View changes

This comment has been minimized.

This comment has been minimized.

str4d assigned ebfull and unassigned str4d Nov 1, 2016

This comment has been minimized.

ebfull commented Nov 1, 2016

This comment has been minimized.

ebfull commented Nov 1, 2016

This comment has been minimized.

str4d commented Nov 1, 2016

This comment has been minimized.

str4d commented Nov 1, 2016

This comment has been minimized.

ebfull commented Nov 1, 2016

This comment has been minimized.

ebfull commented Nov 2, 2016

This comment has been minimized.

str4d commented Nov 2, 2016

This comment has been minimized.

ebfull commented Nov 2, 2016

This comment has been minimized.

str4d commented Nov 2, 2016

This comment has been minimized.

ebfull commented Nov 2, 2016

This comment has been minimized.

bitcartel commented Nov 2, 2016

This comment has been minimized.

ebfull commented Nov 2, 2016

This comment has been minimized.

zkbot commented Nov 2, 2016

This comment has been minimized.

nathan-at-least commented Nov 2, 2016

This comment has been minimized.

nathan-at-least commented Nov 2, 2016

This comment has been minimized.

nathan-at-least commented Nov 2, 2016

This comment has been minimized.

nathan-at-least commented Nov 2, 2016

This comment has been minimized.

ebfull commented Nov 2, 2016

bitcartel reviewed Nov 2, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

str4d commented Nov 2, 2016

This comment has been minimized.

bitcartel commented Nov 2, 2016 • edited Edited 1 time bitcartel edited Nov 2, 2016 (most recent)

bitcartel reviewed Nov 2, 2016 View changes

This comment has been minimized.

nathan-at-least requested changes Nov 2, 2016 View changes

This comment has been minimized.

This comment has been minimized.

ebfull commented Nov 2, 2016

This comment has been minimized.

daira commented Oct 30, 2016 •

edited

Edited 1 time

daira edited Oct 30, 2016 (most recent)

daira reviewed Oct 30, 2016 • edited

View changes

bitcartel commented Oct 31, 2016 •

edited

Edited 1 time

bitcartel edited Oct 31, 2016 (most recent)

str4d reviewed Oct 31, 2016

View changes

bitcartel reviewed Nov 2, 2016

View changes

bitcartel commented Nov 2, 2016 •

edited

Edited 1 time

bitcartel edited Nov 2, 2016 (most recent)

bitcartel reviewed Nov 2, 2016

View changes

nathan-at-least requested changes Nov 2, 2016

View changes