-
Notifications
You must be signed in to change notification settings - Fork 36.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Check specific validation error in miner tests #11220
Check specific validation error in miner tests #11220
Conversation
Most of the rejection reasons are not standardised, and we only care that it fails. It might make the most sense to check that it's a Your example is invalid, because for the |
Concept ACK. I think in general checking for the specific error we expect makes it less likely that we inadvertently introduce a bug in the tests; certainly this kind of thing has cropped up in the functional test suite, where bugs introduced into a test harness cause the test to not actually be testing anything, or to be testing the wrong thing. IMO the downside (in general) to additional specificity is that it adds overhead when making changes to the consensus code rejection reasons, because now all the tests that were hardcoded for a particular reason need updating. This patch seems pretty small though, and I don't think that should be much of a concern here. In fact I think we largely already do similar reject-reason-pattern-matching in the functional tests as well. |
I'll see what I can do about Travis failing on everything but OSX. It's throwing I agree the example isn't great. I did actually run into it while trying an extremely high value for MAX_BLOCK_SIGOPS_COST (250x) in an experiment. In attempt to make the test pass, I increased the transaction generation loops from 1001 to 250001, which obviously(?) took forever to run. I then tried fewer loops to get a sense of performance. To my surprise the test passed for some seemingly random intermediate value. Once I found the actual error message I realized it was an error in the modified test; it just ran out of coins somewhere in the loop. Lowering the coinbase reward in the test was an easier way to reproduce that specific error. |
I pushed a few improvements, apparently to the satisfaction of Travis:
So now each test looks like this: BOOST_CHECK_EXCEPTION(
functionThatThrows(),
std::runtime_error,
CheckRejectInvalid(REJECT_INVALID_BAD_CB_MULTIPLE)
); Unfortunately as a result of this change, when a test throws a different exception, it no longer shows what that exception was. I don't know why. However, this is easy for a developer to debug: just put |
src/test/testutil.h
Outdated
#define REJECT_INVALID_BLOCK_VALIDATION_FAILED "block-validation-failed" | ||
|
||
// BOOST_CHECK_THROW predicates to check the specific validation error | ||
class CheckRejectInvalid { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe move all of this to the cpp?
@MarcoFalke since #11234 just snatched There are a lot of other tests that use |
Please squash your commits according to https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#squashing-commits |
24652c2
to
9dfaced
Compare
Done, as well as rebased to latest master. |
9dfaced
to
ddc20fd
Compare
I pushed again to fix commit message and code comment (it was saying "BOOST_CHECK_THROW" instead of "BOOST_CHECK_EXCEPTION" in two places). |
@Sjors probably just an unrelated failure, Travis does that sometimes. I've restarted that test for you. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree with better testing the exception.
src/test/miner_tests.cpp
Outdated
@@ -26,6 +27,25 @@ | |||
|
|||
BOOST_FIXTURE_TEST_SUITE(miner_tests, TestingSetup) | |||
|
|||
// Validation errors used in tests (prevent typos and facilitate refactoring): | |||
#define REJECT_INVALID_BAD_BLK_SIGOPS "bad-blk-sigops" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO inline these strings.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I used #define
to make it (slightly) easier to find all occurrences in the codebase if these rejection reasons ever need to be renamed. It also helps against typos, although typos would be immediately obvious when you run the test.
Happy to change it to inline if nobody objects.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test will fail if there is a typo. Having inline is good because a rename forces you to update the relevant tests and double check everything related.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using inline now.
src/test/miner_tests.cpp
Outdated
@@ -17,6 +17,7 @@ | |||
#include "uint256.h" | |||
#include "util.h" | |||
#include "utilstrencodings.h" | |||
#include <boost/algorithm/string.hpp> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move down near other boost includes (try to keep it sorted too).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I got rid of this dependency.
src/test/miner_tests.cpp
Outdated
// BOOST_CHECK_EXCEPTION predicates to check the specific validation error | ||
class CheckRejectInvalid { | ||
public: | ||
CheckRejectInvalid(std::string const& arg) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CheckRejectInvalid(const std::string& reason) : m_reason(reason) {}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. If it wasn't obvious, I'm not terribly familiar with C++. :-)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done, and renamed the private variable from reason
to m_reason
src/test/miner_tests.cpp
Outdated
reason = arg; | ||
}; | ||
bool operator() (std::runtime_error const& e) const { | ||
return boost::contains(e.what(), reason); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use std::string::find
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That should indeed help with #8670.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
src/test/miner_tests.cpp
Outdated
CheckRejectInvalid(std::string const& arg) { | ||
reason = arg; | ||
}; | ||
bool operator() (std::runtime_error const& e) const { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit, const std::runtime_error& e
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
src/test/miner_tests.cpp
Outdated
BOOST_CHECK_THROW(AssemblerForTest(chainparams).CreateNewBlock(scriptPubKey), std::runtime_error); | ||
|
||
// This should throw bad-blk-sigops | ||
BOOST_CHECK_EXCEPTION(AssemblerForTest(chainparams).CreateNewBlock(scriptPubKey), std::runtime_error, CheckRejectInvalid(REJECT_INVALID_BAD_BLK_SIGOPS)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO this sounds better:
..., std::runtime_error, HasReason("bad-blk-sigops"));
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I renamed CheckRejectInvalid
to HasReason
; seems more readable indeed.
src/test/miner_tests.cpp
Outdated
return boost::contains(e.what(), reason); | ||
}; | ||
private: | ||
std::string reason; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
µnit: Can be const
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done (Github needs a check-mark emoticon)
bef9996
to
59d0ede
Compare
I believe I addressed all code style issued raised. Rebased to latest master. I also signed the commit this time. |
@JeremyRubin it would be great to get your feedback on this PR, given that you're working on these validation errors. I was able to cherry-pick my commit on top of your #11523 without a problem. |
Concept Ack! I would think one improvement might be if the strings were not to be literals (as Luke points out, they aren't standardized). I think it's ok for now though. |
@JeremyRubin the downside of squashing commits is you can't see history :-) I actually used |
Concept ACK |
utACK 59d0ede514f7754ad4df11ffe11f27e2ea63bf3e |
BOOST_CHECK_THROW merely checks that some std::runtime_error is thrown, but not which one. One example of how this could lead to a test passing when a developer introduces a consensus bug: the test for the sigops limit assumes that CreateNewBlock fails with bad-blk-sigops. However it can also fail with bad-txns-vout-negative, e.g. if a naive developer lowers BLOCKSUBSIDY to 1*COIN in the test. BOOST_CHECK_EXCEPTION allows an additional predicate function. This commit uses this for all exceptions that are checked for in miner_tets.cpp: * bad-blk-sigops * bad-cb-multiple * bad-txns-inputs-missingorspent * block-validation-failed An instance of the CheckRejectInvalid class (for a given validation string) is passed to BOOST_CHECK_EXCEPTION.
59d0ede
to
12781db
Compare
Rebased to account for #11389. |
12781db [Tests] check specific validation error in miner tests (Sjors Provoost) Pull request description: ## Problem `BOOST_CHECK_THROW` merely checks that some `std::runtime_error` is thrown, but not which one. Here's an example of how this can cause a test to pass when a developer introduces a consensus bug. The test for the sigops limit assumes that `CreateNewBlock` fails with `bad-blk-sigops`. However it can also fail with bad-txns-vout-negative, if a naive developer lowers `BLOCKSUBSIDY` to `1*COIN`. ## Solution `BOOST_CHECK_EXCEPTION` allows an additional predicate function. This commit uses this for all exceptions that are checked for in `miner_tets.cpp`: * `bad-blk-sigops` * `bad-cb-multiple` * `bad-txns-inputs-missingorspent` * `block-validation-failed` If the function throws a different error, the test will fail. Although the message produced by Boost is a bit [confusing](http://boost.2283326.n4.nabble.com/Test-BOOST-CHECK-EXCEPTION-error-message-still-vague-tt4683257.html#a4683554), it does show which error was actually thrown. Here's what the above `1*COIN` bug would result in: <img width="1134" alt="schermafbeelding 2017-09-02 om 23 42 29" src="https://user-images.githubusercontent.com/10217/29998976-815cabce-9038-11e7-9c46-f5f6cfb0ca7d.png"> ## Other considerations A more elegant solution in my opinion would be to subclass `std::runtime_error` for each `INVALID_TRANSACTION` type, but this would involve touching consensus code. I put the predicates in `test_bitcoin.h` because I assume they can be reused in other test files. However [serialize_tests.cpp](https://github.com/bitcoin/bitcoin/blob/v0.15.0rc3/src/test/serialize_tests.cpp#L245) also uses `BOOST_CHECK_EXCEPTION` and it defines the predicate in the test file itself. Instead of four `IsRejectInvalidReasonX(std::runtime_error const& e)` functions, I'd prefer something reusable like `bool IsRejectInvalidReason(String reason)(std::runtime_error const& e)`, which would be used like `BOOST_CHECK_EXCEPTION(functionThatThrows(), std::runtime_error, IsRejectInvalidReason("bad-blk-sigops")`. I couldn't figure out how to do that in C++. Tree-SHA512: e364f19b4ac19f910f6e8d6533357f57ccddcbd9d53dcfaf923d424d2b9711446d6f36da193208b35788ca21863eadaa7becd9ad890334d334bccf8c2e63dee1
Summary: 12781db [Tests] check specific validation error in miner tests (Sjors Provoost) Pull request description: ## Problem `BOOST_CHECK_THROW` merely checks that some `std::runtime_error` is thrown, but not which one. Here's an example of how this can cause a test to pass when a developer introduces a consensus bug. The test for the sigops limit assumes that `CreateNewBlock` fails with `bad-blk-sigops`. However it can also fail with bad-txns-vout-negative, if a naive developer lowers `BLOCKSUBSIDY` to `1*COIN`. ## Solution `BOOST_CHECK_EXCEPTION` allows an additional predicate function. This commit uses this for all exceptions that are checked for in `miner_tets.cpp`: * `bad-blk-sigops` * `bad-cb-multiple` * `bad-txns-inputs-missingorspent` * `block-validation-failed` If the function throws a different error, the test will fail. Although the message produced by Boost is a bit [confusing](http://boost.2283326.n4.nabble.com/Test-BOOST-CHECK-EXCEPTION-error-message-still-vague-tt4683257.html#a4683554), it does show which error was actually thrown. Here's what the above `1*COIN` bug would result in: <img width="1134" alt="schermafbeelding 2017-09-02 om 23 42 29" src="https://user-images.githubusercontent.com/10217/29998976-815cabce-9038-11e7-9c46-f5f6cfb0ca7d.png"> ## Other considerations A more elegant solution in my opinion would be to subclass `std::runtime_error` for each `INVALID_TRANSACTION` type, but this would involve touching consensus code. I put the predicates in `test_bitcoin.h` because I assume they can be reused in other test files. However [serialize_tests.cpp](https://github.com/bitcoin/bitcoin/blob/v0.15.0rc3/src/test/serialize_tests.cpp#L245) also uses `BOOST_CHECK_EXCEPTION` and it defines the predicate in the test file itself. Instead of four `IsRejectInvalidReasonX(std::runtime_error const& e)` functions, I'd prefer something reusable like `bool IsRejectInvalidReason(String reason)(std::runtime_error const& e)`, which would be used like `BOOST_CHECK_EXCEPTION(functionThatThrows(), std::runtime_error, IsRejectInvalidReason("bad-blk-sigops")`. I couldn't figure out how to do that in C++. Tree-SHA512: e364f19b4ac19f910f6e8d6533357f57ccddcbd9d53dcfaf923d424d2b9711446d6f36da193208b35788ca21863eadaa7becd9ad890334d334bccf8c2e63dee1 Backport of Core PR11220 bitcoin/bitcoin#11220 Note: the exceptions we check for are slightly different and as follows: `bad-blk-sigops` `bad-tx-coinbase` `bad-txns-inputs-missingorspent` `blk-bad-inputs` Depends on D4081 Test Plan: make check Reviewers: deadalnix, Fabien, jasonbcox, O1 Bitcoin ABC, #bitcoin_abc Reviewed By: jasonbcox, O1 Bitcoin ABC, #bitcoin_abc Differential Revision: https://reviews.bitcoinabc.org/D4042
12781db [Tests] check specific validation error in miner tests (Sjors Provoost) Pull request description: ## Problem `BOOST_CHECK_THROW` merely checks that some `std::runtime_error` is thrown, but not which one. Here's an example of how this can cause a test to pass when a developer introduces a consensus bug. The test for the sigops limit assumes that `CreateNewBlock` fails with `bad-blk-sigops`. However it can also fail with bad-txns-vout-negative, if a naive developer lowers `BLOCKSUBSIDY` to `1*COIN`. ## Solution `BOOST_CHECK_EXCEPTION` allows an additional predicate function. This commit uses this for all exceptions that are checked for in `miner_tets.cpp`: * `bad-blk-sigops` * `bad-cb-multiple` * `bad-txns-inputs-missingorspent` * `block-validation-failed` If the function throws a different error, the test will fail. Although the message produced by Boost is a bit [confusing](http://boost.2283326.n4.nabble.com/Test-BOOST-CHECK-EXCEPTION-error-message-still-vague-tt4683257.html#a4683554), it does show which error was actually thrown. Here's what the above `1*COIN` bug would result in: <img width="1134" alt="schermafbeelding 2017-09-02 om 23 42 29" src="https://user-images.githubusercontent.com/10217/29998976-815cabce-9038-11e7-9c46-f5f6cfb0ca7d.png"> ## Other considerations A more elegant solution in my opinion would be to subclass `std::runtime_error` for each `INVALID_TRANSACTION` type, but this would involve touching consensus code. I put the predicates in `test_bitcoin.h` because I assume they can be reused in other test files. However [serialize_tests.cpp](https://github.com/bitcoin/bitcoin/blob/v0.15.0rc3/src/test/serialize_tests.cpp#L245) also uses `BOOST_CHECK_EXCEPTION` and it defines the predicate in the test file itself. Instead of four `IsRejectInvalidReasonX(std::runtime_error const& e)` functions, I'd prefer something reusable like `bool IsRejectInvalidReason(String reason)(std::runtime_error const& e)`, which would be used like `BOOST_CHECK_EXCEPTION(functionThatThrows(), std::runtime_error, IsRejectInvalidReason("bad-blk-sigops")`. I couldn't figure out how to do that in C++. Tree-SHA512: e364f19b4ac19f910f6e8d6533357f57ccddcbd9d53dcfaf923d424d2b9711446d6f36da193208b35788ca21863eadaa7becd9ad890334d334bccf8c2e63dee1
12781db [Tests] check specific validation error in miner tests (Sjors Provoost) Pull request description: ## Problem `BOOST_CHECK_THROW` merely checks that some `std::runtime_error` is thrown, but not which one. Here's an example of how this can cause a test to pass when a developer introduces a consensus bug. The test for the sigops limit assumes that `CreateNewBlock` fails with `bad-blk-sigops`. However it can also fail with bad-txns-vout-negative, if a naive developer lowers `BLOCKSUBSIDY` to `1*COIN`. ## Solution `BOOST_CHECK_EXCEPTION` allows an additional predicate function. This commit uses this for all exceptions that are checked for in `miner_tets.cpp`: * `bad-blk-sigops` * `bad-cb-multiple` * `bad-txns-inputs-missingorspent` * `block-validation-failed` If the function throws a different error, the test will fail. Although the message produced by Boost is a bit [confusing](http://boost.2283326.n4.nabble.com/Test-BOOST-CHECK-EXCEPTION-error-message-still-vague-tt4683257.html#a4683554), it does show which error was actually thrown. Here's what the above `1*COIN` bug would result in: <img width="1134" alt="schermafbeelding 2017-09-02 om 23 42 29" src="https://user-images.githubusercontent.com/10217/29998976-815cabce-9038-11e7-9c46-f5f6cfb0ca7d.png"> ## Other considerations A more elegant solution in my opinion would be to subclass `std::runtime_error` for each `INVALID_TRANSACTION` type, but this would involve touching consensus code. I put the predicates in `test_bitcoin.h` because I assume they can be reused in other test files. However [serialize_tests.cpp](https://github.com/bitcoin/bitcoin/blob/v0.15.0rc3/src/test/serialize_tests.cpp#L245) also uses `BOOST_CHECK_EXCEPTION` and it defines the predicate in the test file itself. Instead of four `IsRejectInvalidReasonX(std::runtime_error const& e)` functions, I'd prefer something reusable like `bool IsRejectInvalidReason(String reason)(std::runtime_error const& e)`, which would be used like `BOOST_CHECK_EXCEPTION(functionThatThrows(), std::runtime_error, IsRejectInvalidReason("bad-blk-sigops")`. I couldn't figure out how to do that in C++. Tree-SHA512: e364f19b4ac19f910f6e8d6533357f57ccddcbd9d53dcfaf923d424d2b9711446d6f36da193208b35788ca21863eadaa7becd9ad890334d334bccf8c2e63dee1
12781db [Tests] check specific validation error in miner tests (Sjors Provoost) Pull request description: ## Problem `BOOST_CHECK_THROW` merely checks that some `std::runtime_error` is thrown, but not which one. Here's an example of how this can cause a test to pass when a developer introduces a consensus bug. The test for the sigops limit assumes that `CreateNewBlock` fails with `bad-blk-sigops`. However it can also fail with bad-txns-vout-negative, if a naive developer lowers `BLOCKSUBSIDY` to `1*COIN`. ## Solution `BOOST_CHECK_EXCEPTION` allows an additional predicate function. This commit uses this for all exceptions that are checked for in `miner_tets.cpp`: * `bad-blk-sigops` * `bad-cb-multiple` * `bad-txns-inputs-missingorspent` * `block-validation-failed` If the function throws a different error, the test will fail. Although the message produced by Boost is a bit [confusing](http://boost.2283326.n4.nabble.com/Test-BOOST-CHECK-EXCEPTION-error-message-still-vague-tt4683257.html#a4683554), it does show which error was actually thrown. Here's what the above `1*COIN` bug would result in: <img width="1134" alt="schermafbeelding 2017-09-02 om 23 42 29" src="https://user-images.githubusercontent.com/10217/29998976-815cabce-9038-11e7-9c46-f5f6cfb0ca7d.png"> ## Other considerations A more elegant solution in my opinion would be to subclass `std::runtime_error` for each `INVALID_TRANSACTION` type, but this would involve touching consensus code. I put the predicates in `test_bitcoin.h` because I assume they can be reused in other test files. However [serialize_tests.cpp](https://github.com/bitcoin/bitcoin/blob/v0.15.0rc3/src/test/serialize_tests.cpp#L245) also uses `BOOST_CHECK_EXCEPTION` and it defines the predicate in the test file itself. Instead of four `IsRejectInvalidReasonX(std::runtime_error const& e)` functions, I'd prefer something reusable like `bool IsRejectInvalidReason(String reason)(std::runtime_error const& e)`, which would be used like `BOOST_CHECK_EXCEPTION(functionThatThrows(), std::runtime_error, IsRejectInvalidReason("bad-blk-sigops")`. I couldn't figure out how to do that in C++. Tree-SHA512: e364f19b4ac19f910f6e8d6533357f57ccddcbd9d53dcfaf923d424d2b9711446d6f36da193208b35788ca21863eadaa7becd9ad890334d334bccf8c2e63dee1
… tests" This reverts commit 5e294ba.
… tests" This reverts commit d455b44.
12781db [Tests] check specific validation error in miner tests (Sjors Provoost) Pull request description: ## Problem `BOOST_CHECK_THROW` merely checks that some `std::runtime_error` is thrown, but not which one. Here's an example of how this can cause a test to pass when a developer introduces a consensus bug. The test for the sigops limit assumes that `CreateNewBlock` fails with `bad-blk-sigops`. However it can also fail with bad-txns-vout-negative, if a naive developer lowers `BLOCKSUBSIDY` to `1*COIN`. ## Solution `BOOST_CHECK_EXCEPTION` allows an additional predicate function. This commit uses this for all exceptions that are checked for in `miner_tets.cpp`: * `bad-blk-sigops` * `bad-cb-multiple` * `bad-txns-inputs-missingorspent` * `block-validation-failed` If the function throws a different error, the test will fail. Although the message produced by Boost is a bit [confusing](http://boost.2283326.n4.nabble.com/Test-BOOST-CHECK-EXCEPTION-error-message-still-vague-tt4683257.html#a4683554), it does show which error was actually thrown. Here's what the above `1*COIN` bug would result in: <img width="1134" alt="schermafbeelding 2017-09-02 om 23 42 29" src="https://user-images.githubusercontent.com/10217/29998976-815cabce-9038-11e7-9c46-f5f6cfb0ca7d.png"> ## Other considerations A more elegant solution in my opinion would be to subclass `std::runtime_error` for each `INVALID_TRANSACTION` type, but this would involve touching consensus code. I put the predicates in `test_bitcoin.h` because I assume they can be reused in other test files. However [serialize_tests.cpp](https://github.com/bitcoin/bitcoin/blob/v0.15.0rc3/src/test/serialize_tests.cpp#L245) also uses `BOOST_CHECK_EXCEPTION` and it defines the predicate in the test file itself. Instead of four `IsRejectInvalidReasonX(std::runtime_error const& e)` functions, I'd prefer something reusable like `bool IsRejectInvalidReason(String reason)(std::runtime_error const& e)`, which would be used like `BOOST_CHECK_EXCEPTION(functionThatThrows(), std::runtime_error, IsRejectInvalidReason("bad-blk-sigops")`. I couldn't figure out how to do that in C++. Tree-SHA512: e364f19b4ac19f910f6e8d6533357f57ccddcbd9d53dcfaf923d424d2b9711446d6f36da193208b35788ca21863eadaa7becd9ad890334d334bccf8c2e63dee1
… tests" This reverts commit d455b44.
12781db [Tests] check specific validation error in miner tests (Sjors Provoost) Pull request description: ## Problem `BOOST_CHECK_THROW` merely checks that some `std::runtime_error` is thrown, but not which one. Here's an example of how this can cause a test to pass when a developer introduces a consensus bug. The test for the sigops limit assumes that `CreateNewBlock` fails with `bad-blk-sigops`. However it can also fail with bad-txns-vout-negative, if a naive developer lowers `BLOCKSUBSIDY` to `1*COIN`. ## Solution `BOOST_CHECK_EXCEPTION` allows an additional predicate function. This commit uses this for all exceptions that are checked for in `miner_tets.cpp`: * `bad-blk-sigops` * `bad-cb-multiple` * `bad-txns-inputs-missingorspent` * `block-validation-failed` If the function throws a different error, the test will fail. Although the message produced by Boost is a bit [confusing](http://boost.2283326.n4.nabble.com/Test-BOOST-CHECK-EXCEPTION-error-message-still-vague-tt4683257.html#a4683554), it does show which error was actually thrown. Here's what the above `1*COIN` bug would result in: <img width="1134" alt="schermafbeelding 2017-09-02 om 23 42 29" src="https://user-images.githubusercontent.com/10217/29998976-815cabce-9038-11e7-9c46-f5f6cfb0ca7d.png"> ## Other considerations A more elegant solution in my opinion would be to subclass `std::runtime_error` for each `INVALID_TRANSACTION` type, but this would involve touching consensus code. I put the predicates in `test_bitcoin.h` because I assume they can be reused in other test files. However [serialize_tests.cpp](https://github.com/bitcoin/bitcoin/blob/v0.15.0rc3/src/test/serialize_tests.cpp#L245) also uses `BOOST_CHECK_EXCEPTION` and it defines the predicate in the test file itself. Instead of four `IsRejectInvalidReasonX(std::runtime_error const& e)` functions, I'd prefer something reusable like `bool IsRejectInvalidReason(String reason)(std::runtime_error const& e)`, which would be used like `BOOST_CHECK_EXCEPTION(functionThatThrows(), std::runtime_error, IsRejectInvalidReason("bad-blk-sigops")`. I couldn't figure out how to do that in C++. Tree-SHA512: e364f19b4ac19f910f6e8d6533357f57ccddcbd9d53dcfaf923d424d2b9711446d6f36da193208b35788ca21863eadaa7becd9ad890334d334bccf8c2e63dee1
12781db [Tests] check specific validation error in miner tests (Sjors Provoost) Pull request description: ## Problem `BOOST_CHECK_THROW` merely checks that some `std::runtime_error` is thrown, but not which one. Here's an example of how this can cause a test to pass when a developer introduces a consensus bug. The test for the sigops limit assumes that `CreateNewBlock` fails with `bad-blk-sigops`. However it can also fail with bad-txns-vout-negative, if a naive developer lowers `BLOCKSUBSIDY` to `1*COIN`. ## Solution `BOOST_CHECK_EXCEPTION` allows an additional predicate function. This commit uses this for all exceptions that are checked for in `miner_tets.cpp`: * `bad-blk-sigops` * `bad-cb-multiple` * `bad-txns-inputs-missingorspent` * `block-validation-failed` If the function throws a different error, the test will fail. Although the message produced by Boost is a bit [confusing](http://boost.2283326.n4.nabble.com/Test-BOOST-CHECK-EXCEPTION-error-message-still-vague-tt4683257.html#a4683554), it does show which error was actually thrown. Here's what the above `1*COIN` bug would result in: <img width="1134" alt="schermafbeelding 2017-09-02 om 23 42 29" src="https://user-images.githubusercontent.com/10217/29998976-815cabce-9038-11e7-9c46-f5f6cfb0ca7d.png"> ## Other considerations A more elegant solution in my opinion would be to subclass `std::runtime_error` for each `INVALID_TRANSACTION` type, but this would involve touching consensus code. I put the predicates in `test_bitcoin.h` because I assume they can be reused in other test files. However [serialize_tests.cpp](https://github.com/bitcoin/bitcoin/blob/v0.15.0rc3/src/test/serialize_tests.cpp#L245) also uses `BOOST_CHECK_EXCEPTION` and it defines the predicate in the test file itself. Instead of four `IsRejectInvalidReasonX(std::runtime_error const& e)` functions, I'd prefer something reusable like `bool IsRejectInvalidReason(String reason)(std::runtime_error const& e)`, which would be used like `BOOST_CHECK_EXCEPTION(functionThatThrows(), std::runtime_error, IsRejectInvalidReason("bad-blk-sigops")`. I couldn't figure out how to do that in C++. Tree-SHA512: e364f19b4ac19f910f6e8d6533357f57ccddcbd9d53dcfaf923d424d2b9711446d6f36da193208b35788ca21863eadaa7becd9ad890334d334bccf8c2e63dee1
Problem
BOOST_CHECK_THROW
merely checks that somestd::runtime_error
isthrown, but not which one.
Here's an example of how this can cause a test to pass when a developer
introduces a consensus bug. The test for the sigops limit assumes
that
CreateNewBlock
fails withbad-blk-sigops
. However it canalso fail with bad-txns-vout-negative, if a naive developer lowers
BLOCKSUBSIDY
to1*COIN
.Solution
BOOST_CHECK_EXCEPTION
allows an additional predicate function. Thiscommit uses this for all exceptions that are checked for in
miner_tets.cpp
:bad-blk-sigops
bad-cb-multiple
bad-txns-inputs-missingorspent
block-validation-failed
If the function throws a different error, the test will fail. Although the message produced by Boost is a bit confusing, it does show which error was actually thrown. Here's what the above
1*COIN
bug would result in:Other considerations
A more elegant solution in my opinion would be to subclass
std::runtime_error
for eachINVALID_TRANSACTION
type, but this would involve touching consensus code.I put the predicates in
test_bitcoin.h
because I assume they can be reused in other test files. However serialize_tests.cpp also usesBOOST_CHECK_EXCEPTION
and it defines the predicate in the test file itself.Instead of four
IsRejectInvalidReasonX(std::runtime_error const& e)
functions, I'd prefer something reusable likebool IsRejectInvalidReason(String reason)(std::runtime_error const& e)
, which would be used likeBOOST_CHECK_EXCEPTION(functionThatThrows(), std::runtime_error, IsRejectInvalidReason("bad-blk-sigops")
. I couldn't figure out how to do that in C++.