New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[tests] Make comp test framework more debuggable #11468
Conversation
Add logging so that each test prints the line number of the yield statement that generated the test.
ACK 7481579 |
utACK 7481579. I like seeing |
I don't think there's need for that. The problem this is solving is not having a way to tell where in the test script execution has reached when there's a failure. I've done that with the smallest change I could. We should stop using the comparison test framework entirely for these static tests (see https://botbot.me/freenode/bitcoin-core-dev/2017-06-15/?msg=87309124&page=3) so I don't intend to make bigger changes than is necessary. |
Yes, that should be a bigger patch. Just read the IRC log 👍 |
Obvious utACK 7481579. Sorry that I missed this. |
7481579 [tests] Make comp test framework more debuggable (John Newbery) Pull request description: We should remove the comparison test framework entirely (see #10603). Until we do that, let's make it a bit more debuggable. Currently, if there's an assert in the framework, it's very difficult to track down where we are in the test generator. Make the logging a bit better to help with debugging. Before this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:05:11.302000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:11.557000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11975 2017-10-09 14:05:11.712000 TestFramework.comptool (INFO): Test 1: PASS 2017-10-09 14:05:11.947000 TestFramework.comptool (INFO): Test 2: PASS 2017-10-09 14:05:12.057000 TestFramework.comptool (INFO): Test 3: PASS 2017-10-09 14:05:12.058000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 306, in run assert test_number != 4 AssertionError 2017-10-09 14:05:12.059000 TestFramework (INFO): Stopping nodes 2017-10-09 14:05:14.203000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:14.204000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testzdnax_yr/test_framework.log ``` With this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:03:54.069000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:54.329000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11783 2017-10-09 14:03:54.383000 TestFramework.comptool (INFO): Running test 1: ./p2p-fullblocktest.py line 184 2017-10-09 14:03:54.496000 TestFramework.comptool (INFO): Running test 2: ./p2p-fullblocktest.py line 193 2017-10-09 14:03:54.758000 TestFramework.comptool (INFO): Running test 3: ./p2p-fullblocktest.py line 205 2017-10-09 14:03:54.867000 TestFramework.comptool (INFO): Running test 4: ./p2p-fullblocktest.py line 208 2017-10-09 14:03:54.867000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 309, in run assert test_number != 4 AssertionError 2017-10-09 14:03:54.868000 TestFramework (INFO): Stopping nodes 2017-10-09 14:03:56.950000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:56.950000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testuey7t3tf/test_framework.log ``` Tree-SHA512: 5525958b0098d661c281bd955c92c72bf81359464376e96d44e6c88f18aea57ba08270ecd564edda4b47d674e3b27d20e5b1060544bf8dd5c6a68103d2bb35b8
7481579 [tests] Make comp test framework more debuggable (John Newbery) Pull request description: We should remove the comparison test framework entirely (see bitcoin#10603). Until we do that, let's make it a bit more debuggable. Currently, if there's an assert in the framework, it's very difficult to track down where we are in the test generator. Make the logging a bit better to help with debugging. Before this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:05:11.302000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:11.557000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11975 2017-10-09 14:05:11.712000 TestFramework.comptool (INFO): Test 1: PASS 2017-10-09 14:05:11.947000 TestFramework.comptool (INFO): Test 2: PASS 2017-10-09 14:05:12.057000 TestFramework.comptool (INFO): Test 3: PASS 2017-10-09 14:05:12.058000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 306, in run assert test_number != 4 AssertionError 2017-10-09 14:05:12.059000 TestFramework (INFO): Stopping nodes 2017-10-09 14:05:14.203000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:14.204000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testzdnax_yr/test_framework.log ``` With this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:03:54.069000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:54.329000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11783 2017-10-09 14:03:54.383000 TestFramework.comptool (INFO): Running test 1: ./p2p-fullblocktest.py line 184 2017-10-09 14:03:54.496000 TestFramework.comptool (INFO): Running test 2: ./p2p-fullblocktest.py line 193 2017-10-09 14:03:54.758000 TestFramework.comptool (INFO): Running test 3: ./p2p-fullblocktest.py line 205 2017-10-09 14:03:54.867000 TestFramework.comptool (INFO): Running test 4: ./p2p-fullblocktest.py line 208 2017-10-09 14:03:54.867000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 309, in run assert test_number != 4 AssertionError 2017-10-09 14:03:54.868000 TestFramework (INFO): Stopping nodes 2017-10-09 14:03:56.950000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:56.950000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testuey7t3tf/test_framework.log ``` Tree-SHA512: 5525958b0098d661c281bd955c92c72bf81359464376e96d44e6c88f18aea57ba08270ecd564edda4b47d674e3b27d20e5b1060544bf8dd5c6a68103d2bb35b8
7481579 [tests] Make comp test framework more debuggable (John Newbery) Pull request description: We should remove the comparison test framework entirely (see bitcoin#10603). Until we do that, let's make it a bit more debuggable. Currently, if there's an assert in the framework, it's very difficult to track down where we are in the test generator. Make the logging a bit better to help with debugging. Before this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:05:11.302000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:11.557000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11975 2017-10-09 14:05:11.712000 TestFramework.comptool (INFO): Test 1: PASS 2017-10-09 14:05:11.947000 TestFramework.comptool (INFO): Test 2: PASS 2017-10-09 14:05:12.057000 TestFramework.comptool (INFO): Test 3: PASS 2017-10-09 14:05:12.058000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 306, in run assert test_number != 4 AssertionError 2017-10-09 14:05:12.059000 TestFramework (INFO): Stopping nodes 2017-10-09 14:05:14.203000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:14.204000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testzdnax_yr/test_framework.log ``` With this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:03:54.069000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:54.329000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11783 2017-10-09 14:03:54.383000 TestFramework.comptool (INFO): Running test 1: ./p2p-fullblocktest.py line 184 2017-10-09 14:03:54.496000 TestFramework.comptool (INFO): Running test 2: ./p2p-fullblocktest.py line 193 2017-10-09 14:03:54.758000 TestFramework.comptool (INFO): Running test 3: ./p2p-fullblocktest.py line 205 2017-10-09 14:03:54.867000 TestFramework.comptool (INFO): Running test 4: ./p2p-fullblocktest.py line 208 2017-10-09 14:03:54.867000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 309, in run assert test_number != 4 AssertionError 2017-10-09 14:03:54.868000 TestFramework (INFO): Stopping nodes 2017-10-09 14:03:56.950000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:56.950000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testuey7t3tf/test_framework.log ``` Tree-SHA512: 5525958b0098d661c281bd955c92c72bf81359464376e96d44e6c88f18aea57ba08270ecd564edda4b47d674e3b27d20e5b1060544bf8dd5c6a68103d2bb35b8
7481579 [tests] Make comp test framework more debuggable (John Newbery) Pull request description: We should remove the comparison test framework entirely (see bitcoin#10603). Until we do that, let's make it a bit more debuggable. Currently, if there's an assert in the framework, it's very difficult to track down where we are in the test generator. Make the logging a bit better to help with debugging. Before this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:05:11.302000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:11.557000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11975 2017-10-09 14:05:11.712000 TestFramework.comptool (INFO): Test 1: PASS 2017-10-09 14:05:11.947000 TestFramework.comptool (INFO): Test 2: PASS 2017-10-09 14:05:12.057000 TestFramework.comptool (INFO): Test 3: PASS 2017-10-09 14:05:12.058000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 306, in run assert test_number != 4 AssertionError 2017-10-09 14:05:12.059000 TestFramework (INFO): Stopping nodes 2017-10-09 14:05:14.203000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:14.204000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testzdnax_yr/test_framework.log ``` With this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:03:54.069000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:54.329000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11783 2017-10-09 14:03:54.383000 TestFramework.comptool (INFO): Running test 1: ./p2p-fullblocktest.py line 184 2017-10-09 14:03:54.496000 TestFramework.comptool (INFO): Running test 2: ./p2p-fullblocktest.py line 193 2017-10-09 14:03:54.758000 TestFramework.comptool (INFO): Running test 3: ./p2p-fullblocktest.py line 205 2017-10-09 14:03:54.867000 TestFramework.comptool (INFO): Running test 4: ./p2p-fullblocktest.py line 208 2017-10-09 14:03:54.867000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 309, in run assert test_number != 4 AssertionError 2017-10-09 14:03:54.868000 TestFramework (INFO): Stopping nodes 2017-10-09 14:03:56.950000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:56.950000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testuey7t3tf/test_framework.log ``` Tree-SHA512: 5525958b0098d661c281bd955c92c72bf81359464376e96d44e6c88f18aea57ba08270ecd564edda4b47d674e3b27d20e5b1060544bf8dd5c6a68103d2bb35b8
7481579 [tests] Make comp test framework more debuggable (John Newbery) Pull request description: We should remove the comparison test framework entirely (see bitcoin#10603). Until we do that, let's make it a bit more debuggable. Currently, if there's an assert in the framework, it's very difficult to track down where we are in the test generator. Make the logging a bit better to help with debugging. Before this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:05:11.302000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:11.557000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11975 2017-10-09 14:05:11.712000 TestFramework.comptool (INFO): Test 1: PASS 2017-10-09 14:05:11.947000 TestFramework.comptool (INFO): Test 2: PASS 2017-10-09 14:05:12.057000 TestFramework.comptool (INFO): Test 3: PASS 2017-10-09 14:05:12.058000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 306, in run assert test_number != 4 AssertionError 2017-10-09 14:05:12.059000 TestFramework (INFO): Stopping nodes 2017-10-09 14:05:14.203000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:14.204000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testzdnax_yr/test_framework.log ``` With this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:03:54.069000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:54.329000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11783 2017-10-09 14:03:54.383000 TestFramework.comptool (INFO): Running test 1: ./p2p-fullblocktest.py line 184 2017-10-09 14:03:54.496000 TestFramework.comptool (INFO): Running test 2: ./p2p-fullblocktest.py line 193 2017-10-09 14:03:54.758000 TestFramework.comptool (INFO): Running test 3: ./p2p-fullblocktest.py line 205 2017-10-09 14:03:54.867000 TestFramework.comptool (INFO): Running test 4: ./p2p-fullblocktest.py line 208 2017-10-09 14:03:54.867000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 309, in run assert test_number != 4 AssertionError 2017-10-09 14:03:54.868000 TestFramework (INFO): Stopping nodes 2017-10-09 14:03:56.950000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:56.950000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testuey7t3tf/test_framework.log ``` Tree-SHA512: 5525958b0098d661c281bd955c92c72bf81359464376e96d44e6c88f18aea57ba08270ecd564edda4b47d674e3b27d20e5b1060544bf8dd5c6a68103d2bb35b8
7481579 [tests] Make comp test framework more debuggable (John Newbery) Pull request description: We should remove the comparison test framework entirely (see bitcoin#10603). Until we do that, let's make it a bit more debuggable. Currently, if there's an assert in the framework, it's very difficult to track down where we are in the test generator. Make the logging a bit better to help with debugging. Before this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:05:11.302000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:11.557000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11975 2017-10-09 14:05:11.712000 TestFramework.comptool (INFO): Test 1: PASS 2017-10-09 14:05:11.947000 TestFramework.comptool (INFO): Test 2: PASS 2017-10-09 14:05:12.057000 TestFramework.comptool (INFO): Test 3: PASS 2017-10-09 14:05:12.058000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 306, in run assert test_number != 4 AssertionError 2017-10-09 14:05:12.059000 TestFramework (INFO): Stopping nodes 2017-10-09 14:05:14.203000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:14.204000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testzdnax_yr/test_framework.log ``` With this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:03:54.069000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:54.329000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11783 2017-10-09 14:03:54.383000 TestFramework.comptool (INFO): Running test 1: ./p2p-fullblocktest.py line 184 2017-10-09 14:03:54.496000 TestFramework.comptool (INFO): Running test 2: ./p2p-fullblocktest.py line 193 2017-10-09 14:03:54.758000 TestFramework.comptool (INFO): Running test 3: ./p2p-fullblocktest.py line 205 2017-10-09 14:03:54.867000 TestFramework.comptool (INFO): Running test 4: ./p2p-fullblocktest.py line 208 2017-10-09 14:03:54.867000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 309, in run assert test_number != 4 AssertionError 2017-10-09 14:03:54.868000 TestFramework (INFO): Stopping nodes 2017-10-09 14:03:56.950000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:56.950000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testuey7t3tf/test_framework.log ``` Tree-SHA512: 5525958b0098d661c281bd955c92c72bf81359464376e96d44e6c88f18aea57ba08270ecd564edda4b47d674e3b27d20e5b1060544bf8dd5c6a68103d2bb35b8
7481579 [tests] Make comp test framework more debuggable (John Newbery) Pull request description: We should remove the comparison test framework entirely (see bitcoin#10603). Until we do that, let's make it a bit more debuggable. Currently, if there's an assert in the framework, it's very difficult to track down where we are in the test generator. Make the logging a bit better to help with debugging. Before this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:05:11.302000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:11.557000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11975 2017-10-09 14:05:11.712000 TestFramework.comptool (INFO): Test 1: PASS 2017-10-09 14:05:11.947000 TestFramework.comptool (INFO): Test 2: PASS 2017-10-09 14:05:12.057000 TestFramework.comptool (INFO): Test 3: PASS 2017-10-09 14:05:12.058000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 306, in run assert test_number != 4 AssertionError 2017-10-09 14:05:12.059000 TestFramework (INFO): Stopping nodes 2017-10-09 14:05:14.203000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:14.204000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testzdnax_yr/test_framework.log ``` With this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:03:54.069000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:54.329000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11783 2017-10-09 14:03:54.383000 TestFramework.comptool (INFO): Running test 1: ./p2p-fullblocktest.py line 184 2017-10-09 14:03:54.496000 TestFramework.comptool (INFO): Running test 2: ./p2p-fullblocktest.py line 193 2017-10-09 14:03:54.758000 TestFramework.comptool (INFO): Running test 3: ./p2p-fullblocktest.py line 205 2017-10-09 14:03:54.867000 TestFramework.comptool (INFO): Running test 4: ./p2p-fullblocktest.py line 208 2017-10-09 14:03:54.867000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 309, in run assert test_number != 4 AssertionError 2017-10-09 14:03:54.868000 TestFramework (INFO): Stopping nodes 2017-10-09 14:03:56.950000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:56.950000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testuey7t3tf/test_framework.log ``` Tree-SHA512: 5525958b0098d661c281bd955c92c72bf81359464376e96d44e6c88f18aea57ba08270ecd564edda4b47d674e3b27d20e5b1060544bf8dd5c6a68103d2bb35b8
7481579 [tests] Make comp test framework more debuggable (John Newbery) Pull request description: We should remove the comparison test framework entirely (see bitcoin#10603). Until we do that, let's make it a bit more debuggable. Currently, if there's an assert in the framework, it's very difficult to track down where we are in the test generator. Make the logging a bit better to help with debugging. Before this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:05:11.302000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:11.557000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11975 2017-10-09 14:05:11.712000 TestFramework.comptool (INFO): Test 1: PASS 2017-10-09 14:05:11.947000 TestFramework.comptool (INFO): Test 2: PASS 2017-10-09 14:05:12.057000 TestFramework.comptool (INFO): Test 3: PASS 2017-10-09 14:05:12.058000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 306, in run assert test_number != 4 AssertionError 2017-10-09 14:05:12.059000 TestFramework (INFO): Stopping nodes 2017-10-09 14:05:14.203000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:14.204000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testzdnax_yr/test_framework.log ``` With this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:03:54.069000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:54.329000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11783 2017-10-09 14:03:54.383000 TestFramework.comptool (INFO): Running test 1: ./p2p-fullblocktest.py line 184 2017-10-09 14:03:54.496000 TestFramework.comptool (INFO): Running test 2: ./p2p-fullblocktest.py line 193 2017-10-09 14:03:54.758000 TestFramework.comptool (INFO): Running test 3: ./p2p-fullblocktest.py line 205 2017-10-09 14:03:54.867000 TestFramework.comptool (INFO): Running test 4: ./p2p-fullblocktest.py line 208 2017-10-09 14:03:54.867000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 309, in run assert test_number != 4 AssertionError 2017-10-09 14:03:54.868000 TestFramework (INFO): Stopping nodes 2017-10-09 14:03:56.950000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:56.950000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testuey7t3tf/test_framework.log ``` Tree-SHA512: 5525958b0098d661c281bd955c92c72bf81359464376e96d44e6c88f18aea57ba08270ecd564edda4b47d674e3b27d20e5b1060544bf8dd5c6a68103d2bb35b8
7481579 [tests] Make comp test framework more debuggable (John Newbery) Pull request description: We should remove the comparison test framework entirely (see bitcoin#10603). Until we do that, let's make it a bit more debuggable. Currently, if there's an assert in the framework, it's very difficult to track down where we are in the test generator. Make the logging a bit better to help with debugging. Before this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:05:11.302000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:11.557000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11975 2017-10-09 14:05:11.712000 TestFramework.comptool (INFO): Test 1: PASS 2017-10-09 14:05:11.947000 TestFramework.comptool (INFO): Test 2: PASS 2017-10-09 14:05:12.057000 TestFramework.comptool (INFO): Test 3: PASS 2017-10-09 14:05:12.058000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 306, in run assert test_number != 4 AssertionError 2017-10-09 14:05:12.059000 TestFramework (INFO): Stopping nodes 2017-10-09 14:05:14.203000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:14.204000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testzdnax_yr/test_framework.log ``` With this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:03:54.069000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:54.329000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11783 2017-10-09 14:03:54.383000 TestFramework.comptool (INFO): Running test 1: ./p2p-fullblocktest.py line 184 2017-10-09 14:03:54.496000 TestFramework.comptool (INFO): Running test 2: ./p2p-fullblocktest.py line 193 2017-10-09 14:03:54.758000 TestFramework.comptool (INFO): Running test 3: ./p2p-fullblocktest.py line 205 2017-10-09 14:03:54.867000 TestFramework.comptool (INFO): Running test 4: ./p2p-fullblocktest.py line 208 2017-10-09 14:03:54.867000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 309, in run assert test_number != 4 AssertionError 2017-10-09 14:03:54.868000 TestFramework (INFO): Stopping nodes 2017-10-09 14:03:56.950000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:56.950000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testuey7t3tf/test_framework.log ``` Tree-SHA512: 5525958b0098d661c281bd955c92c72bf81359464376e96d44e6c88f18aea57ba08270ecd564edda4b47d674e3b27d20e5b1060544bf8dd5c6a68103d2bb35b8
7481579 [tests] Make comp test framework more debuggable (John Newbery) Pull request description: We should remove the comparison test framework entirely (see bitcoin#10603). Until we do that, let's make it a bit more debuggable. Currently, if there's an assert in the framework, it's very difficult to track down where we are in the test generator. Make the logging a bit better to help with debugging. Before this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:05:11.302000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:11.557000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11975 2017-10-09 14:05:11.712000 TestFramework.comptool (INFO): Test 1: PASS 2017-10-09 14:05:11.947000 TestFramework.comptool (INFO): Test 2: PASS 2017-10-09 14:05:12.057000 TestFramework.comptool (INFO): Test 3: PASS 2017-10-09 14:05:12.058000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 306, in run assert test_number != 4 AssertionError 2017-10-09 14:05:12.059000 TestFramework (INFO): Stopping nodes 2017-10-09 14:05:14.203000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testzdnax_yr 2017-10-09 14:05:14.204000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testzdnax_yr/test_framework.log ``` With this PR: ``` → ./p2p-fullblocktest.py 2017-10-09 14:03:54.069000 TestFramework (INFO): Initializing test directory /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:54.329000 TestFramework.mininode (INFO): Connecting to Bitcoin Node: 127.0.0.1:11783 2017-10-09 14:03:54.383000 TestFramework.comptool (INFO): Running test 1: ./p2p-fullblocktest.py line 184 2017-10-09 14:03:54.496000 TestFramework.comptool (INFO): Running test 2: ./p2p-fullblocktest.py line 193 2017-10-09 14:03:54.758000 TestFramework.comptool (INFO): Running test 3: ./p2p-fullblocktest.py line 205 2017-10-09 14:03:54.867000 TestFramework.comptool (INFO): Running test 4: ./p2p-fullblocktest.py line 208 2017-10-09 14:03:54.867000 TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 117, in main self.run_test() File "./p2p-fullblocktest.py", line 72, in run_test self.test.run() File "/home/ubuntu/bitcoin/test/functional/test_framework/comptool.py", line 309, in run assert test_number != 4 AssertionError 2017-10-09 14:03:54.868000 TestFramework (INFO): Stopping nodes 2017-10-09 14:03:56.950000 TestFramework (WARNING): Not cleaning up dir /tmp/user/1000/testuey7t3tf 2017-10-09 14:03:56.950000 TestFramework (ERROR): Test failed. Test logging available at /tmp/user/1000/testuey7t3tf/test_framework.log ``` Tree-SHA512: 5525958b0098d661c281bd955c92c72bf81359464376e96d44e6c88f18aea57ba08270ecd564edda4b47d674e3b27d20e5b1060544bf8dd5c6a68103d2bb35b8
We should remove the comparison test framework entirely (see #10603).
Until we do that, let's make it a bit more debuggable. Currently, if there's an assert in the framework, it's very difficult to track down where we are in the test generator. Make the logging a bit better to help with debugging.
Before this PR:
With this PR: