RPC Python2 Test Runner

[mdr0id]

This new script is intended to allow the RPC stack to run on a given platform (e.g Windows, Linux, MacOS, ARM?). There will be a separate PR associated with this, which will allow the RPC stack to gracefully run on Windows. (Update: #3781 is separate PR )

I have a few cleanup/graceful state maintenance functions to add. I am currently testing these per platform, but it should be done soon.

[mdr0id]

#3781 can be used to test these on Windows.

[daira]

Why is this printing by twos?

[str4d]

It looks like this is trying to adjust for the fact that stats is being appended for each test (so there is excess data in each test_list entry). test[i] will contain the filename; test[i+1] will contain the elapsed time.

This will break (in that it will print incorrect information) if a test fails, because pass_list won't contain an entry for every test. Say there are four tests, A through D, and test C fails. Then pass_list will contain:

[
    ['fileA', 'elapsedA'],
    ['fileA', 'elapsedA', 'fileB', 'elapsedB'],
    ['fileA', 'elapsedA', 'fileB', 'elapsedB', 'fileC', 'elapsedC', 'fileD', 'elapsedD'],
]

The current logic, when given pass_list, will print the information for tests [A, B, C] instead of [A, B, D]. With more failed tests, the offset will increase correspondingly. fail_list will likely only just print passing test information.

test_list should instead contain 2-tuples (thus only containing pass or failure information, not both):

[
    ('fileA', 'elapsedA'),
    ('fileB', 'elapsedB'),
    ('fileD', 'elapsedD'),
]

This could then be printed with:

logging.info("%s : %s" % test)

[mdr0id]

Str4d is correct about the counting by twos, and the issues with failed/interrupted test scenarios. Looking to address this in next round of commits.

[daira]

I don't understand this -- stats is always the same object within this function, so we're appending that object multiple times?

[str4d]

It looks like this is trying to adjust for the fact that stats is being appended for each test (so there is excess data in each test_list entry). test[i] will contain the filename; test[i+1] will contain the elapsed time.

This will break (in that it will print incorrect information) if a test fails, because pass_list won't contain an entry for every test. Say there are four tests, A through D, and test C fails. Then pass_list will contain:

[
    ['fileA', 'elapsedA'],
    ['fileA', 'elapsedA', 'fileB', 'elapsedB'],
    ['fileA', 'elapsedA', 'fileB', 'elapsedB', 'fileC', 'elapsedC', 'fileD', 'elapsedD'],
]

The current logic, when given pass_list, will print the information for tests [A, B, C] instead of [A, B, D]. With more failed tests, the offset will increase correspondingly. fail_list will likely only just print passing test information.

test_list should instead contain 2-tuples (thus only containing pass or failure information, not both):

[
    ('fileA', 'elapsedA'),
    ('fileB', 'elapsedB'),
    ('fileD', 'elapsedD'),
]

This could then be printed with:

logging.info("%s : %s" % test)

[str4d]

Remove this, and instead construct a fresh 2-tuple for each test containing the desired information. Then simplify the printing logic per my comment there.

[str4d]

Remove the -i argument, and instead check if args.args is not empty, to decide whether to run provided tests or all tests.

[str4d]

This leaves off various arguments and flags that are supposed to be provided to the individual RPC tests. See our current test runner, as well as upstream's Python test runner as of 0.12.0.

Additionally, executing with python ignores the environment command at the top of each file. This might be useful for a CI system, but probably isn't what is indended for general usage, so if the intent is the latter then this should probably be turned on with a CLI flag.

[mdr0id]

Still working through some of the upstream nuances, but it appears the defaults are redundant to the env BITCOIND and BITCOINCLI. Perhaps there are other binaries upstream uses inside of what they user for srcdir, but from what I have tinkered with it doesn't seem we use other binaries in the src dir.

[mdr0id]

It also appears that the other major upgrade is the coverage python file. I'd have to go look at what it does in the field. At a glance it seems to just dump pathways of control flow from RPC, with a given set of params.

[mdr0id]

Addressed a couple comments, and will now check out some of the timeout/ctrl-c items mentioned.