-
Notifications
You must be signed in to change notification settings - Fork 36.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[tests] Add basic coverage reporting for RPC tests #6804
Conversation
Is it required to have the coverage tempdir? Couldn't this be done in memory without disk io? |
@MarcoFalke I can't see how that would work given that we call the individual test scripts in separate subprocesses, but agree that would be preferable if possible. |
d75e1be
to
91ca8d1
Compare
Yargh... build failure due to #6778. Going to try rerunning. |
Concept ACK, although there seems to be overlap (at least in purpose) with #6813 |
Merge conflict resolved here. |
Rebased (thanks again @MarcoFalke). |
I personally have no preference to be honest. I think it could be handy, if new users dive into Bitcoin Core and start with adding additional tests, and I also like the cleanup and documentation. Then again, it's unclear, whether users really are going to use it. Mini-ACK from my side. |
@dexX7 yeah, apologies for conflating some of the cleanup & doc with this feature. Would this PR be more useful if we enabled it by default for travis builds? |
That's a very interesting idea! |
That's pretty cool imho: https://travis-ci.org/bitcoin/bitcoin/jobs/89011588#L3943 edit: you beat me, hehe. :) |
@@ -67,7 +67,18 @@ def EncodeDecimal(o): | |||
class AuthServiceProxy(object): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of changing authproxy, I'd prefer if this was an object that wrapped an authproxy, and logs and passes on all the calls. I think it could be done that way
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good suggestion. I've started working on the implementation of this, and I think it's going to simplify things a bit.
The overview of untested RPC commands is confronting. We should definitely have it. |
Nit: Would it be hard to unfiddle into two commits?
Regardless of that: Concept ACK, I think we could make it fail as soon as all commands are covered. |
@MarcoFalke, going forward I'll certainly be more diligent about separating cleanup changes from new functionality -- unfortunately I think in this case it'll be tough to tease out the distinction between cleanup and coverage here, as there isn't a clean way to do it in terms of git chunks, so I'd have to basically rewrite the PR with that separation in mind. This is test code, and so the commit separability doesn't seem to me to be as important as in other parts of the system (though I understand in general this is what we should aim for). That said, if you think it's important I'd be happy to try. |
@@ -69,7 +69,7 @@ class AuthServiceProxy(object): | |||
|
|||
def __init__(self, service_url, service_name=None, timeout=HTTP_TIMEOUT, connection=None): | |||
self.__service_url = service_url | |||
self.__service_name = service_name | |||
self._service_name = service_name |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@laanwj note here that unfortunately I wasn't able to completely avoid changing authproxy.py
: in order to use composition over inheritance in coverage.AuthServiceProxyWrapper
(which I think is preferable), we have to make this attribute un-name-mangled, i.e. accessible outside of the class itself.
Aside from any further feedback, I'm happy with where this is. |
Ping here -- any further feedback? |
from authproxy import AuthServiceProxy, JSONRPCException | ||
from util import * |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: is the required? (Makes diff larger)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In principle I agree with getting rid of cases of import *
, but what I'd propose is to explicitly import the symbols that are used:
from util import initialize_chain, start_nodes, # etc
Using the full name explicitly in every use is a) more typing b) generates a large diff (as @MarcoFalke already says)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed; explicitly importing symbols reduces namespace pollution and slightly reduces chances for odd bugs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can file a cleanup PR afterwards doing this more broadly; would you guys prefer I (i) revert this diff here, (ii) import the symbols explicitly instead of the util
namespace, or (iii) leave as-is for now?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd prefer (ii), but if you want to revert this and do it in another PR it's fine with me too. I don't like (iii) because it implies changing this now and changing this (partially) back again later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 I'll do (ii) sometime tonight.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FWIW I'd argue that the existing method of importing the util
namespace rather than particular symbols results in less pollution, but this is a small thing.
utACK 37f4481 |
ACK after squashing into one commit |
Thanks to @MarcoFalke @dexX7 @laanwj for review.
a216876
to
b5cbd39
Compare
reACK b5cbd39 |
Squashed. |
b5cbd39 Add basic coverage reporting for RPC tests (James O'Beirne)
Backport RPC test harness PRs Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#6548 - bitcoin/bitcoin#6804 - Just the coverage backend, not the flag to enable it. - bitcoin/bitcoin#7744 - bitcoin/bitcoin#9832 - Excludes `wallet-hd.py` change (missing bitcoin/bitcoin#8309). Part of #2074.
Backport RPC test harness PRs Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#6548 - bitcoin/bitcoin#6804 - Just the coverage backend, not the flag to enable it for all RPC tests. - bitcoin/bitcoin#7744 - bitcoin/bitcoin#9832 - Excludes `wallet-hd.py` change (missing bitcoin/bitcoin#8309). Part of #2074.
Backport migration from rpc-tests.sh to rpc-tests.py Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#6567 - bitcoin/bitcoin#6523 - bitcoin/bitcoin#6616 - bitcoin/bitcoin#6788 - Only the commit fixing `rpc-tests.py` - bitcoin/bitcoin#6791 - Only the fix to `qa/rpc-tests/README.md` - bitcoin/bitcoin#6827 - bitcoin/bitcoin#6930 - bitcoin/bitcoin#6804 - bitcoin/bitcoin#7029 - bitcoin/bitcoin#7028 - bitcoin/bitcoin#7027 - bitcoin/bitcoin#7135 - bitcoin/bitcoin#7209 - bitcoin/bitcoin#7635 - bitcoin/bitcoin#7778 - bitcoin/bitcoin#7851 - bitcoin/bitcoin#7814 - Only the changes to the new .py files in this PR. - bitcoin/bitcoin#7971 - bitcoin/bitcoin#7972 - bitcoin/bitcoin#8056 - Only the first commit. - bitcoin/bitcoin#8098 - bitcoin/bitcoin#8104 - bitcoin/bitcoin#8133 - Only the `rpc-tests.py` commit. - bitcoin/bitcoin#8066 - bitcoin/bitcoin#8216 - Only the last two commits. - bitcoin/bitcoin#8254 - bitcoin/bitcoin#8400 - bitcoin/bitcoin#8482 - Excluding the first commit (only affects RPC tests we don't have). - bitcoin/bitcoin#8551 - bitcoin/bitcoin#8607 - Only the pull-tester commit, for conflict removal. - bitcoin/bitcoin#8625 - bitcoin/bitcoin#8713 - bitcoin/bitcoin#8750 - bitcoin/bitcoin#8789 - bitcoin/bitcoin#9098 - bitcoin/bitcoin#9276 - Excluding the second commit (we don't have the changes it requires). - bitcoin/bitcoin#9657 - bitcoin/bitcoin#9807 - bitcoin/bitcoin#9766 - bitcoin/bitcoin#9823
Backport migration from rpc-tests.sh to rpc-tests.py Cherry-picked from the following upstream PRs: - bitcoin/bitcoin#6567 - bitcoin/bitcoin#6523 - bitcoin/bitcoin#6616 - bitcoin/bitcoin#6788 - Only the commit fixing `rpc-tests.py` - bitcoin/bitcoin#6791 - Only the fix to `qa/rpc-tests/README.md` - bitcoin/bitcoin#6827 - bitcoin/bitcoin#6930 - bitcoin/bitcoin#6804 - bitcoin/bitcoin#7029 - bitcoin/bitcoin#7028 - bitcoin/bitcoin#7027 - bitcoin/bitcoin#7135 - bitcoin/bitcoin#7209 - bitcoin/bitcoin#7635 - bitcoin/bitcoin#7778 - bitcoin/bitcoin#7851 - bitcoin/bitcoin#7814 - Only the changes to the new .py files in this PR. - bitcoin/bitcoin#7971 - bitcoin/bitcoin#7972 - bitcoin/bitcoin#8056 - Only the first commit. - bitcoin/bitcoin#8098 - bitcoin/bitcoin#8104 - bitcoin/bitcoin#8133 - Only the `rpc-tests.py` commit. - bitcoin/bitcoin#8066 - bitcoin/bitcoin#8216 - Only the last two commits. - bitcoin/bitcoin#8254 - bitcoin/bitcoin#8400 - bitcoin/bitcoin#8482 - Excluding the first commit (only affects RPC tests we don't have). - bitcoin/bitcoin#8551 - bitcoin/bitcoin#8607 - Only the pull-tester commit, for conflict removal. - bitcoin/bitcoin#8625 - bitcoin/bitcoin#8713 - bitcoin/bitcoin#8750 - bitcoin/bitcoin#8789 - bitcoin/bitcoin#9098 - bitcoin/bitcoin#9276 - Excluding the second commit (we don't have the changes it requires). - bitcoin/bitcoin#9657 - bitcoin/bitcoin#9807 - bitcoin/bitcoin#9766 - bitcoin/bitcoin#9823
After the missed-unobfuscation snafu of #6777, I realized that we don't have comprehensive coverage of the RPC commands, nor do we have a method of determining which RPC calls are or aren't covered. I'd like to start writing more RPC tests, but first I'd like to have an easy way of figuring out which tests to write.
In this PR I've provided a basic method of getting a binary coverage report for each RPC command, which should be useful to guide adding tests until we have full coverage of the RPC interface. After we're exercising every RPC command, we can possibly gate builds on that full-coverage invariant using a mechanism like this, if that's preferable.
The report looks as follows: