Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add code coverage to tests #8

Closed
sappelhoff opened this issue Sep 5, 2020 · 4 comments
Closed

Add code coverage to tests #8

sappelhoff opened this issue Sep 5, 2020 · 4 comments
Labels
high priority Address before final evaluation. JOSS review https://github.com/openjournals/joss-reviews/issues/2621

Comments

@sappelhoff
Copy link
Contributor

related to JOSS review (openjournals/joss-reviews#2621)

It would be great to add a tool that measures what proportion of your code is actually covered by the tests.

It's helpful to:

  • give users a better impression of how far they can "trust" the tests
  • give the developers a better idea of what aspects of the code are tested, and what aspects are not tested

This would be quite easy to implement, e.g., using pytest extensions and codecov as in this repository: https://github.com/sappelhoff/pyprep

Finally, you could add a coverage badge next to your tests:

image

@TomDonoghue
Copy link

TomDonoghue commented Sep 29, 2020

Somewhat related: it might be worth adding a note somewhere on the requirements for running the tests locally. As well as the standard of needing pytest, when trying to run the tests I got an error that fixture 'qtbot' not found, which I tracked to being because pytest-qt is also required to run the tests. This isn't clearly noted anywhere.

Somewhere, I would add a note of test-requirements, and maybe some instructions on running tests locally (this is perhaps most relevant in a contributor guide, as you would likely want contributors to run and check tests.

As a note, I was able to run tests locally. I'm not sure of the coverage (adding that is a good idea), though the GUI tests looked (in terms of how many things popped up) fairly comprehensive.

@JanCBrammer JanCBrammer added JOSS review https://github.com/openjournals/joss-reviews/issues/2621 high priority Address before final evaluation. labels Sep 29, 2020
@JanCBrammer
Copy link
Owner

it might be worth adding a note somewhere on the requirements for running the tests locally

@TomDonoghue, I've included testing instructions in the contributor guide (see 6685c6e).

@JanCBrammer
Copy link
Owner

It would be great to add a tool that measures what proportion of your code is actually covered by the tests.

@sappelhoff, I now collect and report code coverage. However, the coverage is deceptively low (59%) since pytest-cov does not seem to pick up on code that is run within threads (see here). Looking into it.

@JanCBrammer
Copy link
Owner

Closing and transfering to #21, since original issue has been addressed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
high priority Address before final evaluation. JOSS review https://github.com/openjournals/joss-reviews/issues/2621
Projects
None yet
Development

No branches or pull requests

3 participants