Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor --coverage in the test runner #2667

Merged
merged 10 commits into from Jun 22, 2018

Conversation

Projects
None yet
7 participants
@Cadair
Copy link
Member

commented Jun 20, 2018

This moves us away from using astropy to calculate the code coverage and to using pytest-cov. This means that --parallel and --coverage from the test runner work, as well as enabling --cov-report term to work from setup.py

@sunpy-bot

This comment has been minimized.

Copy link

commented Jun 20, 2018

Thanks for the pull request @Cadair! Everything looks great!

@Cadair Cadair force-pushed the Cadair:travis_race branch from eef31cf to 272f019 Jun 20, 2018

@Cadair Cadair closed this Jun 20, 2018

@Cadair Cadair deleted the Cadair:travis_race branch Jun 20, 2018

@Cadair Cadair reopened this Jun 20, 2018

@Cadair Cadair changed the title Parallelise the online build: TRAVIS RACE Refactor --coverage in the test runner Jun 20, 2018

.travis.yml Outdated

- python: 3.6
stage: Comprehensive tests
env: SETUP_CMD="test --online --coverage --parallel 12"

This comment has been minimized.

Copy link
@astrofrog

astrofrog Jun 20, 2018

Contributor

What if there's a bug for odd numbers? ;)

@pep8speaks

This comment has been minimized.

Copy link

commented Jun 21, 2018

Hello @Cadair! Thanks for updating the PR.

Line 74:101: E501 line too long (101 > 100 characters)

Line 69:51: E231 missing whitespace after ','
Line 75:101: E501 line too long (110 > 100 characters)
Line 76:101: E501 line too long (111 > 100 characters)

Comment last updated on June 22, 2018 at 11:22 Hours UTC

Cadair added some commits Jun 20, 2018

Change to using pytest-cov inside the test runner
This enables us to have either term or html reports (html is still the
default) and allows coverage to work with parallel
Modify the coverage path patching based on astropy
This changes the path in the .coverage file to be based on the source directory
rather than the test directory. This makes it possible for tools like codecov or
coveralls to read the coverage report relative to the git clone.

@Cadair Cadair force-pushed the Cadair:travis_race branch from df98589 to b81c229 Jun 21, 2018

@Cadair Cadair modified the milestones: 0.9.1, 0.9.2 Jun 21, 2018

Cadair added some commits Jun 21, 2018

@Cadair

This comment has been minimized.

Copy link
Member Author

commented Jun 21, 2018

Well the VSO is all on 🔥 again, so it's messing with codecov, but all seems well.

@StanczakDominik
Copy link

left a comment

Looks good to me! Really looking forward to seeing it upstream :)

cmd_pre += pre
cmd_post += post
# Copy the raw .coverage file back so it can be used for CI reports
cwd = os.path.join(os.path.abspath("."))

This comment has been minimized.

Copy link
@ritiek

ritiek Jun 21, 2018

Wouldn't just os.path.abspath(".") would suffice here or maybe os.getcwd()?


# Special case html as the default report
if self.cov_report and (isinstance(self.cov_report, bool) or "html" in self.cov_report):
html_cov = os.path.join(os.path.abspath("."), "htmlcov")

This comment has been minimized.

Copy link
@ritiek

ritiek Jun 21, 2018

This could be os.path.join(cwd, "htmlcov") where cwd comes from above.

@astrofrog
Copy link
Contributor

left a comment

Looks good to me! Just a small comment in terms of documentation.



def _patch_coverage(testdir, sourcedir):
import coverage

This comment has been minimized.

Copy link
@astrofrog

astrofrog Jun 22, 2018

Contributor

It would be good to have some kind of docstring or comment that explains in what way the coverage file will be patched.

improve coverage for tests/
This also includes a nasty little hack for measuring coverage of the test runner

@Cadair Cadair force-pushed the Cadair:travis_race branch from 6dcf5b4 to c77f264 Jun 22, 2018

Cadair added some commits Jun 22, 2018

@nabobalis nabobalis modified the milestones: 0.9.2, 0.9.1 Jun 22, 2018

sunpy/tests/runner.py is correctly measured we force the interpreter to
reload it here while coverage is watching.
"""
imp.reload(sunpy.tests.runner)

This comment has been minimized.

Copy link
@Cadair

Cadair Jun 22, 2018

Author Member

@nabobalis this isn't python 2 compat. I think in 2.7 this will just need to be the reload() builtin

@Cadair Cadair merged commit 58e5b32 into sunpy:master Jun 22, 2018

9 checks passed

Giles Click details to preview the documentation build
Details
ci/circleci: egg-info-36 Your tests passed on CircleCI!
Details
ci/circleci: figure-tests-36 Your tests passed on CircleCI!
Details
ci/circleci: html-docs Your tests passed on CircleCI!
Details
codecov/patch 100% of diff hit (target 71.79%)
Details
codecov/project 84.14% (+12.34%) compared to 4caf4a6
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
sunpy-bot All checks passed
@@ -30,7 +30,7 @@ env:
- MAIN_CMD='python setup.py'
- SETUP_CMD='test --coverage'
- CONDA_CHANNELS='sunpy'
- CONDA_DEPENDENCIES='openjpeg Cython jinja2 scipy matplotlib mock requests beautifulsoup4 sqlalchemy scikit-image pytest-mock lxml pyyaml pandas nomkl pytest-astropy suds-jurko glymur'
- CONDA_DEPENDENCIES='openjpeg Cython jinja2 scipy matplotlib mock requests beautifulsoup4 sqlalchemy scikit-image pytest-mock lxml pyyaml pandas nomkl pytest-astropy suds-jurko glymur pytest-xdist'

This comment has been minimized.

Copy link
@bsipocz

bsipocz Jul 5, 2018

Member

no need for this, pytest-xdist is installed by ci-helpers when SETUP_CMD has parallel or numprocesses in it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.