Coveralls.io #144

Closed
wants to merge 3 commits into
from

Projects

None yet

3 participants

@singingwolfboy
Contributor

Measure test coverage, report results to Coveralls, display a badge in the README. Note that this depends on the tests measuring code coverage -- there's a pull request for that here: pyinvoke/invocations#5

@bitprophet
Member

Thanks, have been meaning to poke more at coverage and linting. Will see if I can get to this sometime soon.

@bitprophet bitprophet added the Support label Jun 15, 2014
singingwolfboy added some commits Jun 13, 2014
@singingwolfboy singingwolfboy Refactor Travis badge, add Coveralls badge 95a3100
@singingwolfboy singingwolfboy Report coverage information to Coveralls.io be14e36
@singingwolfboy singingwolfboy Switch requirement to singingwolfboy fork
Remove this commit when pyinvoke/invocations#5 is
merged
0bef2e0
@singingwolfboy
Contributor
@justinabrahms

I'd (humbly) offer checking out codecov.io. It will report on how a particular PR is covered, in addition to what coveralls does, which is report on how the PR affects coverage.

I'm much less interested in (+0.02% coverage) than I am (This PR is 48% covered).

@bitprophet bitprophet added this to the 0.11 milestone Jun 17, 2015
@bitprophet
Member

a) I've been poking more at vanilla coverage.py lately and it is real nice, so this topic is on my mind (also commented on the invocations PR, since I accidentally wrote my own coverage method)
b) I agree with @justinabrahms that coveralls' output (as seen over on paramiko/paramiko for the last, like, yearish) is really not very helpful, I should check out codecov.io sometime.

@bitprophet
Member

Double checked the current state of features for coveralls.io vs codecov and from their websites at least, codecov does still look more useful, coveralls continues to have the "+N% coverage" focus which is less useful.

Will test out actually using codecov and if I like how it looks, will add that to travis.

@bitprophet
Member

Hrm, don't see any way to tell codecov to give flags to the report generation; currently my coverage stuff has to tell coverage html to --include invoke/* and --omit invoke/vendor/*, otherwise it thinks my suite only covers like 15% because it doesn't test most of the vendored material.

Similarly I don't see a way to give this info to coveralls.io.

Digging further to see why I opted to do the filtering at html-generation time, maybe I'll get lucky and there's a way to do it at coverage-generation time instead. Otherwise this will be kind of a no-op and I'll just have to jury-rig something stupid like "generate coverage report, then inspect it for the coverage percentage".

@bitprophet
Member

Sweet, apparently at the time I was ignorant of the use of .coveragerc or it didn't exist - my suite needs to run under a nosetests wrapper, so coverage run isn't a good option. But nosetests --with-coverage honors .coveragerc so I think I am in business.

@bitprophet
Member

Test upload seems to work reasonably well: https://codecov.io/github/pyinvoke/invoke?ref=07a956e789c25b721bd08442624b60c70efd47a8

It's like 0.5% less coverage than native coverage html reports, and this seems to be because it's only seeing about 1/2 of the branches - coverage html shows 584 branches but codecov only shows 289.

The 1/2-ness is constant across all covered files, which is...pretty bizarre. Can't find much online about this besides a probably-not-related twitter convo: https://twitter.com/glenathan/status/603883608651264000

That said, when it represents such a small amount-of-coverage for now, and only seems to affect branch coverage which is itself always a dicey proposition, Good Enough For Now.

@justinabrahms

Glad to hear that. The founder there is pretty responsive, in my
experience. Consider messaging him.

On Monday, December 21, 2015, Jeff Forcier notifications@github.com wrote:

Test upload seems to work reasonably well:
https://codecov.io/github/pyinvoke/invoke?ref=07a956e789c25b721bd08442624b60c70efd47a8

It's like 0.5% less coverage than native coverage html reports, and this
seems to be because it's only seeing about 1/2 of the branches - coverage
html shows 584 branches but codecov only shows 289.

The 1/2-ness is constant across all covered files, which is...pretty
bizarre. Can't find much online about this besides a probably-not-related
twitter convo: https://twitter.com/glenathan/status/603883608651264000

That said, when it represents such a small amount-of-coverage for now, and
only seems to affect branch coverage which is itself always a dicey
proposition, Good Enough For Now.


Reply to this email directly or view it on GitHub
#144 (comment).

Sent via mobile phone.

@bitprophet
Member

Good to know, thanks again @justinabrahms!

Re: other bits in this PR, codecov's equivalent status badge can be found on https://codecov.io/github/pyinvoke/invoke/features/badges so I'll work that in, and then preserve the Travis status badge too cuz why not :) EDIT: oh right, that stuff's in our website sidebar, so having it in the README is kind of meh. May take a quick stab at adding it to the sidebar under the Travis hook.

@bitprophet
Member

Welp Alabaster 0.7.7 is now out with Codecov button support :3

@bitprophet bitprophet added a commit that closed this pull request Dec 22, 2015
@bitprophet bitprophet Changelog closes #144 38e0b8f
@bitprophet
Member

Set CodeCov PR adjusted-line coverage to 90% for now, partly cuz that'd be nice, partly because I assume lots of PRs will fail and I want to see what failure looks like initially :3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment