Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coveralls.io #144

Closed
wants to merge 3 commits into from
Closed

Coveralls.io #144

wants to merge 3 commits into from

Conversation

@singingwolfboy
Copy link
Contributor

@singingwolfboy singingwolfboy commented Jun 13, 2014

Measure test coverage, report results to Coveralls, display a badge in the README. Note that this depends on the tests measuring code coverage -- there's a pull request for that here: pyinvoke/invocations#5

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Jun 13, 2014

Thanks, have been meaning to poke more at coverage and linting. Will see if I can get to this sometime soon.

Loading

@singingwolfboy
Copy link
Contributor Author

@singingwolfboy singingwolfboy commented Jun 17, 2015

Loading

@justinabrahms
Copy link

@justinabrahms justinabrahms commented Jun 17, 2015

I'd (humbly) offer checking out codecov.io. It will report on how a particular PR is covered, in addition to what coveralls does, which is report on how the PR affects coverage.

I'm much less interested in (+0.02% coverage) than I am (This PR is 48% covered).

Loading

@bitprophet bitprophet added this to the 0.11 milestone Jun 17, 2015
@bitprophet
Copy link
Member

@bitprophet bitprophet commented Jun 17, 2015

a) I've been poking more at vanilla coverage.py lately and it is real nice, so this topic is on my mind (also commented on the invocations PR, since I accidentally wrote my own coverage method)
b) I agree with @justinabrahms that coveralls' output (as seen over on paramiko/paramiko for the last, like, yearish) is really not very helpful, I should check out codecov.io sometime.

Loading

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Dec 21, 2015

Double checked the current state of features for coveralls.io vs codecov and from their websites at least, codecov does still look more useful, coveralls continues to have the "+N% coverage" focus which is less useful.

Will test out actually using codecov and if I like how it looks, will add that to travis.

Loading

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Dec 22, 2015

Hrm, don't see any way to tell codecov to give flags to the report generation; currently my coverage stuff has to tell coverage html to --include invoke/* and --omit invoke/vendor/*, otherwise it thinks my suite only covers like 15% because it doesn't test most of the vendored material.

Similarly I don't see a way to give this info to coveralls.io.

Digging further to see why I opted to do the filtering at html-generation time, maybe I'll get lucky and there's a way to do it at coverage-generation time instead. Otherwise this will be kind of a no-op and I'll just have to jury-rig something stupid like "generate coverage report, then inspect it for the coverage percentage".

Loading

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Dec 22, 2015

Sweet, apparently at the time I was ignorant of the use of .coveragerc or it didn't exist - my suite needs to run under a nosetests wrapper, so coverage run isn't a good option. But nosetests --with-coverage honors .coveragerc so I think I am in business.

Loading

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Dec 22, 2015

Test upload seems to work reasonably well: https://codecov.io/github/pyinvoke/invoke?ref=07a956e789c25b721bd08442624b60c70efd47a8

It's like 0.5% less coverage than native coverage html reports, and this seems to be because it's only seeing about 1/2 of the branches - coverage html shows 584 branches but codecov only shows 289.

The 1/2-ness is constant across all covered files, which is...pretty bizarre. Can't find much online about this besides a probably-not-related twitter convo: https://twitter.com/glenathan/status/603883608651264000

That said, when it represents such a small amount-of-coverage for now, and only seems to affect branch coverage which is itself always a dicey proposition, Good Enough For Now.

Loading

@justinabrahms
Copy link

@justinabrahms justinabrahms commented Dec 22, 2015

Glad to hear that. The founder there is pretty responsive, in my
experience. Consider messaging him.

On Monday, December 21, 2015, Jeff Forcier notifications@github.com wrote:

Test upload seems to work reasonably well:
https://codecov.io/github/pyinvoke/invoke?ref=07a956e789c25b721bd08442624b60c70efd47a8

It's like 0.5% less coverage than native coverage html reports, and this
seems to be because it's only seeing about 1/2 of the branches - coverage
html shows 584 branches but codecov only shows 289.

The 1/2-ness is constant across all covered files, which is...pretty
bizarre. Can't find much online about this besides a probably-not-related
twitter convo: https://twitter.com/glenathan/status/603883608651264000

That said, when it represents such a small amount-of-coverage for now, and
only seems to affect branch coverage which is itself always a dicey
proposition, Good Enough For Now.


Reply to this email directly or view it on GitHub
#144 (comment).

Sent via mobile phone.

Loading

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Dec 22, 2015

Good to know, thanks again @justinabrahms!

Re: other bits in this PR, codecov's equivalent status badge can be found on https://codecov.io/github/pyinvoke/invoke/features/badges so I'll work that in, and then preserve the Travis status badge too cuz why not :) EDIT: oh right, that stuff's in our website sidebar, so having it in the README is kind of meh. May take a quick stab at adding it to the sidebar under the Travis hook.

Loading

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Dec 22, 2015

Welp Alabaster 0.7.7 is now out with Codecov button support :3

Loading

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Dec 22, 2015

Set CodeCov PR adjusted-line coverage to 90% for now, partly cuz that'd be nice, partly because I assume lots of PRs will fail and I want to see what failure looks like initially :3

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants