New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

⚗️ Test coverage in Travis #32

Closed
D4nte opened this Issue May 11, 2018 · 15 comments

Comments

Projects
None yet
2 participants
@D4nte
Copy link
Member

D4nte commented May 11, 2018

Spike:

  • Check if it is easy to see whether new lines (from PR diff) are tested or not
  • Check if test coverage can be ran on the e2e tests.

Notes:
Use cargo make
Add badge to readme
See https://testing.googleblog.com/2014/07/measuring-coverage-at-google.html?m=1

@thomaseizinger

This comment has been minimized.

Copy link
Member

thomaseizinger commented May 12, 2018

I am not sure if that is possible for Rust. We should look into it though.

@D4nte

This comment has been minimized.

Copy link
Member Author

D4nte commented May 13, 2018

@thomaseizinger

This comment has been minimized.

Copy link
Member

thomaseizinger commented Sep 4, 2018

Apparently, tarpaulin is the tool to use.

@D4nte D4nte added the icebox label Nov 22, 2018

@D4nte D4nte added groomed and removed icebox labels Dec 14, 2018

@D4nte D4nte added sprint-backlog and removed groomed labels Jan 8, 2019

@D4nte D4nte changed the title Test coverage in Travis ⚗️ Test coverage in Travis Jan 8, 2019

@D4nte D4nte added research and removed good first issue labels Jan 8, 2019

@D4nte D4nte added this to the Sprint 5 📜🔓 milestone Jan 9, 2019

@D4nte D4nte self-assigned this Jan 10, 2019

@D4nte

This comment has been minimized.

Copy link
Member Author

D4nte commented Jan 10, 2019

You can see better coverage with:

  1. Install Sourcegraph for Chrome or Sourcegraph for Firefox
  2. Enable the Codecov extension on Sourcegraph (little dot)

However, the code coverage is not shown in the diff (which is what I want), ie, need to open the new version of the file standalone.

@D4nte

This comment has been minimized.

Copy link
Member Author

D4nte commented Jan 10, 2019

Need to check:

  • codacy: no Rust support
  • coverall: no Rust support, cool website though https://coveralls.io/, Actually tarpaulin mentions coveralls.io
@D4nte

This comment has been minimized.

Copy link
Member Author

D4nte commented Jan 16, 2019

coveralls: no coverage per line for diff. back to codecov.

@D4nte

This comment has been minimized.

Copy link
Member Author

D4nte commented Jan 17, 2019

Key part I find interesting: https://docs.codecov.io/docs/viewing-source-code
Usage would be:

  1. Do code review
  2. Spot a complex piece of code/change, open codecoverage
  3. Check that the new code is properly covered or that coverage was not lost
@thomaseizinger

This comment has been minimized.

Copy link
Member

thomaseizinger commented Jan 17, 2019

Apparently, there is browser plugin for viewing code coverage in the code review on GitHub.

Check the last screenshot here: https://github.com/marketplace/codecov

@D4nte

This comment has been minimized.

Copy link
Member Author

D4nte commented Jan 17, 2019

Tarpaulin does not work on mac os x: xd009642/tarpaulin#152
So either Docker or travis only.

@D4nte

This comment has been minimized.

Copy link
Member Author

D4nte commented Jan 17, 2019

@thomaseizinger yes, sourcegraph. I installed it and tried last week but was not great. You still go to an external site to see the per line coverage. It does give you more info like coverage per file. Then you click on the button which shows the percent and you get to the external website.
I think going to the codecov diff report is as good.

@D4nte

This comment has been minimized.

Copy link
Member Author

D4nte commented Jan 17, 2019

Creating docker image to play around with tarpaulin on mac os x...
Objective: see if I can get codecov for e2e tests

@thomaseizinger

This comment has been minimized.

Copy link
Member

thomaseizinger commented Jan 17, 2019

Tarpaulin does not work on mac os x: xd009642/tarpaulin#152
So either Docker or travis only.

IMO, code coverage is something you don't do locally because it is usually a more complicated setup. (+ you need historical data for it do be useful)

@thomaseizinger yes, sourcegraph. I installed it and tried last week but was not great.

It doesn't work like in the video demonstrated here? https://docs.codecov.io/docs/browser-extension

Looks like you can see the line-coverage inline in the PR. I don't think we need anything more than that since at this point, I think we should only use code coverage to easily detect untested code.

@thomaseizinger

This comment has been minimized.

Copy link
Member

thomaseizinger commented Jan 17, 2019

The extension changed apparently since the video was made. It no longer has the % button. That is a shame :(

@D4nte

This comment has been minimized.

Copy link
Member Author

D4nte commented Jan 17, 2019

Ah, thanks for the explanation, I was going to reinstall the extension :)
Please note the video is about showing in GitHub on code. It is not about showing on a Pull Request/code diff.

Re. Local installation, actually it can be something interesting. Maybe not on our small software but when it gets bigger, the following local flow can be useful:

  1. write code
  2. run e2e
  3. double check new code is tested
@D4nte

This comment has been minimized.

Copy link
Member Author

D4nte commented Jan 22, 2019

As per #675 I ran into many issues. At the end, it looks like tarpaulin could not compile with zmq (rust bindings). After discussion, we decided not too investigate further and drop this issue.

Code coverage can be reconsidered at a later stage when comit-rs is split into libraries.

@D4nte D4nte closed this Jan 22, 2019

@wafflebot wafflebot bot removed the work-in-progress label Jan 22, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment