New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable the "changes" status from CodeCov #1827

Merged
merged 1 commit into from Dec 6, 2018

Conversation

2 participants
@smashwilson
Copy link
Member

smashwilson commented Dec 6, 2018

Please be sure to read the contributor's guide to the GitHub package before submitting any pull requests.

Requirements

  • Filling out the template is required. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
  • Suggestion: You can use checklists to keep track of progress for the sections on metrics, tests, documentation, and user research.

Description of the Change

Disable the "changes" status reported by CodeCov.

Alternate Designs

I'd originally tried to build us some slack by configuring it with a threshold key, but that doesn't appear to work. In retrospect, that makes sense - the changes status is reported as a total number of files, not by coverage percentage like the others are - and the commit status docs kind of support this.

Benefit

It'll unblock pull requests like #1822 that currently have a red "changes" check that we can't do anything about from merging.

It'll also prevent our CI-watching chatops from spamming the #atom channel on every merge to master.

Possible Drawbacks

There are some situations where we'll be able to incidentally decrease coverage and still get passing statuses from CodeCov. Deleting tests is one example.

Applicable Issues

n/a

Metrics

n/a

Tests

I'm going to see if this pull request doesn't have the "changes" status reported from CodeCov. 馃檭

Documentation

n/a

Release Notes

n/a

User Experience Research (Optional)

n/a

@smashwilson smashwilson added the ci label Dec 6, 2018

@smashwilson smashwilson requested a review from atom/github-package Dec 6, 2018

@smashwilson

This comment has been minimized.

Copy link
Member

smashwilson commented Dec 6, 2018

I've also removed codecov/changes from the required statuses in our protected branches config.

Reviewers: how do you feel about this? My intention would be re-enable this if and when we get to a point where our test coverage is stable. I think there are only a few lines that are causing coverage flapping so I don't think that's too far off.

@kuychaco

This comment has been minimized.

Copy link
Member

kuychaco commented Dec 6, 2018

I think there are only a few lines that are causing coverage flapping so I don't think that's too far off

Any idea what lines? Maybe it's worth trying to fix them upfront?

Also, perhaps it's worth reaching out to Codecov support with this question/issue? They seem to be very welcoming of this... they have an "Ask Codecov staff to review reports" section here

@smashwilson

This comment has been minimized.

Copy link
Member

smashwilson commented Dec 6, 2018

Any idea what lines? Maybe it's worth trying to fix them upfront?

Back on Coveralls I consistently saw a few lines in Repository and I think RootController or GithubPackage that would flap coverage. But I don't have a really good way to see which lines they were right now.

I think the changes status is supposed to link to a view that shows them to us once we have a base commit with a coverage report? The page doesn't load for me, though, which probably is worth reaching out to support about.

@kuychaco
Copy link
Member

kuychaco left a comment

馃憤 on disabling the Changes Status temporarily to unblock other PRs, reaching out to Codecov support, and re-enabling once we figure out what the deal is. Thanks @smashwilson!

Stability Sprint : 20 November 2018 - 8 January 2019 : v0.24.0 automation moved this from In Progress 馃敡 to QA Review 馃敩 Dec 6, 2018

@smashwilson smashwilson merged commit 869d817 into master Dec 6, 2018

2 checks passed

codecov/patch Coverage not affected when comparing 55a6d63...863cd07
Details
codecov/project 89.52% remains the same compared to 55a6d63
Details

Stability Sprint : 20 November 2018 - 8 January 2019 : v0.24.0 automation moved this from QA Review 馃敩 to Merged 鈽戯笍 Dec 6, 2018

@smashwilson smashwilson deleted the aw/codecov-changes-off branch Dec 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment