-
-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC] consider droping codecov #6994
Comments
This is due to flaky coverage - e.g. observed with Windows+xdist - not using xdist makes it stable (for that part). But there are other parts that are racy, would need to be covered explicitly. |
IMO it's important to have coverage measurements, but we don't need to use an external service. For Hypothesis we just use the |
I don't see how switching to another service would change anything - all that codecov does is presenting the data, if that data is flaky, switching to e.g. coveralls won't change much. +1 on disabling the PR status check as long as that's the case, though. |
im fine with that - given the presented arguments i believe status checks should pause but at the same time we should open a project for trying for 100% reported coverage (for which well placed+reasoned coverage ignores may be acceptable) |
I'd approach this from the other direction:
This involves a little more code churn, but means we get to enforce 'all new code must be covered' instantly. It's also pretty easy to check our progress with |
@Zac-HD i love that suggestion from my pov it expands mine by enabling us to place coverage todos and issues while trying to enforce full coverage on new commits so in effect putting much better and stricter tooling into the project i proposed the project would just group the issues/tickets/pr's around sorting out coverage |
I also like the idea of ensuring full coverage for new commits, and the proposed solution by @Zac-HD would take care of getting they flaky coverage out of the way. FWIW the coverage checks are not required at the moment: |
codecov's diff status is usually correct. It defaults to requirering increased coverage, and could be bumped to 100%. |
i created https://github.com/pytest-dev/pytest/projects/6 to organize tracking this effort |
I find the "patch" report useful, it just shows which parts of the diff are covered and fails if something isn't. But the "project" one, not so much, often it fails without any apparent reason and no way to fix in the PR. |
There's also https://github.com/wemake-services/coverage-conditional-plugin to cover some complex cases. |
+1 for disabling codecov/project status but keeping codecov/patch |
How about this then:
I propose this because it might take a while to get to 2, while we can ripe the benefits of 1 fairly quickly. |
Sounds good to me 👍 |
I like this idea |
With codecov i observe a lot of failures/issues/false positives.
So i would appreciate if we found and switched to a more stable alternative.
The text was updated successfully, but these errors were encountered: