Add code coverage options due to many codecov failures recently

[marinegor]

Is your feature request related to a problem?

When contributing to MDAnalysis via fork-commit-pullrequest pipeline, I often see a codecov-related failure on CI/CD, while all the tests run correctly.

Describe the solution you'd like

I'd suggest adding alternative code coverage options, in order to be able to migrate from codecov quickly if nothing changes, and/or provide a better overall contributor experience.

Describe alternatives you've considered

Afaik, choices are: coveralls, codacy, code climate. My source of choices: microsoft/torchgeo#1995

[IAlibay]

copying over my response from discord:

Short term

In the very short term (i.e. the next 3-4 weeks), I would suggest we (MDA) hangs back and waits for whatever solution the codecov team claims they are working on. We have much bigger CI fires going on unfortunately and doing a coverage migration is probably something that'll take a decent amount of discussion & time anyways.

For MDA contributors that are within the org, my very simple solution is directly branch from the MDA repo rather than work from forks. That'll make codecov failures go away for now. Outside of that, we might just need reviewers to manually run coverage on local checkouts for anything they aren't too sure about.

Long term

Longer term - if nothing happens within codecov by end of May, then we should take the time to review the pro/cons of alternate providers and see which is best to move from.

More immediately

• In the immediate term - i.e. today, I would suggest opening up either an issue or discussion where we can keep track of all this and discuss it as we go along. (this is done)

[marinegor]

Following up the discussion on github -- I still don't understand why codecov necessarily should apear separately on checks (like this one), and also for each platform independently. Could we leave only the latter, thereby eliminating the codecov for each platform, but making ci/cd more robust?

[IAlibay]

Following up the discussion on github -- I still don't understand why codecov necessarily should apear separately on checks (like this one), and also for each platform independently. Could we leave only the latter, thereby eliminating the codecov for each platform, but making ci/cd more robust?

Apologies for the briefness - different OS platforms have different code branches (see the various windows skips for example), and different runners have different amounts of optional dependencies, etc... Hence you need multiple reports to cover all the edge cases.

The codecov steps on each test runner is only an upload to codecov, not a report.

The actual codecov check is the check of the sum of the reports - i.e. is my combined percentage coverage going up or down.

[IAlibay]

I believe this has been fixed upstream / no longer an issue, so closing.