codecov.io is flaky and can fail builds

[friedbunny]

When codecov.io connectivity is poor (which seems to happen with some frequency), some of our (otherwise completely successful) builds fail with:

#!/bin/bash -eo pipefail
curl -sSfL -o codecov https://codecov.io/bash
chmod +x codecov
./codecov -c
curl: (7) Failed to connect to codecov.io port 443: Connection timed out
Exited with code 7

I’m not sure it would mitigate the issue, but it seems inherently unsafe to be downloading and executing an arbitrary bash script for every run — committing the script to our repo would at least eliminate one network request.

/cc @jmkiley @springmeyer

[springmeyer]

@friedbunny I've never seen this before, so thanks for ticketing.

./codecov -c
curl: (7) Failed to connect to codecov.io port 443: Connection timed out

Can you link to a few builds where this happened? Do you see any pattern? Given I've not seen this in other repos (and we have a lot with codecov enabled), perhaps gl-native is hitting this more often due to the size of the upload? How big is it?

committing the script to our repo would at least eliminate one network request.

Doubt this would help. The download of that file is tiny and the failure is in the put of coverage data. But definitely fine to check the code in if you want.

[friedbunny]

Thanks, @springmeyer — you’re definitely right about checking in the script not making a difference; I misread the (very short 😅) log and was thinking it was the script download timing out, not the upload.

Can you link to a few builds where this happened?

• https://circleci.com/gh/mapbox/mapbox-gl-native/298437 (linked in original post)
• https://circleci.com/gh/mapbox/mapbox-gl-native/300348

Do you see any pattern? How big is [the upload]?

I don’t see a pattern, since this is failing on iOS-only changes (which aren’t included in the coverage). As for the size of the upload, that’s something that’ll need investigation.

[friedbunny]

Another example: https://circleci.com/gh/mapbox/mapbox-gl-native/307070

[friedbunny]

And: https://circleci.com/gh/mapbox/mapbox-gl-native/307075

[friedbunny]

It appears that this is a common issue with codecov.io — codecov/codecov-python#158 (comment). Workarounds look like just retrying it until it succeeds. For now, I’ve opted to not let it fail builds: #15248

[stale]

This issue has been automatically detected as stale because it has not had recent activity and will be archived. Thank you for your contributions.