Bug Report: codecovcli upload-process 502 - connection failure

[chriscerk]

Describe the bug
A clear and concise description of what the bug is.

codecovcli upload-process is failing which is blocking teams from merging AND proceeding past CI.

info - 2024-04-16 19:18:55,080 -- ci service found: circleci
warning - 2024-04-16 19:19:25,133 -- Response status code was 502. --- {"retry": 0}
warning - 2024-04-16 19:19:25,134 -- Request failed. Retrying --- {"retry": 0}
info - 2024-04-16 19:19:25,676 -- Process Commit creating complete
error - 2024-04-16 19:19:25,676 -- Commit creating failed: upstream connect error or disconnect/reset before headers. reset reason: connection failure

Environment (please complete the following information):

• System: codecovcli via CircleCI

To Reproduce

codecovcli upload-process

This is affecting all repos.

Expected behavior
A clear and concise description of what you expected to happen.

Not to fail.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[chriscerk]

We are no longer getting this error, but this was a serious disruption for us.

I noticed https://status.codecov.com did not report an outage - does any codecov telemetry correlate with when this occurred today?

[chriscerk]

@thomasrockhu-codecov we are encountering this today again.

[thomasrockhu-codecov]

@chriscerk I noticed a spike in 5xx's on our side which we have since mitigated. Are you still seeing this?

[chriscerk]

@thomasrockhu-codecov I haven't noticed any today, but April 16th & 17th we did see them and it was blocking merges + CI proceeding for our teams.

[thomasrockhu-codecov]

@chriscerk got it, we have fixed up an issue in our infra that caused the initial issue. That shouldn't be a root cause anymore.

[chriscerk]

@thomasrockhu-codecov awesome - thanks! 🎉

We usually expect the https://status.circleci.com to reflect if something goes wrong - would this be reported there?
(or this type of infra issue in the future to be observable from that page)

update: looks like it is visible on the status page now

[drazisil-codecov]

Hi @chriscerk ,

Yes, https://status.codecov.com/ usually does list these types of issues. Depending on when they are discovered and what is involved it fixing it, the status is sometimes created after the fact, but we are actively working to remember to update it while we are resolving it, not after.