Codecov uploader hangs at the end of uploading.

[Wayonb]

Describe the bug
It seems like the codecov uploader hangs at the end.
It seems the command shell was never returned to Jenkins.
After codecov loader say it was successful with the upload, our Jenkins job sits there for 3 hours until it timeout.

To Reproduce
This seems to be random so I dont have a repo yet.

Expected behavior
I except the uploader to finish successful.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context

[2023-10-17T01:29:20.379Z] Info: Running script `codecov --verbose --nonZero --rootDir /home/ubuntu/jenkins/workspace --flags faucet-authenticator --dir .`

[Pipeline] isUnix

[Pipeline] sh

[2023-10-17T01:29:20.744Z] + codecov --verbose --nonZero --rootDir /home/ubuntu/jenkins/workspace --flags faucet-authenticator --dir .

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.812Z] ['verbose'] Start of uploader: 1697506160812...

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.814Z] ['info'] 

[2023-10-17T01:29:21.011Z]      _____          _

[2023-10-17T01:29:21.011Z]     / ____|        | |

[2023-10-17T01:29:21.011Z]    | |     ___   __| | ___  ___ _____   __

[2023-10-17T01:29:21.011Z]    | |    / _ \ / _` |/ _ \/ __/ _ \ \ / /

[2023-10-17T01:29:21.011Z]    | |___| (_) | (_| |  __/ (_| (_) \ V /

[2023-10-17T01:29:21.011Z]     \_____\___/ \__,_|\___|\___\___/ \_/

[2023-10-17T01:29:21.011Z] 

[2023-10-17T01:29:21.011Z]   Codecov report uploader 0.6.2

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.814Z] ['info'] => Project root located at: /home/ubuntu/jenkins/workspace

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.819Z] ['info'] ->  Token found by environment variables

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.820Z] ['verbose'] Start of network processing...

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.820Z] ['verbose'] Searching for files in /home/ubuntu/jenkins/workspace

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.838Z] ['verbose'] coveragepy is not installed

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.839Z] ['info'] Searching for coverage files...

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.965Z] ['info'] Warning: Some files located via search were excluded from upload.

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.965Z] ['info'] If Codecov did not locate your files, please review https://docs.codecov.com/docs/supported-report-formats

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.965Z] ['verbose'] Preparing to clean the following coverage paths: coverage/lcov.info,coverage/tmp/coverage-207-1697506151474-0.json,coverage/tmp/coverage-219-1697506151415-0.json

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.965Z] ['info'] => Found 3 possible coverage files:

[2023-10-17T01:29:21.011Z]   coverage/lcov.info

[2023-10-17T01:29:21.011Z]   coverage/tmp/coverage-207-1697506151474-0.json

[2023-10-17T01:29:21.011Z]   coverage/tmp/coverage-219-1697506151415-0.json

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.965Z] ['verbose'] End of network processing

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.966Z] ['info'] Processing coverage/lcov.info...

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.967Z] ['info'] Processing coverage/tmp/coverage-207-1697506151474-0.json...

[2023-10-17T01:29:21.011Z] [2023-10-17T01:29:20.969Z] ['info'] Processing coverage/tmp/coverage-219-1697506151415-0.json...

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.058Z] ['info'] Detected Jenkins CI as the CI provider.

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.058Z] ['verbose'] -> Using the following env variables:

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.058Z] ['verbose']      BRANCH_NAME: dev

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.058Z] ['verbose']      BUILD_NUMBER: 178

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.058Z] ['verbose']      BUILD_URL: https://jenkins.symboldev.com/job/Symbol/job/generated/job/product/job/faucet-authenticator/job/dev/178/

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.059Z] ['verbose']      CHANGE_ID: undefined

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.059Z] ['verbose']      GIT_BRANCH: dev

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.059Z] ['verbose']      GIT_COMMIT: bbd2ece4ade8b8e64f88d9d68c95c4e2ba368c70

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.059Z] ['verbose']      JENKINS_URL: https://jenkins.symboldev.com/

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.059Z] ['verbose']      ghprbActualCommit: undefined

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.059Z] ['verbose']      ghprbPullId: undefined

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.059Z] ['verbose']      ghprbSourceBranch: undefined

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.067Z] ['verbose'] Using the following upload parameters:

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.067Z] ['verbose'] branch

[2023-10-17T01:29:21.268Z] [2023-10-17T01:29:21.067Z] ['verbose'] build

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.067Z] ['verbose'] buildURL

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.067Z] ['verbose'] commit

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.067Z] ['verbose'] job

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.067Z] ['verbose'] pr

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.067Z] ['verbose'] service

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.067Z] ['verbose'] slug

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.067Z] ['verbose'] name

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.067Z] ['verbose'] tag

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.067Z] ['verbose'] flags

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.067Z] ['verbose'] parent

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.069Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.6.2&token=*******&branch=dev&build=178&build_url=https%3A%2F%2Fjenkins.symboldev.com%2Fjob%2FSymbol%2Fjob%2Fgenerated%2Fjob%2Fproduct%2Fjob%2Ffaucet-authenticator%2Fjob%2Fdev%2F178%2F&commit=bbd2ece4ade8b8e64f88d9d68c95c4e2ba368c70&job=&pr=&service=jenkins&slug=Symbol%2Fproduct&name=&tag=&flags=faucet-authenticator&parent=

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.069Z] ['verbose'] Passed token was 36 characters long

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.069Z] ['verbose'] https://codecov.io/upload/v4?package=uploader-0.6.2&branch=dev&build=178&build_url=https%3A%2F%2Fjenkins.symboldev.com%2Fjob%2FSymbol%2Fjob%2Fgenerated%2Fjob%2Fproduct%2Fjob%2Ffaucet-authenticator%2Fjob%2Fdev%2F178%2F&commit=bbd2ece4ade8b8e64f88d9d68c95c4e2ba368c70&job=&pr=&service=jenkins&slug=Symbol%2Fproduct&name=&tag=&flags=faucet-authenticator&parent=

[2023-10-17T01:29:21.269Z]         Content-Type: 'text/plain'

[2023-10-17T01:29:21.269Z]         Content-Encoding: 'gzip'

[2023-10-17T01:29:21.269Z]         X-Reduced-Redundancy: 'false'

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.208Z] ['info'] https://app.codecov.io/github/symbol/product/commit/bbd2ece4ade8b8e64f88d9d68c95c4e2ba368c70

[2023-10-17T01:29:21.269Z] https://storage.googleapis.com/codecov/v4/raw/2023-10-17/FBE5B1B0EDC96BFF7A0B1CD19EF20BCC/bbd2ece4ade8b8e64f88d9d68c95c4e2ba368c70/f031280a-a06f-4d1e-8b12-abde1a04c139.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20231017%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20231017T012921Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=0a84c468a9311690d68058e5de04c50b5e9f91f6c20a9453d842e49053a889fa

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.208Z] ['verbose'] Returned upload url: https://storage.googleapis.com/codecov/v4/raw/2023-10-17/FBE5B1B0EDC96BFF7A0B1CD19EF20BCC/bbd2ece4ade8b8e64f88d9d68c95c4e2ba368c70/f031280a-a06f-4d1e-8b12-abde1a04c139.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20231017%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20231017T012921Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=0a84c468a9311690d68058e5de04c50b5e9f91f6c20a9453d842e49053a889fa

[2023-10-17T01:29:21.269Z] [2023-10-17T01:29:21.208Z] ['info'] Uploading...

[2023-10-17T01:29:21.528Z] [2023-10-17T01:29:21.364Z] ['info'] {"status":"success","resultURL":"https://app.codecov.io/github/symbol/product/commit/bbd2ece4ade8b8e64f88d9d68c95c4e2ba368c70"}

[2023-10-17T01:29:21.528Z] [2023-10-17T01:29:21.364Z] ['verbose'] End of uploader: 552 milliseconds

[2023-10-17T04:27:44.613Z] Cancelling nested steps due to timeout

[2023-10-17T04:27:44.623Z] Sending interrupt signal to process

[2023-10-17T04:27:51.989Z] Terminated

[2023-10-17T04:27:51.993Z] script returned exit code 143

[renefritze]

We're also hit with this bug. Environment is powershell in Azure DevOps.

[drazisil-codecov]

[2023-10-17T01:29:21.528Z] [2023-10-17T01:29:21.364Z] ['verbose'] End of uploader: 552 milliseconds

This tells me the uploader is completing. Is it wrapped in a script or some other process that may not realize this and is hanging?

[renefritze]

I've re-run our pipelines with -Z removed and --verbose added which no longer triggers the hang.

[drazisil-codecov]

That's really strange. The relevant code:

uploader/bin/codecov.ts

Line 32 in 02e1554

process.exit(realArgs.nonZero ? -1 : 0)

and

uploader/src/helpers/logger.ts

Lines 18 to 22 in 02e1554

	export function verbose(message: string, shouldVerbose: boolean): void {
	if (shouldVerbose === true) {
	console.debug(`[${_getTimestamp()}] ['verbose'] ${message}`)
	}
	}

Should not cause a hang at exit time in any way that I can think of. 🤔

[renefritze]

Re-run a couple of times with original settings too and can no longer reproduce.

[drazisil-codecov]

@Wayonb , as the reporter, are you still seeing this issue?

[Wayonb]

No, I haven't seen the issue since we upgraded to 0.7

[drazisil-codecov]

Appears to no longer be an issue