Error Uploading - Status 500 - 3 of 12 jobs failed with differing number of missed files

[jhy]

Describe the bug
For 3 of 12 matrix builds, the upload artifact stage failed the build on status 500s from the server.

Logs are available here: https://github.com/jhy/jsoup/runs/1554796145?check_suite_focus=true (public repo)

I saw other similar but not exactly the same bug (either enterprise repo, or 503s vs 500, so apologies if this is a dupe.

Version

• V1
• V2

Environment

• self-hosted
• Linux
• Windows
• Mac

Screenshots
N/A

Run/Repo Url
https://github.com/jhy/jsoup/runs/1554796145?check_suite_focus=true

How to reproduce
If applicable, add information on how to reproduce the problem.

Additional context
Add any other context about the problem here.

[tqre]

Getting same error:

Describe the bug
Upload artifact stage failed the build with status 500.

Version
V2

Environment
Linux

Run/Repo Url
https://github.com/archlinuxhardened/selinux/pull/79/checks?check_run_id=1757017996

[joshuapinter]

I'm seeing a similar thing on the latest version (I'm assuming, since we're using actions/checkout@v2:

With the provided path, there will be 10 files uploaded
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 5548.578148597127 milliseconds before continuing the upload at offset 0
An error has been caught http-client index 1, retrying the upload
Error: Client has already been disposed.
    at HttpClient.request (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:5694:19)
    at HttpClient.sendStream (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:5655:21)
    at UploadHttpClient.<anonymous> (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:7104:37)
    at Generator.next (<anonymous>)
    at .../_work/_actions/actions/upload-artifact/v2/dist/index.js:6834:71
    at new Promise (<anonymous>)
    at module.exports.608.__awaiter (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:6830:12)
    at uploadChunkRequest (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:7102:46)
    at UploadHttpClient.<anonymous> (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:7139:38)
    at Generator.next (<anonymous>)
    Exponential backoff for retry #1. Waiting for 5773.927291417539 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #2. Waiting for 9362.643214894475 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
Total file count: 10 ---- Processed file #9 (90.0%)
Finished backoff for retry #2, continuing with upload
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #3. Waiting for 14276.414696905516 milliseconds before continuing the upload at offset 0
Total file count: 10 ---- Processed file #9 (90.0%)
Total file count: 10 ---- Processed file #9 (90.0%)
Finished backoff for retry #3, continuing with upload
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #4. Waiting for 20561.212693027403 milliseconds before continuing the upload at offset 0
Total file count: 10 ---- Processed file #9 (90.0%)
Total file count: 10 ---- Processed file #9 (90.0%)
Finished backoff for retry #4, continuing with upload
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #5. Waiting for 31215.480894221528 milliseconds before continuing the upload at offset 0
Total file count: 10 ---- Processed file #9 (90.0%)
Total file count: 10 ---- Processed file #9 (90.0%)
Total file count: 10 ---- Processed file #9 (90.0%)
Finished backoff for retry #5, continuing with upload
A 500 status code has been received, will attempt to retry the upload
##### Begin Diagnostic HTTP information #####
Status Code: 500
Status Message: Internal Server Error
Header Information: {
  "cache-control": "no-store,no-cache",
  "pragma": "no-cache",
  "content-length": "328",
  "content-type": "application/json; charset=utf-8",
  "strict-transport-security": "max-age=2592000",
  "x-tfs-processid": "...",
  "activityid": "...",
  "x-tfs-session": "...",
  "x-vss-e2eid": "...",
  "x-vss-senderdeploymentid": "...",
  "x-frame-options": "SAMEORIGIN",
  "x-cache": "CONFIG_NOCACHE",
  "x-msedge-ref": "Ref A: ... Ref B: ... Ref C: 2021-08-05T01:33:28Z",
  "date": "Thu, 05 Aug 2021 01:33:28 GMT"
}
###### End Diagnostic HTTP information ######
Retry limit has been reached for chunk at offset 0 to https://pipelines.actions.githubusercontent.com/.../_apis/resources/Containers/...?itemPath=...
Warning: Aborting upload for ... due to failure
Error: aborting artifact upload
Total size of all the files uploaded is 329038 bytes
Finished uploading artifact .... Reported size is 329038 bytes. There were 1 items that failed to upload
Error: An error was encountered when uploading .... There were 1 items that failed to upload.

(Redacted for privacy.)

[mattzech]

I'm seeing a similar thing on the latest version (I'm assuming, since we're using actions/checkout@v2:

With the provided path, there will be 10 files uploaded
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #1. Waiting for 5548.578148597127 milliseconds before continuing the upload at offset 0
An error has been caught http-client index 1, retrying the upload
Error: Client has already been disposed.
    at HttpClient.request (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:5694:19)
    at HttpClient.sendStream (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:5655:21)
    at UploadHttpClient.<anonymous> (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:7104:37)
    at Generator.next (<anonymous>)
    at .../_work/_actions/actions/upload-artifact/v2/dist/index.js:6834:71
    at new Promise (<anonymous>)
    at module.exports.608.__awaiter (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:6830:12)
    at uploadChunkRequest (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:7102:46)
    at UploadHttpClient.<anonymous> (.../_work/_actions/actions/upload-artifact/v2/dist/index.js:7139:38)
    at Generator.next (<anonymous>)
    Exponential backoff for retry #1. Waiting for 5773.927291417539 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #2. Waiting for 9362.643214894475 milliseconds before continuing the upload at offset 0
Finished backoff for retry #1, continuing with upload
Total file count: 10 ---- Processed file #9 (90.0%)
Finished backoff for retry #2, continuing with upload
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #3. Waiting for 14276.414696905516 milliseconds before continuing the upload at offset 0
Total file count: 10 ---- Processed file #9 (90.0%)
Total file count: 10 ---- Processed file #9 (90.0%)
Finished backoff for retry #3, continuing with upload
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #4. Waiting for 20561.212693027403 milliseconds before continuing the upload at offset 0
Total file count: 10 ---- Processed file #9 (90.0%)
Total file count: 10 ---- Processed file #9 (90.0%)
Finished backoff for retry #4, continuing with upload
A 500 status code has been received, will attempt to retry the upload
Exponential backoff for retry #5. Waiting for 31215.480894221528 milliseconds before continuing the upload at offset 0
Total file count: 10 ---- Processed file #9 (90.0%)
Total file count: 10 ---- Processed file #9 (90.0%)
Total file count: 10 ---- Processed file #9 (90.0%)
Finished backoff for retry #5, continuing with upload
A 500 status code has been received, will attempt to retry the upload
##### Begin Diagnostic HTTP information #####
Status Code: 500
Status Message: Internal Server Error
Header Information: {
  "cache-control": "no-store,no-cache",
  "pragma": "no-cache",
  "content-length": "328",
  "content-type": "application/json; charset=utf-8",
  "strict-transport-security": "max-age=2592000",
  "x-tfs-processid": "...",
  "activityid": "...",
  "x-tfs-session": "...",
  "x-vss-e2eid": "...",
  "x-vss-senderdeploymentid": "...",
  "x-frame-options": "SAMEORIGIN",
  "x-cache": "CONFIG_NOCACHE",
  "x-msedge-ref": "Ref A: ... Ref B: ... Ref C: 2021-08-05T01:33:28Z",
  "date": "Thu, 05 Aug 2021 01:33:28 GMT"
}
###### End Diagnostic HTTP information ######
Retry limit has been reached for chunk at offset 0 to https://pipelines.actions.githubusercontent.com/.../_apis/resources/Containers/...?itemPath=...
Warning: Aborting upload for ... due to failure
Error: aborting artifact upload
Total size of all the files uploaded is 329038 bytes
Finished uploading artifact .... Reported size is 329038 bytes. There were 1 items that failed to upload
Error: An error was encountered when uploading .... There were 1 items that failed to upload.

(Redacted for privacy.)

@joshuapinter did you ever get this solved? I am facing the exact same issue