Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] Artifact download fails with Unable to download artifact(s): Unable to download and extract artifact: Artifact download failed after 5 retries. #367

Open
fabn opened this issue Dec 6, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@fabn
Copy link

fabn commented Dec 6, 2024

What happened?

I'm using this action (actually called by this one) to process the outputs of a matrix and starting today it fails with these logs (partial)

Artifact download completed successfully.
##[debug]Failed to download artifact after 1 retries due to Not a valid zip file. Retrying in 5 seconds...
Artifact download completed successfully.
##[debug]Failed to download artifact after 2 retries due to Not a valid zip file. Retrying in 5 seconds...
##[debug]Failed to download artifact after 2 retries due to Not a valid zip file. Retrying in 5 seconds...
##[debug]Failed to download artifact after 3 retries due to Not a valid zip file. Retrying in 5 seconds...
##[debug]Failed to download artifact after 3 retries due to Not a valid zip file. Retrying in 5 seconds...
##[debug]Failed to download artifact after 4 retries due to Not a valid zip file. Retrying in 5 seconds...
##[debug]Failed to download artifact after 4 retries due to Not a valid zip file. Retrying in 5 seconds...
##[debug]Failed to download artifact after 5 retries due to Not a valid zip file. Retrying in 5 seconds...
##[debug]Failed to download artifact after 5 retries due to Not a valid zip file. Retrying in 5 seconds...
Error: Unable to download artifact(s): Unable to download and extract artifact: Artifact download failed after 5 retries.
##[debug]response.message: Artifact download failed: Blob storage chunk did not respond in 30000ms
##[debug]response.message: Artifact download failed: Blob storage chunk did not respond in 30000ms
##[debug]response.message: Artifact download failed: Blob storage chunk did not respond in 30000ms
##[debug]response.message: Artifact download failed: Blob storage chunk did not respond in 30000ms
##[debug]response.message: Artifact download failed: Blob storage chunk did not respond in 30000ms
##[debug]response.message: Artifact download failed: Blob storage chunk did not respond in 30000ms
##[debug]response.message: Artifact download failed: Blob storage chunk did not respond in 30000ms
##[debug]response.message: Artifact download failed: Blob storage chunk did not respond in 30000ms
##[debug]response.message: Artifact download failed: Blob storage chunk did not respond in 30000ms
##[debug]response.message: Artifact download failed: Blob storage chunk did not respond in 30000ms

It needs to download 4 to 6 small artifacts (less than 1KB). I used the same configuration in other workflows (matrix-write + matrix-read) and I never encountered this.

What did you expect to happen?

That all artifacts are downloaded with no issues

How can we reproduce it?

Here's an extract of the workflow that is causing the issue:

# Main pipeline run when any commit is pushed in default branch
name: Main Pipeline

on:
  push:

jobs:
  build:
    name: Build docker images
    uses: ./.github/workflows/build.yml
    secrets: inherit
    strategy:
      fail-fast: false
      matrix:
        include:
          - ecr_repository: alert
          - ecr_repository: operations
          - ecr_repository: warehouse
    with:
      ecr_repository: ${{ matrix.ecr_repository }}

  # Last job of reusable workflow is, not called in this one.
  write-matrix-item:
    name: Write matrix item
    runs-on: ubuntu-latest
    needs: configuration
    steps:
      - name: Write Matrix
        uses: cloudposse/github-action-matrix-outputs-write@v1
        id: out
        with:
          matrix-step-name: docker-images
          matrix-key: ${{ needs.configuration.outputs.name }}
          outputs: |-
            content: ${{ toJSON(needs.configuration.outputs.components) }}

  config:
    name: Read matrix output
    needs: build
    runs-on: ubuntu-latest
    steps:
      # I added this to debug and it fails as well
      - uses: actions/download-artifact@v4
      - name: Display structure of downloaded files
        run: ls -R

      - uses: cloudposse/github-action-matrix-outputs-read@v1
        name: Read matrix
        id: read
        with:
          matrix-step-name: docker-images

Anything else we need to know?

If I download the artifacts using the GitHub UI they all works so I don't know what is happening.

I saw this thread https://github.com/orgs/community/discussions/81413 where a similar issue has already addressed but it's happening to me right now.

What version of the action are you using?

actions/download-artifact@v4

What are your runner environments?

linux

Are you on GitHub Enterprise Server? If so, what version?

No response

@fabn fabn added the bug Something isn't working label Dec 6, 2024
@crazy-max
Copy link

uses: ./.github/workflows/build.yml

What this reusable workflow looks like? I guess it uses either docker/build-push-action or docker/bake-action? If that's the case then this is probably related to #328 (comment)

And we got similar reports:

With our actions we are using the upload artifact API to upload the build export artifact and we are not using zip format but gzip one:

And the API always expect a zip artifact when downloading: https://github.com/actions/toolkit/blob/361a115e538ac6d8eb06cc47f3fcecce557d04c8/packages/artifact/src/internal/download/download-artifact.ts#L92

I opened a PR in the toolkit to skip files that don't have the expected content-type before extracting them:

But seems to stale 😞

Here are the debug logs of "Download artifacts" step with toolkit's changes:

Preparing to download the following artifacts:
- docker~test-docker-action~BGHOTQ.dockerbuild (ID: 1654731091, Size: 5567)
- bake-meta (ID: 1654730707, Size: 497)
##[debug]Artifact destination folder does not exist, creating: /home/runner/work/test-docker-action/test-docker-action/docker~test-docker-action~BGHOTQ.dockerbuild
##[debug]Artifact destination folder does not exist, creating: /home/runner/work/test-docker-action/test-docker-action/bake-meta
##[debug]Workflow Run Backend ID: 0cc20625-3561-4440-9048-70024d8ad258
##[debug]Workflow Job Run Backend ID: 937ea504-f21e-52f6-d164-c808765d698a
##[debug][Request] ListArtifacts https://results-receiver.actions.githubusercontent.com/twirp/github.actions.results.api.v1.ArtifactService/ListArtifacts
##[debug]Workflow Run Backend ID: 0cc20625-3561-4440-9048-70024d8ad258
##[debug]Workflow Job Run Backend ID: 937ea504-f21e-52f6-d164-c808765d698a
##[debug][Request] ListArtifacts https://results-receiver.actions.githubusercontent.com/twirp/github.actions.results.api.v1.ArtifactService/ListArtifacts
##[debug][Response] - 200
##[debug]Headers: {
##[debug]  "content-length": "282",
##[debug]  "content-type": "application/json",
##[debug]  "date": "Mon, 01 Jul 2024 09:36:54 GMT",
##[debug]  "x-github-backend": "Kubernetes",
##[debug]  "x-github-request-id": "E00A:2A89D3:1D45CA:25322C:668278B6"
##[debug]}
##[debug]Body: {
##[debug]  "artifacts": [
##[debug]    {
##[debug]      "workflow_run_backend_id": "0cc20625-3561-4440-9048-70024d8ad258",
##[debug]      "workflow_job_run_backend_id": "ca395085-040a-526b-2ce8-bdc85f692774",
##[debug]      "database_id": "1654731091",
##[debug]      "name": "docker~test-docker-action~BGHOTQ.dockerbuild",
##[debug]      "size": "5567",
##[debug]      "created_at": "2024-07-01T09:36:43Z"
##[debug]    }
##[debug]  ]
##[debug]}
##[debug][Request] GetSignedArtifactURL https://results-receiver.actions.githubusercontent.com/twirp/github.actions.results.api.v1.ArtifactService/GetSignedArtifactURL
##[debug][Response] - 200
##[debug]Headers: {
##[debug]  "content-length": "560",
##[debug]  "content-type": "application/json",
##[debug]  "date": "Mon, 01 Jul 2024 09:36:54 GMT",
##[debug]  "x-github-backend": "Kubernetes",
##[debug]  "x-github-request-id": "E00A:2A89D3:1D45DB:253240:668278B6"
##[debug]}
##[debug]Body: {
##[debug]  "signed_url": "https://productionresultssa10.blob.core.windows.net/actions-results/0cc20625-3561-4440-9048-70024d8ad258/workflow-job-run-ca395085-040a-526b-2ce8-bdc85f692774/artifacts/771ba7777401e8a24ea0b5dc7d95a3da79ee6e928254cc46f0f13e40846c0ab1.zip?se=2024-07-01T09%3A46%3A54Z&sig=B9B9ehrtCs66uPrB9jHHRXfHsr6n0g8hnLS1TuwBTtc%3D&ske=2024-07-01T19%3A00%3A39Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-07-01T07%3A00%3A39Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2023-11-03&sp=r&spr=https&sr=b&st=2024-07-01T09%3A36%3A49Z&sv=2023-11-03"
##[debug]}
Redirecting to blob download url: https://productionresultssa10.blob.core.windows.net/actions-results/0cc20625-3561-4440-9048-70024d8ad258/workflow-job-run-ca395085-040a-526b-2ce8-bdc85f692774/artifacts/771ba7777401e8a24ea0b5dc7d95a3da79ee6e928254cc46f0f13e40846c0ab1.zip
Starting download of artifact to: /home/runner/work/test-docker-action/test-docker-action/docker~test-docker-action~BGHOTQ.dockerbuild
##[debug][Response] - 200
##[debug]Headers: {
##[debug]  "content-length": "246",
##[debug]  "content-type": "application/json",
##[debug]  "date": "Mon, 01 Jul 2024 09:36:54 GMT",
##[debug]  "x-github-backend": "Kubernetes",
##[debug]  "x-github-request-id": "E00B:1FBE19:1D1E16:24FD99:668278B6"
##[debug]}
##[debug]Body: {
##[debug]  "artifacts": [
##[debug]    {
##[debug]      "workflow_run_backend_id": "0cc20625-3561-4440-9048-70024d8ad258",
##[debug]      "workflow_job_run_backend_id": "ca395085-040a-526b-2ce8-bdc85f692774",
##[debug]      "database_id": "1654730707",
##[debug]      "name": "bake-meta",
##[debug]      "size": "497",
##[debug]      "created_at": "2024-07-01T09:36:37Z"
##[debug]    }
##[debug]  ]
##[debug]}
##[debug][Request] GetSignedArtifactURL https://results-receiver.actions.githubusercontent.com/twirp/github.actions.results.api.v1.ArtifactService/GetSignedArtifactURL
##[debug][Response] - 200
##[debug]Headers: {
##[debug]  "content-length": "562",
##[debug]  "content-type": "application/json",
##[debug]  "date": "Mon, 01 Jul 2024 09:36:54 GMT",
##[debug]  "x-github-backend": "Kubernetes",
##[debug]  "x-github-request-id": "E00B:1FBE19:1D1E23:24FDAA:668278B6"
##[debug]}
##[debug]Body: {
##[debug]  "signed_url": "https://productionresultssa10.blob.core.windows.net/actions-results/0cc20625-3561-4440-9048-70024d8ad258/workflow-job-run-ca395085-040a-526b-2ce8-bdc85f692774/artifacts/906bf0728887597ba91b16b1778f41cea66aa49961106539f0c04e0b11d3abd5.zip?se=2024-07-01T09%3A46%3A54Z&sig=ArW2G%2BWxAGsJgNZB2X2kZhBt2RmgSbTSwR6atQG4Hwo%3D&ske=2024-07-01T20%3A16%3A40Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2024-07-01T08%3A16%3A40Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2023-11-03&sp=r&spr=https&sr=b&st=2024-07-01T09%3A36%3A49Z&sv=2023-11-03"
##[debug]}
Redirecting to blob download url: https://productionresultssa10.blob.core.windows.net/actions-results/0cc20625-3561-4440-9048-70024d8ad258/workflow-job-run-ca395085-040a-526b-2ce8-bdc85f692774/artifacts/906bf0728887597ba91b16b1778f41cea66aa49961106539f0c04e0b11d3abd5.zip
Starting download of artifact to: /home/runner/work/test-docker-action/test-docker-action/bake-meta
##[debug]response.message.headers: {"content-length":"5567","content-type":"application/gzip","content-md5":"s35vhwNK24ehlOSl2bTFUA==","last-modified":"Mon, 01 Jul 2024 09:36:43 GMT","accept-ranges":"bytes","etag":"\"0x8DC99B15171B60C\"","server":"Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0","x-ms-request-id":"e9c547ca-a01e-0012-649a-cb4aa8000000","x-ms-version":"2023-11-03","x-ms-creation-time":"Mon, 01 Jul 2024 09:36:43 GMT","x-ms-lease-status":"unlocked","x-ms-lease-state":"available","x-ms-blob-type":"BlockBlob","content-disposition":"attachment; filename=\"docker~test-docker-action~BGHOTQ.dockerbuild\"","x-ms-server-encrypted":"true","access-control-expose-headers":"x-ms-request-id,Server,x-ms-version,Content-Type,Last-Modified,ETag,x-ms-creation-time,Content-MD5,x-ms-lease-status,x-ms-lease-state,x-ms-blob-type,Content-Disposition,x-ms-server-encrypted,Accept-Ranges,Content-Length,Date,Transfer-Encoding","access-control-allow-origin":"*","date":"Mon, 01 Jul 2024 09:36:53 GMT"}
##[debug]Invalid content-type: application/gzip, skipping download
Artifact download completed successfully.
##[debug]response.message.headers: {"content-length":"497","content-type":"zip","last-modified":"Mon, 01 Jul 2024 09:36:37 GMT","accept-ranges":"bytes","etag":"\"0x8DC99B14DA08F62\"","server":"Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0","x-ms-request-id":"63899c09-b01e-0021-749a-cb1503000000","x-ms-version":"2023-11-03","x-ms-creation-time":"Mon, 01 Jul 2024 09:36:37 GMT","x-ms-lease-status":"unlocked","x-ms-lease-state":"available","x-ms-blob-type":"BlockBlob","x-ms-server-encrypted":"true","access-control-expose-headers":"x-ms-request-id,Server,x-ms-version,Content-Type,Last-Modified,ETag,x-ms-creation-time,x-ms-lease-status,x-ms-lease-state,x-ms-blob-type,x-ms-server-encrypted,Accept-Ranges,Content-Length,Date,Transfer-Encoding","access-control-allow-origin":"*","date":"Mon, 01 Jul 2024 09:36:54 GMT"}
(node:1468) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Artifact download completed successfully.
Total of 2 artifact(s) downloaded
Download artifact has finished successfully
##[debug]Node Action run completed with exit code 0
##[debug]Set output download-path = /home/runner/work/test-docker-action/test-docker-action
##[debug]Finishing: Download artifacts

We have two files download by "Download artifacts". After adding some logging on response headers we can see that the regular artifact uploaded with actions/upload-artifact@v4 has zip as content-type header but one uploaded by docker/build-push-action has application/gzip:

{
  "content-length": "5572",
  "content-type": "application/gzip",
  "content-md5": "yPIHPOPuYDEHs/vabwyt6A==",
  "last-modified": "Mon, 01 Jul 2024 09:22:34 GMT",
  "accept-ranges": "bytes",
  "etag": "\"0x8DC99AF5777FC1C\"",
  "server": "Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0",
  "x-ms-request-id": "b45769a4-301e-00cf-5498-cb2dab000000",
  "x-ms-version": "2023-11-03",
  "x-ms-creation-time": "Mon, 01 Jul 2024 09:22:34 GMT",
  "x-ms-lease-status": "unlocked",
  "x-ms-lease-state": "available",
  "x-ms-blob-type": "BlockBlob",
  "content-disposition": "attachment; filename=\"docker~test-docker-action~44M6YV.dockerbuild\"",
  "x-ms-server-encrypted": "true",
  "access-control-expose-headers": "x-ms-request-id,Server,x-ms-version,Content-Type,Last-Modified,ETag,x-ms-creation-time,Content-MD5,x-ms-lease-status,x-ms-lease-state,x-ms-blob-type,Content-Disposition,x-ms-server-encrypted,Accept-Ranges,Content-Length,Date,Transfer-Encoding",
  "access-control-allow-origin": "*",
  "date": "Mon, 01 Jul 2024 09:22:47 GMT"
}

I think this change in the toolkit should mitigate this issue by making sure we try to extract a valid zip file. cc @joshmgross @bethanyj28 @robherley @konradpabjan

@Memohalliwell
Copy link

Thanks

fruch added a commit to scylladb/scylla-cqlsh that referenced this issue Jan 20, 2025
seem like we are hit by a bug when trying to download
create by `docker/build-push-action` that we don't even need

filtering to the exact ones we do need, to avoid this issue

Ref: actions/download-artifact#367
@RobbieMcKinstry
Copy link

I'm experiencing this same error but I'm not working with Docker images. My artifact is a simple JSON file.

@brainstorm
Copy link

Same issue, my artifacts are just small binary blobs: https://github.com/umccr/gatk-bwamem-jni/actions/runs/13716508594/job/38362482726

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants