Transfer-Artifact v4

This uploads and downloads artifacts from your workflow to S3, allowing you to share data between jobs and store data once a workflow is complete.

This is based on this fork from the original Github action. See also the original actions, upload-artifact and download-artifact.

Usage

See action.yml

Upload an Individual File

steps:
  - uses: actions/checkout@v3

  - run: mkdir -p path/to/artifact

  - run: echo hello > path/to/artifact/world.txt

  - uses: NHSDigital/transfer-artifact@s3
    env:
      bucket: abcd-123456789-eu-west-2-my-S3-bucket
    with:
      name: my-folder
      direction: 'upload'
      path: path/to/artifact/world.txt

Upload an Entire Directory

- uses: NHSDigital/transfer-artifact@s3
  env:
    bucket: abcd-123456789-eu-west-2-my-S3-bucket
  with:
    name: my-folder
    direction: 'upload'
    path: path/to/artifact/ # or path/to/artifact

Upload using a Wildcard Pattern

- uses: NHSDigital/transfer-artifact@s3
  env:
    bucket: abcd-123456789-eu-west-2-my-S3-bucket
  with:
    name: my-folder
    direction: 'upload'
    path: path/**/[abc]rtifac?/*

Upload using Multiple Paths and Exclusions

- uses: NHSDigital/transfer-artifact@s3
  env:
    bucket: abcd-123456789-eu-west-2-my-S3-bucket
  with:
    name: my-folder
    direction: 'upload'
    path: |
      path/output/bin/
      path/output/test-results
      !path/**/*.tmp

For supported wildcards along with behavior and documentation, see @actions/glob which is used internally to search for files.

If a wildcard pattern is used, the path hierarchy will be preserved after the first wildcard pattern:

path/to/*/directory/foo?.txt =>
    ∟ path/to/some/directory/foo1.txt
    ∟ path/to/some/directory/foo2.txt
    ∟ path/to/other/directory/foo1.txt

would be flattened and uploaded as =>
    ∟ some/directory/foo1.txt
    ∟ some/directory/foo2.txt
    ∟ other/directory/foo1.txt

If multiple paths are provided as input, the least common ancestor of all the search paths will be used as the root directory of the artifact. Exclude paths do not affect the directory structure.

Relative and absolute file paths are both allowed. Relative paths are rooted against the current working directory. Paths that begin with a wildcard character should be quoted to avoid being interpreted as YAML aliases.

The @actions/artifact package is used internally to handle most of the logic around uploading an artifact. There is extra documentation around upload limitations and behavior in the toolkit repo that is worth checking out.

Customization if no files are found

If a path (or paths), result in no files being found for the artifact, the action will succeed but print out a warning. In certain scenarios it may be desirable to fail the action or suppress the warning. The if-no-files-found option allows you to customize the behavior of the action if no files are found:

- uses: NHSDigital/transfer-artifact@s3
  env:
    bucket: abcd-123456789-eu-west-2-my-S3-bucket
  with:
    name: my-folder
    direction: 'upload'
    path: path/to/artifact/
    if-no-files-found: error # 'warn' or 'ignore' are also available, defaults to `warn`

Conditional Artifact Upload

To upload artifacts only when the previous step of a job failed, use if: failure():

- uses: NHSDigital/transfer-artifact@s3
  env:
    bucket: abcd-123456789-eu-west-2-my-S3-bucket
  if: failure()
  with:
    name: my-folder
    direction: 'upload'
    path: path/to/artifact/

Uploading without an artifact name

You can upload an artifact without specifying a name

- uses: NHSDigital/transfer-artifact@s3
  env:
    bucket: abcd-123456789-eu-west-2-my-S3-bucket
  with:
    name: my-folder
    direction: 'upload'
    path: path/to/artifact/world.txt

If not provided, artifact will be used as the default name for the artifact, upload-artifacts will be the default name for the folder, and upload will be the direction of travel.

Uploading to the same artifact

With the following example, the available artifact (named artifact by default if no name is provided) would contain both world.txt (hello) and extra-file.txt (howdy):

- run: echo hi > world.txt
- uses: NHSDigital/transfer-artifact@s3
  env:
    bucket: abcd-123456789-eu-west-2-my-S3-bucket
  with:
    name: my-folder
    path: world.txt
    direction: 'upload'

- run: echo howdy > extra-file.txt
- uses: NHSDigital/transfer-artifact@s3
  env:
    bucket: abcd-123456789-eu-west-2-my-S3-bucket
  with:
    name: my-folder
    path: extra-file.txt
    direction: 'upload'

- run: echo hello > world.txt
- uses: NHSDigital/transfer-artifact@s3
  env:
    bucket: abcd-123456789-eu-west-2-my-S3-bucket
  with:
    name: my-folder
    path: world.txt
    direction: 'upload'

Each artifact behaves as a file share. Uploading to the same artifact multiple times in the same workflow can overwrite and append already uploaded files:

strategy:
  matrix:
    node-version: [8.x, 10.x, 12.x, 13.x]
steps:
  - name: Create a file
    run: echo ${{ github.run_number }} > my_file.txt
  - name: Accidentally upload to the same artifact via multiple jobs
    uses: NHSDigital/transfer-artifact@s3
    env:
      bucket: abcd-123456789-eu-west-2-my-S3-bucket
    with:
      name: my-folder
      direction: 'upload'
      path: ${{ github.workspace }}

Warning: Be careful when uploading to the same artifact via multiple jobs as artifacts may become corrupted. When uploading a file with an identical name and path in multiple jobs, uploads may fail with 503 errors due to conflicting uploads happening at the same time. Ensure uploads to identical locations to not interfere with each other.

In the above example, four jobs will upload four different files to the same artifact but there will only be one file available when my-artifact is downloaded. Each job overwrites what was previously uploaded. To ensure that jobs don't overwrite existing artifacts, use a different name per job:

uses: NHSDigital/transfer-artifact@s3
env:
  bucket: abcd-123456789-eu-west-2-my-S3-bucket
with:
  name: my-folder
  direction: 'upload'
  path: ${{ github.workspace }}

Environment Variables and Tilde Expansion

You can use ~ in the path input as a substitute for $HOME. Basic tilde expansion is supported:

  - run: |
      mkdir -p ~/new/artifact
      echo hello > ~/new/artifact/world.txt
  - uses: NHSDigital/transfer-artifact@s3
    env:
      bucket: abcd-123456789-eu-west-2-my-S3-bucket
    with:
      name: my-folder
      name: Artifacts-V3
      path: ~/new/**/*
      direction: 'upload'

Environment variables along with context expressions can also be used for input. For documentation see context and expression syntax:

steps:
  - run: |
      mkdir -p ${{ github.workspace }}/artifact
      echo hello > ${{ github.workspace }}/artifact/world.txt
  - uses: NHSDigital/transfer-artifact@s3
    env:
      bucket: abcd-123456789-eu-west-2-my-S3-bucket
    with:
      name: my-folder
      path: ${{ github.workspace }}/artifact/**/*#
      direction: 'upload'

For environment variables created in other steps, make sure to use the env expression syntax

    steps:
    - run: |
        mkdir testing
        echo "This is a file to upload" > testing/file.txt
        echo "artifactPath=testing/file.txt" >> $GITHUB_ENV
    - uses: NHSDigital/transfer-artifact@s3
      env:
        bucket: abcd-123456789-eu-west-2-my-S3-bucket
      with:
        name: my-folder
        name: artifact
        path: ${{ env.artifactPath }} # this will resolve to testing/file.txt at runtime
        direction: 'upload'

Retention Period

Artifacts are retained for 90 days by default. You can specify a shorter retention period using the retention-days input:

- name: Create a file
  run: echo "I won't live long" > my_file.txt

- name: Upload Artifact
  uses: NHSDigital/transfer-artifact@s3
  env:
    bucket: abcd-123456789-eu-west-2-my-S3-bucket
  with:
    name: my-folder
    path: my_file.txt
    retention-days: 5
    direction: 'upload'

The retention period must be between 1 and 90 inclusive. For more information see artifact and log retention policies.

Where does the upload go?

Artifacts are uploaded to the specified S3 bucket, into a folder called folder-name. The artifacts for each pipeline are put in a subfolder named ${{ github.run_number }}-folder-name

Downloading all files

steps:
  - uses: actions/checkout@v3

  - uses: NHSDigital/transfer-artifact@s3
    env:
      bucket: abcd-123456789-eu-west-2-my-S3-bucket
    with:
      name: my-folder/path/to/artifact
      direction: 'download'
      path: local/download/folder

This will download every object in the S3 bucket which matches the my-folder/my-artifact/path/to/artifact name prefix into a folder called local/download/folder.

Downloading one file

steps:
  - uses: actions/checkout@v3

  - uses: NHSDigital/transfer-artifact@s3
    env:
      bucket: abcd-123456789-eu-west-2-my-S3-bucket
    with:
      name: my-folder/path/to/artifact/word.txt
      direction: 'download'
      path: local/download/folder

This will download only the file my-artifact/path/to/artifact/word.txt into a folder called local/download/folder.