Skip to content

Commit

Permalink
Add artifact-id output to v4-beta (#448)
Browse files Browse the repository at this point in the history
* Add artifact-id to outputs

* ncc

---------

Co-authored-by: Rob Herley <robherley@github.com>
  • Loading branch information
konradpabjan and robherley committed Oct 26, 2023
1 parent f87a408 commit aa5cae1
Show file tree
Hide file tree
Showing 4 changed files with 72 additions and 104 deletions.
153 changes: 61 additions & 92 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,18 @@
# Upload-Artifact v3
# Upload-Artifact v4 beta

❗ Not publicly available. If you try to use this version then it will fail. Available only internally at GitHub while in development. Stay tuned for public announcements soon about broader availability❗

This uploads artifacts from your workflow allowing you to share data between jobs and store data once a workflow is complete.

See also [download-artifact](https://github.com/actions/download-artifact).

# What's new

- Easier upload
- Specify a wildcard pattern
- Specify an individual file
- Specify a directory (previously you were limited to only this option)
- Multi path upload
- Use a combination of individual files, wildcards or directories
- Support for excluding certain files
- Upload an artifact without providing a name
- Fix for artifact uploads sometimes not working with containers
- Proxy support out of the box
- Port entire action to typescript from a runner plugin so it is easier to collaborate and accept contributions
🚧 Under construction 🚧

Big changes coming...

Refer [here](https://github.com/actions/upload-artifact/tree/releases/v1) for the previous version
Refer [here](https://github.com/actions/upload-artifact/tree/releases/v3) for the previous version

# Usage

Expand All @@ -34,7 +28,7 @@ steps:

- run: echo hello > path/to/artifact/world.txt

- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4-beta
with:
name: my-artifact
path: path/to/artifact/world.txt
Expand All @@ -43,7 +37,7 @@ steps:
### Upload an Entire Directory

```yaml
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4-beta
with:
name: my-artifact
path: path/to/artifact/ # or path/to/artifact
Expand All @@ -52,7 +46,7 @@ steps:
### Upload using a Wildcard Pattern

```yaml
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4-beta
with:
name: my-artifact
path: path/**/[abc]rtifac?/*
Expand All @@ -61,7 +55,7 @@ steps:
### Upload using Multiple Paths and Exclusions

```yaml
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4-beta
with:
name: my-artifact
path: |
Expand Down Expand Up @@ -97,7 +91,7 @@ The [@actions/artifact](https://github.com/actions/toolkit/tree/main/packages/ar
If a path (or paths), result in no files being found for the artifact, the action will succeed but print out a warning. In certain scenarios it may be desirable to fail the action or suppress the warning. The `if-no-files-found` option allows you to customize the behavior of the action if no files are found:

```yaml
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4-beta
with:
name: my-artifact
path: path/to/artifact/
Expand All @@ -109,7 +103,7 @@ If a path (or paths), result in no files being found for the artifact, the actio
To upload artifacts only when the previous step of a job failed, use [`if: failure()`](https://help.github.com/en/articles/contexts-and-expression-syntax-for-github-actions#job-status-check-functions):

```yaml
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4-beta
if: failure()
with:
name: my-artifact
Expand All @@ -121,7 +115,7 @@ To upload artifacts only when the previous step of a job failed, use [`if: failu
You can upload an artifact without specifying a name

```yaml
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4-beta
with:
path: path/to/artifact/world.txt
```
Expand All @@ -130,51 +124,21 @@ If not provided, `artifact` will be used as the default name which will manifest

### Uploading to the same artifact

With the following example, the available artifact (named `artifact` by default if no name is provided) would contain both `world.txt` (`hello`) and `extra-file.txt` (`howdy`):
Unlike earlier versions of `upload-artifact`, uploading to the same artifact via multiple jobs is _not_ supported with `v4`.

```yaml
- run: echo hi > world.txt
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4-beta
with:
path: world.txt

- run: echo howdy > extra-file.txt
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4-beta
with:
path: extra-file.txt

- run: echo hello > world.txt
- uses: actions/upload-artifact@v3
with:
path: world.txt
```

Each artifact behaves as a file share. Uploading to the same artifact multiple times in the same workflow can overwrite and append already uploaded files:

```yaml
strategy:
matrix:
node-version: [8.x, 10.x, 12.x, 13.x]
steps:
- name: Create a file
run: echo ${{ matrix.node-version }} > my_file.txt
- name: Accidentally upload to the same artifact via multiple jobs
uses: actions/upload-artifact@v3
with:
name: my-artifact
path: ${{ github.workspace }}
```

> **_Warning:_** Be careful when uploading to the same artifact via multiple jobs as artifacts may become corrupted. When uploading a file with an identical name and path in multiple jobs, uploads may fail with 503 errors due to conflicting uploads happening at the same time. Ensure uploads to identical locations to not interfere with each other.
In the above example, four jobs will upload four different files to the same artifact but there will only be one file available when `my-artifact` is downloaded. Each job overwrites what was previously uploaded. To ensure that jobs don't overwrite existing artifacts, use a different name per job:

```yaml
uses: actions/upload-artifact@v3
with:
name: my-artifact ${{ matrix.node-version }}
path: ${{ github.workspace }}
```
Artifact names must be unique since each created artifact is idempotent so multiple jobs cannot modify the same artifact.

### Environment Variables and Tilde Expansion

Expand All @@ -184,9 +148,9 @@ You can use `~` in the path input as a substitute for `$HOME`. Basic tilde expan
- run: |
mkdir -p ~/new/artifact
echo hello > ~/new/artifact/world.txt
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4-beta
with:
name: Artifacts-V3
name: Artifacts-V4-beta
path: ~/new/**/*
```

Expand All @@ -199,7 +163,7 @@ Environment variables along with context expressions can also be used for input.
- run: |
mkdir -p ${{ github.workspace }}/artifact
echo hello > ${{ github.workspace }}/artifact/world.txt
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4-beta
with:
name: ${{ env.name }}-name
path: ${{ github.workspace }}/artifact/**/*
Expand All @@ -213,7 +177,7 @@ For environment variables created in other steps, make sure to use the `env` exp
mkdir testing
echo "This is a file to upload" > testing/file.txt
echo "artifactPath=testing/file.txt" >> $GITHUB_ENV
- uses: actions/upload-artifact@v3
- uses: actions/upload-artifact@v4-beta
with:
name: artifact
path: ${{ env.artifactPath }} # this will resolve to testing/file.txt at runtime
Expand All @@ -228,7 +192,7 @@ Artifacts are retained for 90 days by default. You can specify a shorter retenti
run: echo "I won't live long" > my_file.txt

- name: Upload Artifact
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4-beta
with:
name: my-artifact
path: my_file.txt
Expand All @@ -237,50 +201,55 @@ Artifacts are retained for 90 days by default. You can specify a shorter retenti

The retention period must be between 1 and 90 inclusive. For more information see [artifact and log retention policies](https://docs.github.com/en/free-pro-team@latest/actions/reference/usage-limits-billing-and-administration#artifact-and-log-retention-policy).

## Where does the upload go?

At the bottom of the workflow summary page, there is a dedicated section for artifacts. Here's a screenshot of something you might see:
## Outputs

<img src="https://user-images.githubusercontent.com/16109154/103645952-223c6880-4f59-11eb-8268-8dca6937b5f9.png" width="700" height="300">

There is a trashcan icon that can be used to delete the artifact. This icon will only appear for users who have write permissions to the repository.
If an artifact upload is successful then an `artifact-id` output is available. This ID is a unique identifier that can be used with [Artifact REST APIs](https://docs.github.com/en/rest/actions/artifacts).

The size of the artifact is denoted in bytes. The displayed artifact size denotes the raw uploaded artifact size (the sum of all the individual files uploaded during the workflow run for the artifact), not the compressed size. When you click to download an artifact from the summary page, a compressed zip is created with all the contents of the artifact and the size of the zip that you download may differ significantly from the displayed size. Billing is based on the raw uploaded size and not the size of the zip.
### Example output between steps

# Limitations

### Zipped Artifact Downloads

During a workflow run, files are uploaded and downloaded individually using the `upload-artifact` and `download-artifact` actions. However, when a workflow run finishes and an artifact is downloaded from either the UI or through the [download api](https://developer.github.com/v3/actions/artifacts/#download-an-artifact), a zip is dynamically created with all the file contents that were uploaded. There is currently no way to download artifacts after a workflow run finishes in a format other than a zip or to download artifact contents individually. One of the consequences of this limitation is that if a zip is uploaded during a workflow run and then downloaded from the UI, there will be a double zip created.

### Permission Loss

:exclamation: File permissions are not maintained during artifact upload :exclamation: For example, if you make a file executable using `chmod` and then upload that file, post-download the file is no longer guaranteed to be set as an executable.

### Case Insensitive Uploads
```yml
- uses: actions/upload-artifact@v4-beta
id: artifact-upload-step
with:
name: my-artifact
path: path/to/artifact/content/

:exclamation: File uploads are case insensitive :exclamation: If you upload `A.txt` and `a.txt` with the same root path, only a single file will be saved and available during download.
- name: Output artifact ID
run: echo 'Artifact ID is ${{ steps.artifact-upload-step.outputs.artifact-id }}'
```

### Maintaining file permissions and case sensitive files
### Example output between jobs

If file permissions and case sensitivity are required, you can `tar` all of your files together before artifact upload. Post download, the `tar` file will maintain file permissions and case sensitivity:
```yml
jobs:
job1:
runs-on: ubuntu-latest
outputs:
output1: ${{ steps.my-artifact.outputs.artifact-id }}
steps:
- uses: actions/upload-artifact@v4-beta
id: artifact-upload-step
with:
name: my-artifact
path: path/to/artifact/content/
job2:
runs-on: ubuntu-latest
needs: job1
steps:
- env:
OUTPUT1: ${{needs.job1.outputs.output1}}
run: echo "Artifact ID from previous job is $OUTPUT1"
```

```yaml
- name: Tar files
run: tar -cvf my_files.tar /path/to/my/directory
## Where does the upload go?

- name: Upload Artifact
uses: actions/upload-artifact@v3
with:
name: my-artifact
path: my_files.tar
```
At the bottom of the workflow summary page, there is a dedicated section for artifacts. Here's a screenshot of something you might see:

### Too many uploads resulting in 429 responses
<img src="https://user-images.githubusercontent.com/16109154/103645952-223c6880-4f59-11eb-8268-8dca6937b5f9.png" width="700" height="300">

A very minute subset of users who upload a very very large amount of artifacts in a short period of time may see their uploads throttled or fail because of `Request was blocked due to exceeding usage of resource 'DBCPU' in namespace` or `Unable to copy file to server StatusCode=TooManyRequests`.
There is a trashcan icon that can be used to delete the artifact. This icon will only appear for users who have write permissions to the repository.

To reduce the chance of this happening, you can reduce the number of HTTP calls made during artifact upload by zipping or archiving the contents of your artifact before an upload starts. As an example, imagine an artifact with 1000 files (each 10 Kb in size). Without any modification, there would be around 1000 HTTP calls made to upload the artifact. If you zip or archive the artifact beforehand, the number of HTTP calls can be dropped to single digit territory. Measures like this will significantly speed up your upload and prevent uploads from being throttled or in some cases fail.
The size of the artifact is denoted in bytes. The displayed artifact size denotes the size of the zip that `upload-artifact` creates during upload.

## Additional Documentation

Expand Down
8 changes: 7 additions & 1 deletion action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,12 @@ inputs:
Minimum 1 day.
Maximum 90 days unless changed from the repository settings page.
outputs:
artifact-id:
description: >
A unique identifier for the artifact that was just uploaded. Empty if artifact upload failed.
This ID can be used as input to other APIs to download, delete or get more information about an artifact: https://docs.github.com/en/rest/actions/artifacts
runs:
using: 'node16'
using: 'node20'
main: 'dist/index.js'
6 changes: 2 additions & 4 deletions dist/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -121406,9 +121406,6 @@ function run() {
const s = searchResult.filesToUpload.length === 1 ? '' : 's';
core.info(`With the provided path, there will be ${searchResult.filesToUpload.length} file${s} uploaded`);
core.debug(`Root artifact directory is ${searchResult.rootDirectory}`);
if (searchResult.filesToUpload.length > 10000) {
core.warning(`There are over 10,000 files in this artifact, consider creating an archive before upload to improve the upload performance.`);
}
const artifactClient = (0, artifact_1.create)();
const options = {};
if (inputs.retentionDays) {
Expand All @@ -121419,7 +121416,8 @@ function run() {
core.setFailed(`An error was encountered when uploading ${inputs.artifactName}.`);
}
else {
core.info(`Artifact ${inputs.artifactName} has been successfully uploaded! Final size is ${uploadResponse.size} bytes. Artifact ID is ${uploadResponse.id}}`);
core.info(`Artifact ${inputs.artifactName} has been successfully uploaded! Final size is ${uploadResponse.size} bytes. Artifact ID is ${uploadResponse.id}`);
core.setOutput('artifact-id', uploadResponse.id);
}
}
}
Expand Down
9 changes: 2 additions & 7 deletions src/upload-artifact.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,6 @@ async function run(): Promise<void> {
)
core.debug(`Root artifact directory is ${searchResult.rootDirectory}`)

if (searchResult.filesToUpload.length > 10000) {
core.warning(
`There are over 10,000 files in this artifact, consider creating an archive before upload to improve the upload performance.`
)
}

const artifactClient = create()
const options: UploadOptions = {}
if (inputs.retentionDays) {
Expand All @@ -62,8 +56,9 @@ async function run(): Promise<void> {
)
} else {
core.info(
`Artifact ${inputs.artifactName} has been successfully uploaded! Final size is ${uploadResponse.size} bytes. Artifact ID is ${uploadResponse.id}}`
`Artifact ${inputs.artifactName} has been successfully uploaded! Final size is ${uploadResponse.size} bytes. Artifact ID is ${uploadResponse.id}`
)
core.setOutput('artifact-id', uploadResponse.id)
}
}
} catch (error) {
Expand Down

0 comments on commit aa5cae1

Please sign in to comment.