Question: What API is used to upload artifacts?

[ianfixes]

Apologies if this is not the proper place to ask; this repo is the closest I could get to what I was looking for.

I am curious about uploading artifacts from my own custom Docker action.
The API does not list an upload function: https://docs.github.com/en/rest/reference/actions#artifacts

I tried to dig into how this action performs the upload step and it rapidly becomes a bit lower-level than I was expecting to see:

• upload-artifact/src/upload-artifact.ts

  Line 54 in ea3d524

  const uploadResponse = await artifactClient.uploadArtifact(

• upload-artifact/dist/index.js

  Line 3815 in ea3d524

  uploadArtifact(name, files, rootDirectory, options) {

• upload-artifact/dist/index.js

  Line 6908 in ea3d524

  uploadArtifactToFileContainer(uploadUrl, filesToUpload, options) {

Is this hitting an API that's not available to the public? I only ask because I don't see anything related to passing in a GITHUB_TOKEN like other actions (e.g. super-linter).

[konradpabjan]

You can find all the APIs here: https://github.com/actions/toolkit/tree/main/packages/artifact

All the core APIs interactions are in our @actions/artifact NPM package. We split this so that other action authors could use the NPM package to more easily integrate artifact upload/download into their actions.

See also: https://github.com/actions/upload-artifact#additional-documentation

[ianfixes]

Hi, I'm aware of the NPM-based API, but I am referring more to the REST API that (I assume) underlies it. What HTTP endpoints are hit in order to upload artifacts, and where are they documented?

[andersson09]

Would also like to know this. Is there a rest API in which developers can interact with?

[boemekeld]

Any news about an API endpoints?

[lmussier]

Sorry to ping you @konradpabjan, just want to make to sure you get the REST API question (I'm searching for the same thing to work around actions/runner#1491)

[andersson09]

I gave up and started using GitHub packages for artifacts. They are billed the same afaik.

[lmussier]

@andersson09 are we able to deal with let's say zip files or plain binary ? (sounds a good way to work around if we are able to do this kind of thing)

[andersson09]

Yep. Currently I'm zipping and renaming them to .nupkg and treating them as a nuget package. I Would assume there are other ways too though.

[konradpabjan]

What HTTP endpoints are hit in order to upload artifacts, and where are they documented?

They're not documented anywhere. You'd have to effectively dig through the NPM package to figure all the calls that are made and it's not very intuitive.

There is a HTTP POST call to create the artifact container:
https://github.com/actions/toolkit/blob/03eca1b0c77c26d3eaa0a4e9c1583d6e32b87f6f/packages/artifact/src/internal/upload-http-client.ts#L101

Then multiple PUT calls to actually upload the content:
https://github.com/actions/toolkit/blob/03eca1b0c77c26d3eaa0a4e9c1583d6e32b87f6f/packages/artifact/src/internal/upload-http-client.ts#L421

Followed by a PATCH call to finalize the artifact upload:
https://github.com/actions/toolkit/blob/03eca1b0c77c26d3eaa0a4e9c1583d6e32b87f6f/packages/artifact/src/internal/upload-http-client.ts#L542

A bit more information here:
https://github.com/actions/toolkit/blob/main/packages/artifact/docs/implementation-details.md#uploadcompression-flow

They're effectively "internal" APIs that don't hit api.github.com but some of our backend services. Anyone can hit them but we're deliberately not advertising this and these APIs are not documented on https://docs.github.com/en/rest/reference/actions#artifacts because it works with a special token the runner has as opposed to GITHUB_TOKEN or a PAT. Outside of the context of a run you can't upload artifacts.

The upload process is unnecessarily complex for what it is (because we proxy everything through our backend before uploading to blob storage). We're already working on a pretty major overhaul (think upload-artifact v4) that will just give the runner or action a simple SAS URL to directly upload to blob storage so things will be faster and much easier to interact with

[ianfixes]

I'm grateful for all the attention this has received, and the fact that an "upload URL" is in the works is excellent news!

The restriction on uploads (being allowed only from a runner and in the context of a run) is fine for my use case, so I'll look forward to v4.

[edumserrano]

@konradpabjan thank you a lot for describing the flow and pointing out in the code where this is happening. Thanks to you I've been able to implement a GitHub action to share data across workflow jobs. When I started working on my action, I had no idea how I was going to handle uploading artifacts or downloading artifacts from the current workflow and I couldn't have done it without your guidance above.

I still had to dig a bit back and forth on the repo to understand all the request headers/bodies and response headers/bodies involved in the upload artifact and download artifact from the current workflow API operations. For those that are also wondering what is required take a look at the snapshot files below. These compliment @konradpabjan explanation:

• snapshot for the HTTP request/responses done when uploading a file: note that in the second request captured in this snapshot, the request content and respective headers are not shown because it's a byte array of the data to upload. In addition to specifying the request content as a byte array make sure you set the Content-Type and Content-Range headers.
• snapshot for the HTTP request/responses done when download a file from the same workflow: to download an artifact that was uploaded in the same workflow you cannot use the download artifact API. But following @konradpabjan directions for upload I also found how the download can be done.

If you don't feel comfortable exploring typescript code from the links that @konradpabjan provided, you can check these for equivalent C# versions:

• API operations to upload an artifact and configuration for the HTTP client.
• API operations to download an artifact from the current workflow and configuration for the HTTP client.

Hope this helps out fellow developers until we get an official API for this.

Note

The links from the C# version I provided are enough if you want to upload a single file. If you want to upload multiple files one by one or in a concurrent manner, then you will have to implement that yourself. Looking at the examples from @konradpabjan will show you how it's done in the GitHub Actions Toolkit.

[asottile]

ok tried to simplify this down further and made a few discoveries...

composite actions (for whatever reason) don't seem to receive the environment variables necessary to do this -- no problem it's easy to wrap some other language in a little bit of js -- here's (for example) a wrapper which calls main.sh:

import child_process from 'child_process';
import path from 'path';
import url from 'url';

const __dirname = path.dirname(url.fileURLToPath(import.meta.url));
const exe = path.resolve(__dirname, 'main.sh');
try {
    child_process.execFileSync(exe, {stdio: 'inherit'});
} catch (e) {
    process.exit(e.status);
}

then I've distilled down the "create an artifact and upload one file" to these curl commands:

#!/usr/bin/env bash
set -euxo pipefail

ARTIFACT_NAME='artifact-name'

HEADERS=(
  --header 'Accept: application/json;api-version=6.0-preview'
  --header "Authorization: Bearer $ACTIONS_RUNTIME_TOKEN"
)
ARTIFACT_BASE="${ACTIONS_RUNTIME_URL}_apis/pipelines/workflows/${GITHUB_RUN_ID}/artifacts?api-version=6.0-preview"

RESOURCE_URL="$(
  curl \
    -XPOST \
    --silent \
    --fail-with-body \
    "${HEADERS[@]}" \
    --header 'Content-Type: application/json' \
    --data '{"type": "actions_storage", "name": "'"${ARTIFACT_NAME}"'"}' \
    "$ARTIFACT_BASE" | jq --exit-status --raw-output .fileContainerResourceUrl
)"

curl \
  -XPUT \
  --silent \
  --fail-with-body \
  "${HEADERS[@]}" \
  --header 'Content-Type: application/octet-stream' \
  --header 'Content-Range: bytes 0-10/11' \
  --data 'hello world' \
  "${RESOURCE_URL}?itemPath=${ARTIFACT_NAME}/data.txt"

curl \
  -XPATCH \
  --silent \
  --fail-with-body \
  "${HEADERS[@]}" \
  --header 'Content-Type: application/json' \
  --data '{"size": 11}' \
  "${ARTIFACT_BASE}&artifactName=${ARTIFACT_NAME}"

some important things I missed and spent a bunch of time on:

• the Content-Type headers for the json requests are definitely needed
• artifacts must be uploaded to a "subdir" of the artifact name (in my example the artifact name is artifact-name so the files must be artifact-name/foo.txt)
• the content-range header doesn't understand */* unfortunately so you have to be explicit about the size there

[clayellis]

When I dump the env like so:

jobs:
  dump-env:
    runs-on: macos-latest
    steps:
    - run: env

Neither ACTIONS_RUNTIME_TOKEN nor ACTIONS_RUNTIME_URL appear in the list of environment variables. Does the step have to use a published action in order for those environment variables to be present?

EDIT: Just found this (Emphasis mine):

It is not possible to run end-to-end tests for artifacts as part of a PR in this repo because certain env variables such as ACTIONS_RUNTIME_URL are only available from the context of an action as opposed to a shell script.

[asottile]

there are three types of github actions: node, docker, and composite

[edumserrano]

@clayellis I also tried to do what you did before I developed my GitHub action that uploads/downloads artifacts and therefore I had to be able to access ACTIONS_RUNTIME_URL and ACTIONS_RUNTIME_TOKEN.

At first I was also puzzled by the fact I couldn't list those env variables and eventually found similar info to what you linked. I just wanted to confirm that indeed those env variables will be available to your action, regardless of the type, when the action is being executed.

Below is an extract of the log from running my action:

Run edumserrano/share-jobs-data@v1
  with:
   ...
/usr/bin/docker run --name ghcrioedumserranosharejobsdatav1_edd336 --label c9a4a5 --workdir /github/workspace --rm -e 
"INPUT_COMMAND" -e "INPUT_ARTIFACT-NAME" -e "INPUT_DATA" -e "INPUT_DATA-FILENAME" -e "INPUT_OUTPUT" -e 
"INPUT_AUTH-TOKEN" -e "INPUT_REPO" -e "INPUT_RUN-ID" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" 
-e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" 
-e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" 
-e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" 
-e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" 
-e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" 
-e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" 
-e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" 
-e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" 
-e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" 
-e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" 
-e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" 
-v "/home/runner/work/_temp/_github_home":"/github/home" 
-v "/home/runner/work/_temp/_github_workflow":"/github/workflow" 
-v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" 
-v "/home/runner/work/share-jobs-data/share-jobs-data":"/github/workspace" 
ghcr.io/edumserrano/share-jobs-data:v1  ...

My action is a docker action and the part I want to call your attention to are the env variables passed into the action. Note that both of the env variables you mentioned are passed to my action via -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN".

In the code of my app that is running inside the docker action I can access then access them as expected:

internal sealed record GitHubEnvironment : IGitHubEnvironment
{
    public string GitHubActionRuntimeToken { get; } = Environment.GetEnvironmentVariable("ACTIONS_RUNTIME_TOKEN") ?? string.Empty;

    public string GitHubActionRuntimeUrl { get; } = Environment.GetEnvironmentVariable("ACTIONS_RUNTIME_URL") ?? string.Empty;

    ...
}

[clayellis]

For anyone who comes across this in the future looking for yet another example of how to hit these APIs, here's a CICD action written in Swift. This action doesn't follow the exact logic that the official action uses in its upload compression flow but was sufficient for my needs.