New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: argo cp command to download artifacts #695

Open
jessesuen opened this Issue Jan 19, 2018 · 0 comments

Comments

Projects
None yet
2 participants
@jessesuen
Contributor

jessesuen commented Jan 19, 2018

This proposal is for an argo CLI command to download artifacts from a workflow step. Given a workflow result like:

$ argo get artifact-passing-jg2k8 -o wide
Name:             artifact-passing-jg2k8
Namespace:        default
ServiceAccount:   default
Status:           Succeeded
Created:          Thu Jan 18 15:02:42 -0800 (9 hours ago)
Started:          Thu Jan 18 15:02:42 -0800 (9 hours ago)
Finished:         Thu Jan 18 15:03:31 -0800 (9 hours ago)
Duration:         49 seconds

STEP                       PODNAME                            DURATION  ARTIFACTS  MESSAGE
 ✔ artifact-passing-jg2k8
 ├---✔ generate-artifact   artifact-passing-jg2k8-2089725168  25s       hello-art
 └---✔ consume-artifact    artifact-passing-jg2k8-2707584000  23s

One could do something like:

$ argo cp artifact-passing-jg2k8-2089725168:hello-art /tmp/

The workflow YAML already has most of the information required to download the artifact (e.g. bucket name and key) as seen in the node status:

    artifact-passing-jg2k8-2089725168:
      finishedAt: 2018-01-18T23:03:07Z
      id: artifact-passing-jg2k8-2089725168
      name: artifact-passing-jg2k8[0].generate-artifact
      outputs:
        artifacts:
        - name: hello-art
          path: /tmp/hello_world.txt
          s3:
            accessKeySecret:
              key: accesskey
              name: argo-artifacts-minio-user
            bucket: my-bucket
            endpoint: argo-artifacts-minio-svc.default:9000
            insecure: true
            key: artifact-passing-jg2k8/artifact-passing-jg2k8-2089725168/hello-art.tgz
            secretKeySecret:
              key: secretkey
              name: argo-artifacts-minio-user

The issues here are that:

  1. the artifact repository may not directly accessible from the client (e.g. internal minio server)
  2. the secrets may not be accessible

However, we could at least support the use case for GCS and AWS, where the repository will be directly accessible. Regarding bucket secrets, we can either lookup the secrets using the k8s go-client, or rely on the .aws config/credentials in the user's home directory for the download.

@jessesuen jessesuen added the proposal label Jan 19, 2018

@edlee2121 edlee2121 added this to the V2.3 milestone Aug 29, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment