Proposal: task images should be specified as resources

[vito]

The goal of this proposal is to solve the following problems:

1. Implement more explicit caching semantics, not subject to Garden Linux's image caching, which should fix the issue of running out of disk as images are fetched over time.
2. Support images that live in private registries or require credentials, without having to reconfigure the workers to support insecure registries.
3. Fix a sort of hole in the abstraction wherein task images are not versioned yet have a clear impact on the task's result.
4. Support arbitrary container image mediums (Docker, ACI downloads, etc.) via the resource interface.
5. Support building-and-then-using Docker images in a pipeline flow, which is a pattern people already do for e.g. baking in dependencies at the start of a pipeline so that later stages don't have to fetch them.
6. Support remembering the exact version of the image used for the steps in the build, making it entirely reproducible once we implement build re-triggering.

Concrete proposal:

1. Change image in tasks to refer to a resource description, rather than a URI string. (The old URI string would be still supported for a period of time, for backwards-compatibility.)
2. Allow pipeline configs to override image by pointing to their own resource.

The intent of 1) is to keep task config files self-contained, so that fly execute and such still work.

The intent of 2) is to support configuring resources with credentials in a pipeline, so that credentials don't have to be hardcoded into a task config.

I'm not wild about 2) as it introduces the possibility of the pipeline config specifying an entirely different image than is specified in the task config. It also doesn't cover running fly execute with credentials.

Anyway, assuming those two changes, task configs would look something like:

atc-go.yml (using public image)

---
platform: linux

image:
  type: docker-image
  source: {repository: concourse/atc-ci}

inputs:
- name: concourse

run:
  path: concourse/ci/scripts/atc-go

Here's an example of overriding it with a private image in a pipeline:

resources:
- name: my-repo
  type: git
  source: # not interesting ...

- name: my-secret-image
  type: docker-image
  source:
    repository: user/repo
    username: secret-user
    password: secret-password

jobs:
- name: some-job
  plan:
  - get: my-repo
  - task: task-using-secret-image
    file: my-repo/atc-go.yml
    image: my-secret-image

Another option instead of 2) would be to introduce some sort of templating, but that'd take some thinking since right now all templating is done client-side (and I quite like it that way).

Issues this is relevant to:

• Don't re-run all Tasks when re-running a Job #157
• As a user I would like to see the docker container that each task ran in on the web UI. vmware-archive/atc#30
• [req] Private Registry Support #27

[concourse-bot]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• #111569920 Task images should be specified as resources
• #112018195 I should be able to see the output of my image_resource's get in the build
• #111911083 Only the version of an image_resource used in the latest build of a job should be cached forever
• #111911497 Images can configure env for a task's container
• #111911313 Task images can be overridden with an artifact in the plan

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[cjcjameson]

+1 to following on with https://www.pivotaltracker.com/n/projects/1059262/stories/111911313 when possible. It seems like the best way to get the benefit of keeping credentials in the pipeline, and letting task.yml's be flexible.

And congrats on 1.0!!!!!!

[concourse-bot]

Hello again!

All stories related to this issue have been accepted, so I'm going to automatically close this issue.

At the time of writing, the following stories have been accepted:

• #111569920 Task images should be specified as resources
• #112018195 I should be able to see the output of my image_resource's get in the build
• #111911083 Only the version of an image_resource used in the latest build of a job should be cached forever
• #111911497 Images can configure env for a task's container
• #111911313 Task images can be overridden with an artifact in the plan

If you feel there is still more to be done, or if you have any questions, leave a comment and we'll reopen if necessary!

[muv]

+1. Found this as I tried to use private Docker Hub repo in task yml config without hardcoding credentials. For pipeline ymls I use {{parameters}} and then inject them with --load-vars-from. Seems these parameters are not available in task files.

Is this proposal integrated into 1.2.0 or some future release?

[muv]

Perhaps support for {{parameters}} in tasks is a separate issue.