Skip to content
Concourse CI Resource for uploading files to S3
Branch: master
Clone or download
Pull request Compare This branch is 10 commits ahead of neutrinog:master.
Latest commit 53ef9da Mar 1, 2019

Simple S3 Resource for Concourse

This is a fork of the Concourse resource s3-resource-simple.

Consuming this resource

- name: s3-resource-simple
  type: docker-image
    # No `tag:` specified: will pick up `latest`.
    repository: MYREGISTRY/s3-resource-simple

Pipelines to build and test this resource

The pipeline file at ci/s3-resource-simple-pipeline.yml enables a workflow where you can have per-feature branch pipelines plus a master pipeline.

Master branch pipeline

The master branch pipeline, s3-resource-simple-master, will push the Docker image for the Concourse resource to repository s3-resource-simple with tag latest, so that any client pipeline will pick it up immediately.

Each time a feat branch is merged into the master branch, this pipeline will run and publish a new Docker image.

Feature branch pipelines

Given feature branch feat-x, the associated feature branch pipeline s3-resource-simple-feat-x will push the Docker image for the Concourse resource to repository s3-resource-simple-scratch with tag feat-x-latest, so that additional integrations tests can be done by referring to the scratch repository with the tag feat-x-latest without impacting users of the published resource.


  • Add to your Concourse credentials manager the needed secrets.
  • Optionally change files ci/settings/master-branch.yml and ci/settings/feature-branch.yml.

Setting the master pipeline

fly -t developers set-pipeline -p s3-resource-simple-master \
    -c ci/s3-resource-simple-pipeline.yml \
    -l ci/settings/master-branch.yml

fly -t developers unpause-pipeline -p s3-resource-simple-master

Setting the feature branch pipeline

export BRANCH=$(git rev-parse --abbrev-ref HEAD)
fly -t developers set-pipeline -p s3-resource-simple-$BRANCH \
    -c ci/s3-resource-simple-pipeline.yml \
    -l ci/settings/feature-branch.yml \
    -v branch=$BRANCH \
    -v tag=$BRANCH-latest \
    -v tag_prefix=$BRANCH-

fly -t developers unpause-pipeline -p s3-resource-simple-$BRANCH

Understanding the different Docker images at play

With reference to the pipeline, there are two jobs, work-img and resource-img.

  • Job work-img is used to build images for the workings of the pipeline itself, that is, to run the test tasks.
  • Job resource-img builds the final product of this pipeline: the Concourse resource s3-resource-simple.

WARNING: Everything below is outdated

Resource to upload files to S3. Unlike the the official S3 Resource, this Resource can upload or download multiple files.


Include the following in your Pipeline YAML file, replacing the values in the angle brackets (< >):

  - name: <resource type name>
    type: docker-image
      repository: 18fgsa/s3-resource-simple
  - name: <resource name>
    type: <resource type name>
      access_key_id: { { aws-access-key } }
      secret_access_key: { { aws-secret-key } }
      bucket: { { aws-bucket } }
      path: <optional, use to sync to a specific path of the bucket instead of root of bucket>
      options: [<optional, see note below>]
      region: <optional, see below>
      sync: <optional, see below>
  - name: <job name>
      - <some Resource or Task that outputs files>
      - put: <resource name>
          dir: assets

AWS Credentials

The access_key_id and secret_access_key are optional and if not provided the EC2 Metadata service will be queried for role based credentials.


The options parameter is synonymous with the options that aws cli accepts for sync. Please see S3 Sync Options and pay special attention to the Use of Exclude and Include Filters.

Given the following directory test:

├── results
│   ├── 1.json
│   └── 2.json
└── scripts

we can upload only the results subdirectory by using the following options in our task configuration:

  - "--exclude '*'"
  - "--include 'results/*'"


Interacting with some AWS regions (like London) requires AWS Signature Version

  1. This options allows you to explicitly specify region where your bucket is located (if this is set, AWS_DEFAULT_REGION env variable will be set accordingly).
region: eu-west-2


By default files will not be synced from s3. Disabling this will download all files.

sync: true


Will change the working directory to subdirectory before invoking the sync

dir: subdirectory
You can’t perform that action at this time.