Concourse resource for interacting with Google Cloud Storage
Go Other
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
check More lenient version comparisions with github.com/cppforlife/go-semi-… Jul 13, 2018
ci Remove unused docker_email from CI pipeline Feb 1, 2018
cmd Project is not needed Mar 25, 2016
fakes Revert "Merge pull request #10 from aemengo/add-unpack-to-in" Apr 24, 2018
in Run make format to format code in in/*.go Apr 29, 2018
integration Add integration tests for "params.unpack" parameter May 12, 2018
out Fix more format Feb 7, 2018
scripts Adding build script and remove deprecated Aug 3, 2016
vendor More lenient version comparisions with github.com/cppforlife/go-semi-… Jul 13, 2018
versions More lenient version comparisions with github.com/cppforlife/go-semi-… Jul 13, 2018
.gitignore Revert "Merge pull request #10 from aemengo/add-unpack-to-in" Apr 24, 2018
.travis.yml Only tag docker image with latest on release versions Apr 24, 2018
CODE_OF_CONDUCT.md Update copyright and add contributing guidelines Feb 1, 2018
CONTRIBUTING.md Update copyright and add contributing guidelines Feb 1, 2018
Dockerfile Add CI build task Mar 26, 2016
Gopkg.lock More lenient version comparisions with github.com/cppforlife/go-semi-… Jul 13, 2018
Gopkg.toml More lenient version comparisions with github.com/cppforlife/go-semi-… Jul 13, 2018
LICENSE Update copyright and add contributing guidelines Feb 1, 2018
Makefile Refactor travis.yml Feb 1, 2018
NOTICE Update copyright and add contributing guidelines Feb 1, 2018
README.md Update README.md to reflect "params.unpack" functionality Apr 29, 2018
VERSION Concourse GCS resource v0.3.0 Jul 28, 2018
client_secrets.json.enc Update client_secrets file Feb 1, 2018
gcsclient.go Add go:generate for counterfieter mock generation Apr 28, 2018
models.go Fix error when putting to versioned bucket Nov 23, 2016
utils.go Initial version Mar 22, 2016

README.md

GCS Resource Build Status

Versions objects in a Google Cloud Storage (GCS) bucket, by pattern-matching filenames to identify version numbers.

This resource is based on the official S3 resource.

Source Configuration

  • bucket (required): the name of the bucket.

  • json_key (required): the contents of your GCS Account JSON Key file to use when accessing the bucket. Example:

    json_key: |
      {
        "private_key_id": "...",
        "private_key": "...",
        "client_email": "...",
        "client_id": "...",
        "type": "service_account"
      }
    

File Names

One of the following two options must be specified:

  • regexp: the pattern to match filenames against within GCS. The first grouped match is used to extract the version, or if a group is explicitly named version, that group is used.

    The version extracted from this pattern is used to version the resource. Semantic versions, or just numbers, are supported. Accordingly, full regular expressions are supported, to specify the capture groups.

  • versioned_file: if you enable versioning for your GCS bucket then you can keep the file name the same and upload new versions of your file without resorting to version numbers. This property is the path to the file in your GCS bucket.

Behavior

check: Extract versions from the bucket.

Objects will be found via the pattern configured by regexp or versioned_file. The versions will be used to order them (using semver). Each object's filename is the resulting version.

in: Fetch an object from the bucket.

Places the following files in the destination:

  • (filename): the file fetched from the bucket.

  • url: a file containing the URL of the object.

  • version: the version identified in the file name (only if using regexp).

  • generation: the object's generation (only if using versioned_file).

Parameters

  • unpack: Optional. If true and the file is an archive (tar, gzipped tar, other gzipped file, or zip), unpack the file. Gzipped tarballs will be both ungzipped and untarred.

out: Upload an object to the bucket.

Given a file specified by file, upload it to the GCS bucket. If regexp is specified, the new file will be uploaded to the directory that the regex searches in. If versioned_file is specified, the new file will be uploaded as a new version of that file.

Parameters

  • file (required): path to the file to upload, provided by an output of a task. If multiple files are matched by the glob, an error is raised. The file which matches will be placed into the directory structure on GCS as defined in regexp in the resource definition. The matching syntax is bash glob expansion, so no capture groups, etc.

  • predefined_acl (optional): the predefined ACL for the object. Acceptable values are:

    • authenticatedRead: Object owner gets OWNER access, and allAuthenticatedUsers get READER access.
    • bucketOwnerFullControl: Object owner gets OWNER access, and project team owners get OWNER access.
    • bucketOwnerRead: Object owner gets OWNER access, and project team owners get READER access.
    • private: Object owner gets OWNER access.
    • projectPrivate: Object owner gets OWNER access, and project team members get access according to their roles.
    • publicRead: Object owner gets OWNER access, and allUsers get READER access.
    • publicReadWrite: Object owner gets OWNER access, and allUsers get READER and WRITER access.
  • content_type (optional): Set the MIME type for the object to be uploaded, eg. application/octet-stream.

Example Configuration

Resource Type

resource_types:
  - name: gcs-resource
    type: docker-image
    source:
      repository: frodenas/gcs-resource

Resource

resources:
  - name: release
    type: gcs-resource
    source:
      bucket: releases
      json_key: <GCS-ACCOUNT-JSON-KEY-CONTENTS>
      regexp: directory_on_gcs/release-(.*).tgz

Plan

- get: release
- put: release
  params:
    file: path/to/release-*.tgz
    predefined_acl: publicRead
    content_type: application/octet-stream

Development

  • Get the resource: go get github.com/frodenas/gcs-resource
  • Run the unit-tests: make
  • Run the integration-tests: make integration-tests
  • Build the source code using concourse: fly -t ConcourseTarget execute -c ci/tasks/build.yml -i gcs-resource-src=. -o built-resource=.
  • Build the Docker image to be use inside your pipeline: docker build -t frodenas/gcs-resource .

Contributing

Refer to the contributing guidelines.

License

Apache License 2.0, see LICENSE.