Skip to content

peakon/s3-cache-buildkite-plugin

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
lib
 
 
 
 
 
 
 
 
 
 
 
 

S3 Cache Buildkite Plugin

CI

Save and restore cache to and from AWS S3.

Example

Add the following to your pipeline.yml:

steps:
  - command: npm install && npm test
    plugins:
      - peakon/s3-cache#v2.2.1:
          id: CACHE_IDENTIFIER # optional, default: none
          aws_profile: aws-profile-name # optional, default: none
          restore_dry_run: false # set it to "true" to only check if cacheKey is present on S3 (no download / restoring)
          save:
            - key: 'v1-node-modules-{{ checksum("package-lock.json") }}' # required
              paths: [ "node_modules" ] # required, array of strings
              when: on_success # optional, one of {always, on_success, on_failure}, default: on_success
              overwrite: false # optional, set true to overwrite cache on S3 even if object already exists
          restore:
            - keys:
                - 'v1-node-modules-{{ checksum "package-lock.json" }}'
                - 'v1-node-modules-' # will load latest cache starting with v1-node-modules- (not yet implemented)

Configuration

Prerequisites

Make sure to set BUILDKITE_PLUGIN_S3_CACHE_BUCKET_NAME=your-cache-bucket-name before using this plugin.

Plugin

You can specify either save or restore or both of them for a single pipeline step.

Checking if cache was successfully restored

In some cases you may need to build a conditional logic in the build command based on the results of cache restore operation (for example, to avoid re-generating the cache which already exists and was restored successfully).

To support this use-case, this plugin exports environment variables that can be used during a command step. The feature is opt-in and requires id to be specified in plugin configuration.

For example, this step generates a cache of node_modules (which is then used by all jobs that need it):

steps:
  - command: "[ ! \"${BUILDKITE_PLUGIN_S3_CACHE_npm_0_KEY_0_HIT}\" =~ ^(true)$ ] && npm install"
    plugins:
      - peakon/s3-cache#2.2.1:
          id: npm
          restore_dry_run: true # This saves runtime, but doesn't check for integrity 
          restore:
            - keys: [ 'v1-node-modules-{{ checksum "package-lock.json" }}' ]
          save:
            - key: 'v1-node-modules-{{ checksum "package-lock.json" }}'
              paths: [ "node_modules" ]

Supported functions

  • checksum 'filename' - sha256 hash of a filename

  • epoch - time in seconds since Unix epoch (in UTC)

  • .Environment.SOME_VAR - a value of environment variable SOME_VAR

AWS profiles

You can specify a custom AWS profile to be used by AWS CLI

  • in pipeline YAML (aws_profile: profile_name)
  • via BUILDKITE_PLUGIN_S3_CACHE_AWS_PROFILE environment variable (e.g. inside agent environment hook).

Developing

To run the tests:

docker-compose run --rm tests

Contributing

  1. Fork the repo
  2. Make the changes
  3. Run the tests
  4. Commit and push your changes
  5. Send a pull request

About

Buildkite plugin for caching build files in AWS S3 bucket

Topics

Resources

License

Stars

Watchers

Forks

Languages