gocd-b2-artifacts
Plugins for GoCD to publish, fetch and poll artifacts to/from Backblaze B2 storage.
Requirements
Should work with GoCD >= 17.11.0.
Overview
This project consists of 3 plugins that make it possible to integrate GoCD with the Backblaze B2 Cloud Storage:
- Publish plugin - pushes artifacts to B2.
- Material plugin - polls B2 for new artifacts.
- Fetch plugin - downloads artifacts from B2.
Publish plugin can be used standalone to push artifacts to a desired bucket and path prefix. Material and Fetch plugins depend on each other. Material plugin only polls for the new packages (published artifacts) and does not download any files on its own. Fetch plugin has to be used in the same pipeline as Material plugin to download desired files. This follows GoCD design. Please read GoCD Package Material to get more information on package management in GoCD.
Installation
- Download the plugin(s) from Releases.
- Follow plugin installation instructions.
Usage
Publish plugin
Publish plugin is a simple task plugin and as such it needs some additional configuration to work. B2 credentials have to be set as environmental variables, either on a pipeline level, stage level or job level.
B2_ACCOUNT_ID and B2_APPLICATION_KEY should contain account ID and application key of your B2 account. Bucket name
is set for each publish task separately as described below.
Go to Edit pipeline view, choose a job, Tasks tab and then click Add new task > Publish to B2.
Fill out the form and click Save.
The plugin can upload files from multiple sources. Each source can be defined using wildcards and have its own destination, that is, additional prefix when uploaded.
By default Publish plugin uploads artifacts using PIPELINE_NAME/STAGE_NAME/JOB_NAME/PIPELINE_COUNTER.STAGE_COUNTER
prefix. For the pipeline visible in the above pictures this would mean for example build-linux/build/build-app/10.2
prefix, where 10 is the pipeline counter and 2 is the stage counter when this job was run. If you don't intend to
use Material/Fetch plugins from this project you can override this behavior by setting a custom prefix in
Destination Prefix field. Otherwise, leave it empty.
Bucket name is a name of an existing bucket in the account configured using B2_ACCOUNT_ID and B2_APPLICATION_KEY
environmental variables. Optionally, you can set GO_ARTIFACTS_B2_BUCKET environmental variable. This field has
precedence over the variable.
Material plugin
Configuration of Material plugin requires adding a package repository definition. Go to Admin > Package Repositories
and click Add New Repository. Fill out the form and click CHECK CONNECTION to check whether the plugin can connect
to the account and find the bucket. If everything is OK, click Save.
With repository configured, go to your pipeline settings and tab Materials. Click Add Material > Package.
Choose previously configured repository and click Define new if you haven't configured any packages yet. Fill out the
form with pipeline, stage and job names, which uniquely identify a job that pushes artifacts using Publish plugin.
Optionally, click CHECK PACKAGE to check if the plugin can find it. In case you haven't pushed any artifacts at this
point you are going to get an error message, which you can ignore. Click Save Package and Material to finish.
The material is now ready and will poll for new packages. When found, it will trigger the pipeline. Apart from doing just that, material plugin also sets very important environmental variables, with information containing repository name, package name, credentials and polled package version. This is needed in order for Fetch plugin to work properly. You can see these variables in a job log.
Fetch plugin
If there is a B2 material defined for a given pipeline you can use Fetch plugin to download its artifacts to a job's
workdir. Go to the job's settings, tab Tasks, and click Add new task > Fetch from B2.
Now fill out the form with the previously defined repository and package names. Optionally, provide destination path
where artifacts should be downloaded. Click Save and you are good to go!
Bugs
Let me know if you find any.
Credits
This project was inspired by indix/gocd-s3-artifacts.