Concourse Pipelines Integration with CredHub
Concourse supports integration with CredHub for credentials management in its pipelines, which can reference encrypted secrets stored in a CredHub server and get them automatically interpolated during execution of tasks.
This article provides samples of deployment manifests and pipelines for the integration of a BOSH-deployed Concourse server with CredHub. See the documentation page for information about CredHub integration with a Concourse server deployed using some other method.
Table of Contents
- Concourse Configuration
- Integration Topologies
- Deploying a CredHub server colocated with a Concourse VM
- Adding secrets to CredHub
- Sample pipeline to use CredHub secrets
- Configuring a UAA client for Concourse-CredHub integration
Concourse can be integrated with a CredHub server by configuring its ATC job's deployment properties with information about the CredHub server and corresponding UAA authentication credentials.
Here is an excerpt of a Concourse deployment file sample:
jobs: - name: atc ... properties: ... credhub: url: https://<cred_hub_server_ip_or_fqdn>:<credhub_server_port> tls: ca_cert: <cred_hub_server_ca> client_id: <uaa_client_id_for_concourse> client_secret: <uaa_client_secret_for_concourse>
client_id and a
client_secret are required to be created on the UAA server that manages authentication for CredHub. Concourse will use those credentials to access the CredHub API endpoint (i.e.
https://<cred_hub_server_ip_or_fqdn>:<credhub_server_port>) and request values of secrets injected into a pipelines.
See this section for an example on how to configure
client_secret with the UAA server.
a. Independent CredHub Server
If the CredHub server already exists or if it is deployed on an independent set of dedicated VMs (see docs), then two steps are required to integrate Concourse with it:
- Make sure that the Concourse ATC VM can access the CredHub API endpoint and
- Configure a set of
client_secretproperties on the CredHub's UAA for authentication. See section Configuring a UAA client for Concourse-CredHub integration for an example.
See CredHub documentation for more information on CredHub server deployment alternatives.
b. CredHub Server colocated with Concourse ATC
In certain cases, such as experimental deployments or environments that do not require an independent CredHub server, CredHub can be deployed as a set of jobs colocated in the same VM as Concourse's ATC.
This integration approach provides a CredHub server that is dedicated to the Concourse pipelines and is fully managed by Concourse administrators. Also, during Concourse upgrades, the CredHub server will only be down when Concourse ATC job is also down, minimizing potential credential server downtime for pipelines.
This deployment manifest provides an example of how both
uaa jobs could be deployed along with Concourse's ATC VM.
- Bosh Director to orchestrate and manage the Concourse and CredHub deployment. Please refer to bosh.io for documentation on how to deploy and connect to a Bosh Director for your specific IaaS.
- Cloud Config set to the Bosh Director to define network, disk and VM settings.
Create and update your Concourse deployment YML file containing the CredHub jobs defined as part of the Concourse's
webVM. See this deployment file as an example for a starting point, or use the operations files provided in this repo to transform an existing manifest.
Configure concourse-with-credhub-params.yml with the corresponding values for your deployment.
Upload required release files to the Bosh Director.
Deploy Concourse and CredHub with the Bosh CLI:
bosh -e <bosh-alias> -d concourse deploy concourse-with-credhub.yml -l concourse-with-credhub-params.yml --vars-store=./concourse-gen-vars.yml
The sample deployment manifest uses the feature of automatic generation of variables, certificates and keys from Bosh (see
variablessection of the file). Bosh will will add the values of the newly crated variables to a local file (defined by
--vars-store) for future reference purposes.
Once the deployment is finished, all servers will be available from the
concourse_host IP address or domain name set in the
Download the CredHub CLI
Target the CredHub API endpoint
credhub api https://<concourse_host>:8844 --ca-cert <cred_hub_server_ca>
For the server deployed in the example above, you can use Bosh 2's interpolate command to retrieve generated certificates and variables:
credhub api https://<concourse_host>:8844:8844 --ca-cert <(bosh int ./concourse-gen-vars.yml --path /concourse-tls/ca)
Set CREDHUB_PASSWORD environment variable
export CREDHUB_PASSWORD=$(bosh int ./concourse-gen-vars.yml --path /uaa-users-admin)
Login to CredHub
credhub login -u admin -p "$CREDHUB_PASSWORD"
Create credentials using the path required by Concourse
Concourse will search for secrets using the path setup for the CredHub integration in its deployment file. By default such path is
/concourse/<team-name>/<pipeline-name>/<secretId>. If the secret is not found, Concourse will then search it again at the corresponding
For example, to create a team-level secret of ID
hellowith the CredHub using the cli:
credhub set --type value --name '/concourse/myteam/hello' --value 'World'
To refer to a CredHub secret in a Concourse pipeline, simply add the within double parenthesis to the pipeline.
For example, samples/hello-credhub.yml:
jobs: - name: hello-credhub plan: - do: - task: hello-credhub config: platform: linux image_resource: type: docker-image source: repository: ubuntu run: path: sh args: - -exc - | echo "Hello $WORLD_PARAM" params: WORLD_PARAM: ((hello))
The parameter definition
WORLD_PARAM: ((hello)) tells Concourse to retrieve secret
/concourse/<team-name>/hello from CredHub during that pipeline execution.
If the secret exists, then Concourse will proceed with running the pipeline using the retrieved value (it should print "Hello World"), otherwise it will halt the pipeline with the corresponding error message.
A client ID with authorities
credhub.write, credhub.read is required to be set in UAA for the Concourse-CredHub integration to work.
Such client ID can be created either with the
uaac cli or by updating UAA's deployment with such settings.
Here is an excerpt of a UAA job's
clients section that defines a
concourse_to_credhub client ID:
clients: ... concourse_to_credhub: override: true authorized-grant-types: client_credentials scope: "" authorities: credhub.read,credhub.write access-token-validity: 30 refresh-token-validity: 3600 secret: ((concourse_to_credhub_secret))