This contains the configuration for the Cloud Foundry Buildpacks team Concourse deployment.
- binary-builder-new: build binaries for Cloud Foundry buildpacks
- buildpacks: test and release all of the buildpacks
- bosh-lite: deploy CF LTS environment
- edge-shared: deploy CF Deployment environment
- brats: run BRATS against the master branch of buildpacks
- buildpack-verification: generate static site for buildpack verification
- buildpacks-ci: testing tasks for correct usage, rebuild CI docker images
- cf-release: deployment of latest buildpacks to cf-release develop
- gems-and-extensions: gems and extensions that support buildpack development and deployment
- notifications: monitor upstream sources for changes and notify on Slack
- cflinuxfs2: test and release Cloud Foundry cflinuxfs2
Jobs and tasks in the
buildpacks-ci repository store state in public-buildpacks-ci-robots. See repository README for details.
Commands and recipes
Updating all the Pipelines
Debugging the build
fly intercept -j $JOB_NAME -t task -n $TASK_NAME
Clearing the git resources
fly intercept -c $RESOURCE_NAME rm -rf /tmp/git-resource-repo-cache
To build a new version of a binary
Check out the
Edit the YAML file appropriate for the build (e.g.
Find the version number and package SHA256 of the new binary. For many binaries, the project website provides the SHA256 along with the release (for example, jruby.org/download provides the SHA256 along with each JRuby release). For others (such as Godep), you download the .tar.gz file and run
shasum -a 256 <tar_file>to obtain the SHA256.
Add any number of versions and their checksums to the array, e.g.
ruby: - version: 2.2.2 sha256: 5ffc0f317e429e6b29d4a98ac521c3ce65481bfd22a8cf845fa02a7b113d9b44
git commit -am 'Build ruby 2.2.2' && git push
Build should automatically kick off at
https://buildpacks.ci.cf-app.com/pipelines/binary-builder and silently
upload a binary to the
pivotal-buildpacks bucket under
Note that the array is a stack, which will be emptied as the build succeeds in packaging successive versions.
Running the Test Suite
If you are running the full test suite, some of the integration tests are dependent on the Lastpass CLI and correctly targeting the fly CLI.
To login to the Lastpass CLI:
lpass login $USERNAME
You will then be prompted for your Lastpass password and Google Authenticator Code.
To login to the Fly CLI and target the buildpacks CI:
fly -t buildpacks login
You will be prompted to select either the Github or Basic Auth authentication methods.
After these are set up, you will be able to run the test suite via:
Making Changes to Build Scripts
When you want to change how a binary gets built, there are two places you may need to make changes. All binaries are built by the
binary-builder-new pipeline, and you may need to change the task that builds them. For many binaries, the
binary-builder-new pipeline runs recipes from the
binary-builder repo; for those binaries, you will usually need to change the recipe rather than the concourse task.
For the list of currently supported binaries, check out our
The concourse task that orchestrates the building is
buildpacks-ci/tasks/build-binary-new/builder.rb; many of the recipes are in binary-builder.
To test these changes locally, you can execute the concourse task for it, but point to local changes. For instance:
$ cd buildpacks-ci $ STACK=cflinuxfs2 fly -t buildpacks e -c tasks/build-binary-new/build.yml -j binary-builder-new/build-r-3.4.X -i buildpacks-ci=.
For binaries that use recipes in
binary-builder, you can also test in Docker. For instance:
$ docker run -w /binary-builder -v `pwd`:/binary-builder -it cloudfoundry/cflinuxfs2:ruby-2.2.4 ./bin/binary-builder --name=ruby --version=2.2.3 --md5=150a5efc5f5d8a8011f30aa2594a7654 $ ls ruby-2.2.3-linux-x64.tgz
Buildpack Repositories Guide
buildpacks-ci pipelines and tasks refer to many other repositories. These repos are where the buildpack team and others develop buildpacks and related artifacts.
Each officially-supported buildpack has a
develop and a
Active development happens on
develop. Despite our best efforts,
develop will sometimes be unstable and is not production-ready.
Our release branch is
master. This is stable and only updated with new buildpack releases.
Tooling for Development and Runtime
- buildpack-packager Builds cached and uncached buildpacks
- machete Buildpack integration testing framework.
- compile-extensions Suite of utility scripts used in buildpacks at runtime
- libbuildpack Library used for writing buildpacks in Golang
- binary-builder Builds binaries against specified rootfs
- cflinuxfs2 Tooling to build cflinuxfs2 root file system ("rootfs") for CF
- brats Buildpack Runtime Acceptance Test Suite, a collection of smoke tests
BOSH releases are used in the assembly of
Experimental or unsupported
These buildpacks are possible candidates for promotion, or experimental architecture explorations.
- concourse-filter Redacts credentials from Concourse logs
- new_version_resource Concourse resource to track dependency versions by scraping webpages
Some repositories are private for historical or security reasons. We list them for completeness.