bosh deployment of Atlassian Bamboo and remote agents with orchestration support from Bamboo Agent APIs This project is mostly a proof of concepts that uses bosh as well as agent apis for bamboo to support administration of a large scale Bamboo build farm.
- Building & Deploying the Release
- Allows fully automated upgrades to all remote agents with a single command
- Intelligently manages lifecycle of build agents (using pre-start, start, and drain)
- will wait for existing builds to complete
- mark agent disabled
- take agent offline, upgrade/replace
- bring agent online
bosh startwill mark agent enabled check for any open tasks to complete (see api docs)
- Uses persistent store for consistent agent IDs as they update over time
- Sets agent name in Bamboo to the job instance container id.
- BOSH provides main orchestration of services
- monit provides service monitoring and automated restarts
- Java used in related Bamboo plugin
- Python, Bash/Shell and various linux tools support scripts
Agent names in Bamboo will use bosh container IDs. BOSH drain script marks agents disabled to prevent new work from being assigned to them BOSH drain scripts for agents block for any running workloads to complete
Building & Deploying the Release
- Install Bosh-lite based on vendor docs (this repo assumes default network on virtualbox defined there)
- Download JDK8 and Atlassian Bamboo from vendor sites. If they differ than those defined in blobs.yml then you'll need to update file names and possibly some templates.
./buildAndDeployRelease --us --postgre --alias --route
- uploads stemcells
- sets cloud config for network and vminfo
- uploads community postgres releases
- builds and uploads this bamboo releases
- Triggers initial deploy of sample manifest
- to iterate development re-run the build script omitting flags.
Note on Blob versions
You'll need to download jdk8 and the atlassian bamboo version to match those found in blobs.yml. You may manually add them using CLI, or leave them in
~/Downloads for the setup script to add them as part of setup.
Build release and push to bosh director
See bosh-lite docks for quick setup, or target your existing director
buildAndDeployRelease.sh for individual steps, or run it to deploy the current config.
api.enabled- make use of Agent APIs? (otherwise shutdowns will just be a hard kill, interupting any running jobs. And capabilities in config may not match the current agent, risking build failures)
api.token.uuid- Valid token from agent apis for bamboo created in admin UI with 'change' permission
processblock (just tips, defaults work)
watch_timedefines how long we let jobs run before forcing an agent or giving up (depening on cli command)
max_in_flight- this si critical to make sure all your agents don't go offline at once! Set to 1 for small farms or a small percentage of large farms
canaries- 1 is fine, bosh release should include all smoke tests ot use on start
This approach to virtualbox networking is a result of my learnings on https://github.com/eddiewebb/concourse-pipeline-bosh-virtualbox
Understanding the lifecycle of BOSH jobs is critical to understand how this plugin interacts with the APIS at the right times, as well as the proper design of job specs. https://bosh.io/docs/job-lifecycle.html
- monit unmonitor is called for each process
- drain scripts run for all jobs on the VM in parallel (waits for all drain scripts to finish)
- monit stop is called for each process
- Persistent disks are unmounted on the VM if configured
Bosh VM Directories
Useful info for troubleshooting.
bosh -e vbox -d bamboo ssh bamboo-server
/var/vcap/data/: Directory that is used by the release jobs to keep ephemeral data. Each release job usually creates a sub-folder with its name for namespacing (e.g. redis-server will place data into /var/vcap/data/redis-server).
/var/vcap/store/: Directory that is used by the release jobs to keep persistent data. Each release job usually creates a sub-folder with its name for namespacing (e.g. redis-server will place data into /var/vcap/store/redis-server).
/var/vcap/sys/run/: Directory that is used by the release jobs to keep miscellaneous ephemeral data about currently running processes, for example, pid and lock files. Each release job usually creates a sub-folder with its name for namespacing (e.g. redis-server will place data into /var/vcap/sys/run/redis-server).
/var/vcap/sys/log/: Directory that is used by the release jobs to keep logs. Each release job usually creates a sub-folder with its name for namespacing (e.g. redis-server will place data into /var/vcap/sys/log/redis-server). Files in this directory are log rotated on a specific schedule configure by the Agent.
- Script to setup agent tokens and rules
- Install agent API plugins
- Separate agent API job/specialites from bamboo with bosh "operator" files provided by CLI v2