Bosh Repave

This is a Concourse pipeline sample that will perform the repave of VMs of selected bosh deployments in a PCF installation on a pre-defined schedule.

The pipeline also allows for the selection of the VMs to be repaved within each deployment, either ALL or only ones of Non-Singleton instances.

The pipeline scripts use bosh recreate as the mechanism to perform the repave of VMs.

The scripts iterate through the list of selected deployments provided as a pipeline parameter (e.g. cf, apm, ...) and then issue a bosh recreate command for the targeted instances.

Notes

• The purpose of the pipeline is to provide a mechanism to selectively repave VMs of selected deployments on a regular basis, for organizations that require such procedure for any reason such as internal security regulations.

• If you do not simply wish to target all VMs within one or more deployments, a typical pattern is to have a pipeline per deployment (only one deployment specified in each pipeline's deployments parameter). You can then set the jobs parameter for fine-grained control of what's repaved.

• The use and the execution of this pipeline should be done with proper caution and planning, as the recreation of VMs that implement singleton jobs may cause outages to the platform. The pipeline provides a flag to skip VMs of singleton jobs.

• This pipeline should be scheduled to run in a time-window that does not coincide with other procedures that may also affect the platform's availability or performance, such as backups, updates or upgrades. A scheduler (time resource) is provided by default in the pipeline implementation.

• In addition to the scheduler resource, a "deployments lock" mechanism is implemented by the pipeline. Such mechanism should also be used in other pipelines such as the ones for backups and upgrades, so their execution will not collide with the repaving of VMs.

How to use the pipeline

1. Clone this git project locally

2. Make a copy of ci/secrets.sample.yml: cp ci/secrets.sample.yml ci/secrets.yml

3. Edit the copy of the secrets file:
   opsman-url: Ops Manager URL. e.g. https://pcf.example.com
   opsman-username: Ops Manager admin user. e.g. admin
opsman-password: Ops Manager user password
   skip-ssl-validation: skip SSL validation for Ops Manager login e.g. true or false
deployments: comma-separated list of bosh deployments to repave. It has to contain the prefix of PCF deployed releases (from the output of bosh deployments, remove the -XXXXXXX... numeric suffix from the deployment names generated by PCF), e.g. cf,apm
jobs: comma-separated list of jobs to repave. When using this, you should only specify a single deployment. This can be used to repave select VMs e.g. Diego cells within the cf deployment. See the related note above for a common pattern when using this feature.
   repave-singleton-jobs: flag to control which VM instances will be repaved. e.g. true repave all VMs of all jobs, false repave only jobs with more than one instance.
   perform-dry-run-only: for testing the repave action with a dry-run of bosh recreate without actually recreating any VMs. e.g. true perform dry-run, false execute repave
   scheduler-time-window-start: initial time for the scheduler's execution time window, e.g. 1:00 AM
scheduler-time-window-stop: end time for the scheduler's execution time window, e.g. 2:00 AM
scheduler-time-location: time zone for the scheduler. e.g. America/Phoenix. See docs for accepted values.
   lock-git-repo-uri: The git repo URL for the pool resource used as a deployment lock. See docs for more information on how to bootstrap a pool resource repository.
   lock-git-repo-branch: git branch for the lock repository. e.g. master
lock-pool-name: the pool name for the lock
   lock-git-private-key: private key for the pool resource repository

4. Create the pipeline with the fly command e.g. fly -t <target> sp -p repave -c ci/pipeline.yml -l ci/secrets.yml

5. Unpause the pipeline to either run it manually or when the scheduler generates an event within the specified time-window.