CI CD Pipelines

Continuous Integration

Continuous Integration is a solution for a hopefully bi-gone problem of integrating software in a Waterfall era. Developers worked independently for weeks or even months creating libraries which depended on each other according to specification documents. After development was complete, the applications had to be compiled, and surprise, surprise nothing worked. It took yet more weeks of painful effort to fix all the incompatibilities. Often when something is painful, you can reduce the pain, by doing it more frequently. This is what led to continuous integration. Early implementations were probably a combination of Makefiles and cron jobs. Automated testing made the process more robust. Eventually CI servers were developed. There are now many CI servers, but the fundamental steps of a good CI pipeline haven't changed.

1. Start with a clean environment
2. Get a known version of the source in a reproducible way
3. Build the artefacts
4. Test the artefacts
5. Version the artefacts
6. Publish the artefacts to a repository

Like with many things in software development, it is easy to get enamoured with tooling and over complicate this.

• Be suspicious of elaborate, multi-stage pipelines with checkpoints and gates.
• Be conscious of the trade off between build speed and starting with a clean environment.
• Minimise differences between your CI environment and local one to avoid server only build issues.
• Take time to make build scripts/definitions short and extremely simple.

Continuous Deployment

Continuous deployment is an extension to continuous integration. It encourages small, frequent releases over big bang releases. Small releases are less is likely to go wrong and when they do, easier to diagnose / fix (roll forward) and less painful to roll back. Continuous deployment is more risky than manually initiated deployments, and practices should be adopted in order to mitigate, for example

• Automated testing must be thorough
• Alerting and monitoring must be effective
• Canary deployments should be used to mitigate impact
• Suspicious deployments should be automatically rolled back

If the deployment pipeline is not sufficiently mature to support this, then consider a continuous deployment pipeline to pre-production environments and a 1-click deployment to live.

Hosted CI

Hosted CI is an attractive proposition, but comes with some significant drawbacks.

1. Hosted services typically providing new virtual machines per build. While scoring highly against the "start with a clean environment" principle, it the VMs take time to provision and prevent caching of source code, dependencies and docker images.

2. Many hosted services attempted to improve build time by bundling common dependencies (e.g. multiple version of MySQL) with their base images. Ironically this can make it harder to install custom dependencies, and can delay the hosted service provider upgrading their VMs.

3. Many hosted service providers did not anticipate microservice architectures and do not support shared configuration. Build definitions and secrets must be replicated for each project

Be suspicious of hosted CD. It implies opening your network to a 3rd party and trusting the 3rd party with deployment secrets. Instead consider configuring the build or artefact repository trigger deployments performed by an internal service.