Skip to content
Browse files

feat(cicd): improve documentation on CI vs CD and re-write CI (#183)

  • Loading branch information...
ahmadnassri authored and kspaans committed Oct 16, 2018
1 parent e2207d2 commit a58f90e2855a5f324d3e2769c7c86b0761f3338c
Showing with 93 additions and 37 deletions.
  1. +3 −2 process/
  2. +35 −0 process/
  3. +2 −0 process/
  4. +53 −35 process/
@@ -1,7 +1,8 @@
## Process

- [Continuous Integration](
- [Continuous Delivery](
- [CI/CD](
- [Continuous Integration](
- [Continuous Delivery](
- [User Stories](
- [Smaller Stories Are Faster](
- [Project Template](
@@ -0,0 +1,35 @@
# Continuous Integration (CI) & Continuous Delivery (CD)

## Why

[Continuous Delivery]( is closely related to [Continuous Integration]( but they are often provided within the same tools, and while the topics are co-dependant, they do represent two separate paradigms.

## What

Areas of concern:

### Continuous Integration

- Focuses on enabling a smooth developer experience
- Coupled to Code Repositories
- Functional testing _(unit, quality, interaction, headless, etc ...)_
- Produces build artifacts
- Promotes release _candidates_
- Allows for flexibility of developer workflows _(branch based, pull requests, etc ...)_
- Code Isolation _(No Customer Data, or live databses)_
- Facilitates Quality measurements through integrations with other tools
- Provides Dashboards for build status and visibility into build artifacts

### Continuous Delivery

- Focuses on protecting and gating infrastructure environments _(e.g. production, staging)_
- Coupled to Infrastructure Design
- Integration testing _(knows about secrets, tokens, environments)_
- Load & Performance testing in the right infrastructure environment
- Environment management and deployment configuration
- Provides deployment strategies & rollbacks

## How

- [Continuous Integration](
- [Continuous Delivery](
@@ -1,5 +1,7 @@
# Continuous Delivery (and Deployment)

> **CI vs CD ?** Please refer to the [CI/CD topic](
## Why

Our [continuous integration]( build pipeline asserts that our application artifact is built and tested correctly. Given the confidence that our pipeline instills in our code, when we have proper testing practices, we should also be able to automate the deployment of any version of our application all the way to production.
@@ -1,76 +1,94 @@
# Continuous Integration
# Continuous Integration (CI)

> **CI vs CD ?** Please refer to the [CI/CD topic](
## Why

In an Agile development team, we want to release changes quickly. We want to minimize the time it takes to release code. However, we are working with many other teams in a large organization... we need some way to minimize the integration friction that we experience, when several teams are working on different features in the same codebases. Integration issues need to be caught as early as possible.
When developers are integrating code into a shared repository several times a day. Each code commit needs to be verified, tested and inspected to ensure it follows our Reference Architecture standards, security requirements, and general policies and best practices.

Continuous Integration is cheap. Not integrating continuously is expensive. When running integrations and tests so frequently, there is significantly less back-tracking to discover where things went wrong, so we can spend more time building features.

## What

Continuous integration is a set of principles, practices and procedures that makes it easier for large teams to collaborate together. Previously, under waterfall development, large chunks of scope would be peeled off and worked on in isolation, until the entirety of the scope was completely finished. Integration was pushed as late as possible, because it causes lots of friction due to exposing numerous bugs all at once.
In our context, given the technology platform and tools we use, we've identified the following key areas of interest for CI:

With continuous integration, teams push changes to the shared codebase several times per day. We have an automated build pipeline that tests every commit, which detects the bugs as early as possible in the development process. This allows us to move much quicker with feature rollouts, minimizes friction between teams, and ensures we don't release broken code.
### Critical Needs

On our previous PHP My-Account codebase, it would take a week to manually merge dozens of feature branches into a single release. If any changes to any feature in any branch of any github repository caused issues, the whole release would be delayed or called off. Integration friction happened in the majority of releases. Now with continuous integration in our new architecture, we release dozens of times per day, to dozens of apps, with FAR fewer issues than before.
The following areas are critical needs for TELUS:

## How
#### Modern developer experience

All modern organizations heavily depend on a digital workforce to achive their outcomes, in the context of software development, the developer workforce requires higher order of digital friendly products that _accelerate_ and _enable_ rather than _gate_ and _hinder_ their outcomes.

While this is a hard and ever evolving target to measure, it's immediately clear a balance needs to be struck between stability and modern practices, with an eye on the technology platforms of choice and the market forces driving these changes.

#### Declarative Config

### Story sizing
Modern CI products rely on declartive configuration with focus on simplicity and customization to enable a modern developer experience, and while this might be limiting in some cases, it is more empowering for cross-team collaboration and provides better visibility and auditability than programatic configuration systems.

It is important that user stories are kept small. Large tasks are broken down into smaller ones, until they can be accomplished within a day.
#### Single Sign On

### Trunk based development
We have a large team, and we need to ensure onboarding and offboarding to developer tooling is inclusive of our SSO corporate identity

For most changes, we now use Trunk Based Development. This means most changes are committed directly to master, without first making a branch. Sometimes branches are unavoidable, e.g. when doing large refactoring, etc., but these should be as short-lived as possible.
#### iOS support

### Feature toggles
There are many _mobile app_ focused CI solutions. However, we prefer to keep our web and mobile tooling close and consistent. _iOS_ makes this very difficult as an added complexity for supporting the Apple platform.

For stories that take longer than a day, consider using a "feature toggle" (in its most basic form, a boolean configuration plus an if/else block), where your feature is disabled in production upon deployment. You can leave it enabled in development and staging environments, for integration testing. This still ensures that code is integrated continuously, so there are less, if any, merge conflicts.
#### Security Compatibility with our corporate policies

### Test driven development
As a Canadian Telecom business, we have a set of security and privacy policies that govern our operations, therefore any product or service we rely on needs to support those policies, _(e.g. encryption, access controls, credential binding, log filtering, etc ...)_

Before a feature is developed, we write a unit test that tests the feature we want to implement. First we run the test, to ensure that it fails. This is a good thing! Now we write the feature that satisfies the test. If your test doesn't make breaking changes to the interface, it is safe to commit at this point. If it does modify the interface, the integration test needs to be updated to account for the new feature.
#### Docker as first class citizen

### Security testing
Since we rely on [Docker](../delivery/ for our local development and in our deployment infrastructure, our CI tools should also reflect these choices to ensure highest level of interoperability.

Additionally, we include tools like `nsp` and `snyk` to test our code for known vulnerabilities. More can (and will!) be added here... e.g. Twistlock allows us to monitor all running Docker containers on our cluster for application vulnerabilities as well.
#### Decoupled from Infrastructure

### Git commit hooks
While solutions like [Docker](../delivery/ allow for a higher order of de-coupling between code and infrustructure, we found that in many cases, special care is needed to configure Docker to work in both CI and CD domains, which leads to sacrifices and unwanted complexity.

We use a pre-commit hook to guarantee that tests are run, before committing a feature. This ensures that the delivery pipeline is not broken very often.
### Ideal CI Features

### Delivery pipeline
The following areas are nice-to-have but not critical features we expect in CI tooling:

The delivery pipeline takes each and every commit, builds the application artifact and unit tests it. If the tests pass, it deploys it to a pre-production integration testing environment, where integration tests are run to ensure the application functions (either end-to-end browser tests for UI or API contract tests for backend). Finally if all stages have passed, the option is given to deploy to production.
#### SaaS First

### Visibility
In alignment with our tooling choices and a digital-first products, we choose to rely on SaaS solutions _FIRST_, until a need arises where we have to maintain a solution within our own infrastructure _(e.g. security and privacy concerns)_

Failures and production deployments should be as visible as possible. We have dashboards in Jenkins and OpenShift, as well as Slack notifications, to tell us when things are good or bad.
#### Tooling Integration

### Smaller applications
CI Solutions need to enable a higher level of integrations out of the box with other developer tooling, to better enable developer acceleration, without relying on custom integrations _(unless absolutely needed)_

Monolithic architecture, waterfall development and large code changes are not conducive to independent, high performance teams. It is hard for many teams to truly _own_ a single project. And tight-coupling between shared libraries and frameworks causes integration friction that harms development.
#### Dashboards & Reporting

Rather than our previous single shared codebase, switching to microservices and standalone single-page-app UIs has been one of the major keys to unlocking delivery performance and release safety. Microservices can have well-defined contracts for integrating with each other. Our standalone UIs leverage a shared design framework to ensure that, while they are still independent codebases, they still look the same.
Automation without visibitliy is meaningless, products in the CI space need to provide clear and open dashboards, reports and build history to track the status of code commits and enable auditabiility

### Product Ownership vs Project Management
#### Build Logs

TELUS digital teams have restructured into outcome teams, to improve ownership of code, and independence of releasing. With smaller projects, teams can now own entire github repos, with their own standalone build pipelines, etc.
Relying on local development environments is not enough, therefore any CI solution needs to surface detailed logs to allow for debugging build issues.

### Declarative programming
#### Flexible Capacity

The tools we have chosen for reference architecture support **declarative programming**. Rather than traditional applications, which are not immutable or ephemeral, declarative code describes the structure, without describing its control flow.
We're a big development team that is constantly committing code to our repositories every day, therefore CI solutions need to accomodate our size and our speed.

For example, in React.js, we define a template that displays a specific view model. To change the page, we simply update the view model with new data. React will declaratively reconstruct the page, on the fly, without requiring us to manually edit the DOM.
### API Enabled

Similarly, with our container platform Kubernetes, we upload a "state" of an application, e.g. deploy container X with environment variables Y and this much RAM. Kubernetes doesn't ask "how": it internally decides where to deploy your application, on any node in the cluster with enough space.
We should be able to trigger, configure and completely manage any development tooling product though APIs, especially CI solutions.

### Dev/prod parity
### Product Maturity

Minimizing the differences between development and production also decreases the number of missed bugs. Similarly, all pre-production environments should be built as close to, if not identical to, the production environment.
New CI products surface on the market almost every month, we should be careful to what products and services we adopt if they are not mature enough or the company / open source community supporting them is not mature enough.

## Who
### Build Stages

Modern CI tooling enable complex workflow design through a "staged" approach to a build process, this is key for complex and large applications to enable shortened build times in context of change and target environment.

### Centralized Configuration

A large team has an even larger application catalogue, which leads to an even larger amount of code repositories. Having centralized configuration alleviates the need for duplication as well as enable governance and policy controls.

## How

We're currently using Jenkins for both CI and CD domains. and we're in the process of evaluating separate CI tooling.

## References

0 comments on commit a58f90e

Please sign in to comment.
You can’t perform that action at this time.