JEP-400: Jenkins X: Jenkins for Kubernetes CD
Jenkins X: Jenkins for Kubernetes CD
- Backwards Compatibility
- Infrastructure Requirements
- Prototype Implementation
Jenkins X provides a Kubernetes-native CI / CD platform for developing Cloud Native applications. Jenkins X uses a distribution of Jenkins as the core CI / CD engine.
This JEP is a “stake on the ground” that captures the mission and the general directions.
The Jenkins X sub project focuses 100% on Kubernetes, CI/CD and Cloud Native use cases to provide great developer productivity. To that end, we have so far identified the following major deliverables.
Jenkins X is targeted at existing and new Jenkins users who are either:
Already using Kubernetes and want to adopt CI / CD
Want CI / CD and increasingly to move to the public cloud - though don’t necessary know anything about Kubernetes
In both cases we expect these users to "just want get things done" by using best practices rather than discovering the best practices on their own.
Jenkins X promotes a particular Git branching & repository model. The rest of the tools and services are built to fit this model.
This development practice represents the best practice of developing Kubernetes applications, based in part on the experience of developing Fabric8, a project of a similar mission and on the results of the State of DevOps report.
The value of Jenkins X is that if you follow this best practice, then Jenkins X assembles all the pieces for you (e.g., Jenkins, Kubernetes, Git, CI/CD etc) such that developers can be instantly productive. This is akin to what Maven brought over Ant — encouraging developers to use a standard Lifecycle-model provided by Maven (DRY) to achieve greater productivity.
Here are some examples of the practice:
The master branch shall be always clean and release ready. No long running feature branches to stay lean.
Pull requests are used to work on new changes that then land to the master. CI tests are triggered on changes to a PR and the PR can only merged if all the CI checks pass and it passes any required code reviews etc.
A release is produced from the master branch to create versioned immutable artifacts (jars, binaries, docker images, helm charts etc). A release might be triggered manually, or automatically upon a new PR merge, or even at set frequency.
Which version of what services run in which environment is managed declaratively in a separate environment Git repo. You make deployment happen by pushing changes into the environment git repository which then triggers the Environments pipeline. This approach is often referred to as GitOps and is similar to how people develop Chef recipes or Ansible playbooks with Git.
Among web application developers practicing Continuous Delivery, the notion of “environment” is a well-established practice. It allows orderly flow of changes from a developer, through testing and staging to production. Yet, Jenkins does not have the environment concept as the first class citizen.
Jenkins X bridges this gap by introducing the concept of ‘environment’ on top of more generic Kubernetes concepts, such as namespaces and GitOps. The development practice then talks in terms of promoting apps from one environment to another in a cascading manner.
For example a release occurs for a new version
1.2.3 resulting in immutable artifacts being created.
So docker images and helm charts with the version
1.2.3 are created along with other things (npm packages, maven jars or whatever).
Then the promotion of an application from version
1.2.3 for an Environment in the Jenkins X GitOps model results in:
A Pull Request being created in the environment’s git repository to update the version of the application
When the Pull Request is approved and merged, a Pipeline runs in the environment applying the new changes to the underlying Environment in a Kubernetes namespace
In classic web application development, environments are often static. Jenkins X leverages the elasticity of Kubernetes to offer more dynamic environments to users. For example, we offer “preview environments” that are tied to GitHub pull requests, to let reviewers see the running changes before approving the merge to master.
Following the development practice discussed above, one finds that some common sequences of tasks will happen over and over.
One such sequence of tasks is the act of “promoting” an application to an environment. As an input, it takes what version of which application to promote which environment, then it needs to execute a series of appropriate Kubernetes commands to change some pods that are running, with appropriate metadata.
In order to make these common tasks easy, Jenkins X defines a command line tool, jx, which encapsulates them as high-level operations. Jenkins X CLI is used not only by developers from their computers, but also used by Jenkins Pipeline, which will be discussed later.
Jenkins X CLI is a central user interface of Jenkins X. It makes it easy to:
install Jenkins X on any kubernetes cluster
create new kubernetes clusters from scratch on the public cloud
setting up Environments for each Team
import existing projects or creating new Spring Boot applications then:
automatically set up the CI / CD pipeline and webhooks
create new releases and promote them through the Environments on merge to master
support Preview Environments on Pull Requests
Pipeline where “the right thing is the easiest thing”
At the heart of Jenkins X is an automation that controls how applications are built, tested, deployed, and promoted across different environments. Jenkins Pipeline plays a major role in achieving this.
Jenkins X extends Declarative Pipeline, so that a pipeline that embodies the best development practice discussed above is be represented in a Jenkinsfile that’s very simple, easy to understand, and declarative. This allows Jenkins X to leverage the ease of use of Blue Ocean and its visual editor to make it accessible for our target audience.
CLI provides the building blocks of common tasks, which helps keep Jenkinsfile small and concise.
Jenkins X reuses tooling from Draft so that language and framework specific “build packs” can be maintained which contain the default Dockerfile, Jenkinsfile and Helm chart files required to build, test, release and deploy different kinds of application.
When users create or import new applications, the build packs get applied to generate the default Dockerfile for building docker images, the Jenkinsfile for the CI / CD together with the Helm chart files to deploy it on Kubernetes.
Teams and communities can share and customise their build packs allowing builds, CI and CD to be automated for a wide range of languages and frameworks.
Helm charts are the standard packaging mechanism for installing and upgrading applications on Kubernetes. Jenkins X provides a Helm chart so it is easy to install Jenkins X on any kubernetes cluster.
Helm charts can also be composed and configured in various ways; so we can provide customised Jenkins X helm charts tailored to different public clouds so that Jenkins X can automatically make the best use of the available services on the cloud without the user needing to configure anything. E.g. so that Jenkins X can use Stackdriver on Google or CloudWatch on AWS for application and pipeline logs.
Traditionally the plugin model of Jenkins has been to add jars inside the JVM of the Jenkins master. With Kubernetes the extension model can be more flexible and allow extension through separate microservices and configurations distributed in Helm charts. These microservices then communicate with each other via Kubernetes resources which are exposed in the Kubernetes REST APIs, kubernetes language clients, Kubernetes Dashboard and the kubectl command line tool.
One goal of Jenkins X is to foster a community of Jenkins X addons which are optional Helm charts that can extend Jenkins X with additional capabilities; whether its for different source control repositories, issue management, code quality tools, testing tools, chat integration, UIs or operational management tooling.
E.g. we expect there to be Jenkins X addons like:
Nexus / Artifactory for Java artifacts
Chart Museum/Monocular for Helm chart repositories
Metrics and monitoring (e.g. Grafana and Prometheus)
Gitea for on premise git hosting
Kubernetes Native Jenkins
Jenkins X is not a general purpose Jenkins that can modified to do anything. Instead, it is tailored to focus on Kubernetes and Cloud Native use cases, hence the name “Kubernetes native Jenkins.”
Jenkins X includes Jenkins core with a specific set of plugins bundled together to provide the necessary user experience out of the box, without the user needing to “assemble their own LEGO blocks.”
Over time, we see an opportunity to improve Jenkins core itself based on our learning in Jenkins X, so that Jenkins itself can be used in more cloud native configurations. This should benefit not only Jenkins X but other uses of Jenkins. These changes will result in separate JEP proposals.
E.g. to allow storage of Jenkins resources (config, pipelines, builds, credentials, artifacts, logs) outside of the Jenkins master local file system to avoid a single point of failure and make managing persistent disk easier.
It is common in the Kubernetes ecosystem to store system state in highly available persistent Kubernetes resources (e.g. Pods, Services, ConfigMaps, Secrets etc). This allows microservices running on Kubernetes to integrate via Kubernetes resources. As part of the Kubernetes native Jenkins initiative we hope to increasingly use Kubernetes resources for the storage of Jenkins state. E.g. to represent Environments, Pipeline Activity (Runs) and Releases we will use Kubernetes resources (Custom Resources). We intend to extend this to other Jenkins resources like MultiBranchProject, Pipeline, Run, Credentials etc.
This will allow microservices on kubernetes to ‘plugin’ to Jenkins X in a similar way conceptually to the way current Jenkins java plugins but without the code having to be colocated inside the Jenkins master. Any microsevice with the right RBAC rules will be allowed to list, get, update, delete Jenkins resources via the kubernetes REST API, kubectl or the kubernetes language clients. E.g. so a ChatBot gateway component could watch for pipelines starting, changing, terminating and notify users of the state - without having to modify the Jenkins master.
Another area of extensibility is via Helm hooks which allows any kubernetes Job to be triggered after an application is installed or upgraded to perform any kind of test to ensure the new version is good. E.g. we can use Helm Hooks to implement quality gates to ensure a release is good enough in terms of operational monitoring of failures, performance and SLA metrics - if a release is not good enough we can automatically roll back.
UpdateBot is a CLI tool which automates updating downstream dependencies as part of a CI / CD pipeline by generating Pull Requests to eagerly push changes through repositories and give better feedback to upstream libraries/component developers if changes break the CI pipelines of downstream repositories.
E.g. as a developer of an upstream library, as you release new versions of your library the release pipeline can invoke UpdateBot to generate Pull Requests on all downstream repositories using your library. UpdateBot can then wait for those Pull Requests to trigger the downstream CI pipelines and pass/fail. If any downstream repositories CI jobs fail then your upstream library pipeline fails; giving valuable early feedback to both the upstream team making the change and downstream teams who become aware of which versions cause failures. We should make the waiting configurable so upstream development teams can choose which repositories to wait for and how long to wait before failing a pipeline.
Libraries and base docker images are often not deployed directly to Environments; but they are promoted into downstream repositories via Pull Requests changing dependency information in files like pom.xml, package.json, Dockerfile or helm charts etc. So you can think of UpdateBot as a tool for providing Continuous Delivery for libraries, base docker images or base helm charts.
Longer Term integrations: Prow & Argo
The kubernetes ecosystem uses Prow for its CI / CD infrastructure - which is a cloud native solution for handling events & webhooks from GitHub such as for processing comments and commands on Issues & Pull Requests. Prow can trigger arbitrary jobs on events; including Jenkins builds/pipelines. There are lots of similarities between the goals of Prow and Jenkins X (e.g. Kubernetes native and use of highly available Kubernetes resources to store state).
Argo is another useful technology which handles Kubernetes-native long-term human workflows such as approvals and promotions.
Longer term we’d like to integrate Prow and Argo more into Jenkins X as part of the CI / CD Platform for when they make sense; so that Jenkins X pipelines can use Prow to handle events and commands on issues & pull requests or Argo for longer term human approval of promotions in a kubernetes native way - while hiding the implementation details of Prow v Jenkins Pipeline v Argo from the user so the end user just gets awesome CI / CD for Kubernetes that just works and uses resources efficiently.
Kubernetes provides the ideal abstraction for building cloud native applications and running them at scale on any public cloud, laptop or data centre. It’s rapidly gaining traction in the industry, is adopted by all the major public clouds and it will play a key role in how people develop web apps tomorrow.
People developing apps on Kubernetes can today use Jenkins to do CI/CD, but it requires work. That’s true not just with Kubernetes, but with any development environment. Mobile apps, embedded apps, you name it. You know Jenkins, so you know what that is like --- finding the right set of plugins, designing the CD process, writing pipeline definitions, and so on. This is both the strength and the weakness of Jenkins.
But with Kubernetes lies an unique opportunity. It defines the structure and high-level constructs that we can leverage through Jenkins. When we integrate Kubernetes and Jenkins together, developers need not be familiar with how best to do CD on Kubernetes (most people are not) nor do they need to be familiar with Jenkins Pipeline (most people are not). This much better ease of use can make Kubernetes itself a much more attractive platform, and make Jenkins a de-facto CI/CD platform for Kubernetes. This is the motivation of Jenkins X.
Many of the reasonings for the design of Jenkins X comes from our experience with Fabric8, where we worked on a similar set of problems. The key points are discussed below.
Ease of Use
The focus on ease of use for the target audience drives a number of design choices.
First, in order to make Jenkins X approachable, we didn’t want to invent a whole new workflow, concept, or the way of interacting with Jenkins X. Instead, we choose to meet people where they are by adopting existing well-established practices, gluing them together, and amplifying the usefulness of them. For example, UpdateBot reflects the popularity of pull requests as a mechanism to propagate changes. Preview environments amplify the value of pull requests.
Second, it necessitates Jenkins X to take care of how people go from zero to productivity. Therefore, Jenkins X will pay a lot of attention to how we can make the first encounter smooth. This is why we put such emphasis on the CLI tool 'jx' to provision new Kubernetes cluster, new Jenkins, new environments, and then create or import new applications.
We also believe that automation should be a “butler”, providing backstage invisible assistance to make things go smoothly, as opposed to being front & center of the attention itself. From this perspective comes CLI, bot, etc.
CLI as Main Developer Experience
Emphasis on CLI also comes from this thinking. Kubernetes users are used to get things done with the Kubernetes CLI tool, kubectl. So we choose to expose Jenkins X data through Kubernetes by using its extensibility mechanism, which allows people to use Kubernetes CLI to interact with Jenkins X. Where this is not practical, Jenkins X defines its own CLI in a similar pattern, so that the familiarity with Kubernetes CLI will translate to Jenkins X CLI.
Longer term we hope to add jx into kubectl as a plugin to expose more of the capabilities of jx directly into kubectl so that Jenkins X and CI / CD feels a natural part of the kubernetes platform.
Jenkins in Kubernetes
While powerful, the existing extensibility model of Jenkins is not without challenges. Among other things, the fact that every piece of code needs to run inside a single process makes it difficult to isolate failures, scale better, and replace pieces.
So in Jenkins X, as we run Jenkins on Kubernetes, we aim to leverage the underlying capabilities of Kubernetes to solve these problems. For example, in Jenkins X, many features should be developed as independent microservices as add-ons.
This philosophy expands beyond the implementation details of Jenkins X, as discussed in the “helm charts” section. We intend this to become the next extensibility layer for Jenkins X.
Finally, achieving the level of integration and ease of use that we aim requires aggressive omission/demotion/hiding of certain features in Jenkins and addition of many more pieces at the same time. It is no longer an all-purpose Jenkins packaged for a specific platform. It is rather a purpose-built flavor of Jenkins for one purpose. This calls for an extension to Jenkins and a distribution with a name that’s related but separate from Jenkins, hence "Jenkins X."
The Jenkins X is a distribution of Jenkins and additional software for Kubernetes packaged as Helm charts and so it does not introduce any backwards compatibility issues on Jenkins itself.
From a Jenkins perspective the security model of the Jenkins master inside Jenkins X is that of the regular Jenkins distribution. We hope to make it configurable in the Helm chart to use either basic auth or OAuth.
The kubernetes resources used by Jenkins X will all follow the Kubernetes RBAC security approach.
There are no new infrastructure requirements related to this proposal - as we intend to use a hosted Jenkins X on Kubernetes to provide all of the CI / CD.
Jenkins X will add its own unit tests (e.g. in the jenkins-x/jx repo) together with its own BDD tests for testing the integration of Kubernetes + Jenkins X for CI / CD.
We also intend to use Jenkins X to host the CI / CD environment for Jenkins X; storing most of the results of these in GitHub (e.g. as comments on issues or Pull Requests or as releases in GitHub). Each Pull Request and merge to master should have its status updated via Jenkins X and be linked to the CI / CD pipelines.
e.g. each Pull Request on any Jenkins X git repository will run a CI pipeline to test the code change. When Pull Requests are merged to master then the CD pipeline will trigger to release the repository’s artifacts (docker images, binaries, helm charts etc).
At this early stage of the project there are no specific coverage or performance metric goals but we can figure that out in the community.
Jenkins X will consist of many repositories. While we can create them in the jenkinsci org, for practical reasons it’s easier to group them in a separate org, just like jenkins-infra and jenkins-docs are.
We have been putting all the github repositories for Jenkins X in the github organisation jenkins-x
The main repositories are: