JEP |
307 |
Title |
Evergreen Update Client/Server Lifecycle |
Sponsor |
|
Status |
Withdrawn ✋ |
Type |
Standards |
Created |
2018-04-17 |
BDFL-Delegate |
|
Requires |
JEP-300, JEP-303 |
Integral to Jenkins Evergreen is the client’s update lifecycle with the Evergreen hosted service layer. Starting with the first boot, the client must frequently inform the server of its current versions, retrieve new versions, and apply updates. This document specifies those API interactions and expectations to allow this update lifecycle to be performed correctly.
There are two services involved behind the scenes which are responsible for
facilitating the client’s update lifecycle: updates and versions.
The role of the versions service is simply to maintain an audit trail
for the versions of software that each client has installed locally. As clients
go through various upgrade processes, they will periodically POST
Version Manifest records which are stored in the Evergreen hosted service
layer’s storage.
The role of the update service is to use the essentials.yaml and retrieve the
latest versions of software for a given client UUID and compute the appropriate
update response to be sent to the client.
update versions evergreen
service service client
| | |
| | POST version manifest |
| | from local hpis/core |
| |<--------------------------o
| o-----------/ok/----------->|
| | |
| GET /update |
|<------------------------------------------------------o
| GET /versions for uuid | |
o-------------------------->| |
|<-----/200 json payload/---o |
| | |
| 200 with computed JSON update manifest |
| indicating what files should |
| be downloaded |
o------------------------------------------------------>|
| | | update versions evergreen
service service client
| | |
| GET /update |
|<------------------------------------------------------o
| GET /versions for uuid | |
o-------------------------->| |
|<-----/200 json payload/---o |
| | |
| 304 Not Modified |
o------------------------------------------------------>|
| | |In the backend datastore, updates must be considered immutable and sequential. These are referred to hereafter as "Update Levels". A single update record must have the following data structure:
-
id: Numerical sequential integer to identify the update -
commit: a Git commit corresponding to a change ofessentials.yamlin the jenkins-infra/evergreen repository -
channel: String denoting the Update Channel for the update. -
manifest: JSON formatted representation ofessentials.yamlfor ease of storage and query in PostgreSQL -
tainted: Boolean to indicate whether this update should be considered tainted and therefore not offered to clients. -
createdAt: Timestamp when this update was created.
For a given Update Channel, it is expected that each client "experiences" each
Update Level as time goes on, excepting tainted records.
in order to track a given update across the different channels, records should
be duplicated between different channels while maintaining the same commit.
For example, given an Update Level 1 (UL1) for the 'canary" channel. Once UL1
has been deemed safe to deploy to the next channel, "beta", a new Update record
will be created with all the same information as UL1, with except with a
differing id and channel. Thus allowing clients to enter a different
channel if desired, without re-installation.
The Update Channel is simply a means of segmenting updates to different types of clients. For example, testing or "dogfood" infrastructure would reasonably use a "canary" channel, receiving updates first, before they are added to subsequent channels.
The Update Channels are:
-
canary: bleeding edge updates, initial channel for all updates. -
beta: secondary channel for updates aftercanary, recommended for Jenkins contributors and other power-users. -
general: general use channel, with updates deployed after deemed sufficiently stable in previous channels.
While most users should default to the general channel, each instance must be
able to select its own channel via a user-supplied argument.
The responses sent to the client must be well-formed JSON documents, referred to as "update manifests" which the client must understand.
The Update Manifest should have a consistent structure which is given to but will be dynamically generated per client in order to ensure that the client is only downloading what is necessary to update that specific client.
{
"schema" : 1,
"meta" : {
"level" : 4,
"channel" : "general"
},
"core" : {
"url" : "https://update-cdn.example.com/some/path/to/a/jenkins.war",
"checksum" : {
"type" : "sha256",
"signature" : "somechecksumforthefile"
}
},
"plugins" : {
"updates" : [
{
"url" : "https://update-cdn.example.com/some/path/to/a/plugin.hpi",
"checksum" : {
"type" : "sha256",
"signature" : "somechecksumforthefile"
}
},
{
"url" : "https://update-cdn.example.com/some/path/to/another/plugin.hpi",
"checksum" : {
"type" : "sha256",
"signature" : "somechecksumforthefile"
}
}
]
},
"client" : {
"url" : "https://update-cdn.example.com/some/path/to/a/evergreen-client.tar.gz",
"checksum" : {
"type" : "sha256",
"signature" : "somechecksumforthefile"
}
}
}The four primary keys of the update manifest are:
-
metais an object which contains information about the instance’s update cycle itself, such as thechannelandlevel.of the enclosed manifest. -
corewhich indicates that a new jenkins.war is necessary. -
pluginswhich will include a list ofupdatesfor plugins. This is an object within the JSON structure rather than a flat array as it is expected that at some point in the future we may require aremoveslist to properly unpublish legacy or out-dated plugins from instances. -
clientwhich indicates a new tarball for upgrading theevergreen-clientitself.
Additional keys should be ignored by clients not supporting them to allow the Update Manifest to safely include things which are not yet supported.
|
ℹ️
|
There may be opportunities to cache the Update Manifest in the future, but this is considered a potential optimization which will be contingent on observation of real world usage for Jenkins Evergreen. |
The checksums provided in the Update Manifest are not generated or validated by Jenkins Evergreen but rather the Artifactory instance from which plugin, core, and other binaries are pulled.
In essence, every foo-1.2.3.hpi has a corresponding foo-1.2.3.hpi.sha256
file, the contents of which will be included as the checksum in the Update
Manifest to enable clients to perform archive integrity validation.
The client must perform the necessary downloading of items referenced in the Update Manifest and perform checksum validation before initiating a client update process. The exact sequence of events and what machinery must execute on the client is considered outside of the scope of this document.
The client should also post a new Version Manifest once an update lifecycle successfully completed to ensure that subsequent update check-ins result in accurate generated Update Manifest.
A version manifest is the symmetrically opposite of the Update Manifest in that it should include the actual versions of software present on a Jenkins Evergreen instance. This may include software which is outside of the update lifecycle.
The purpose of the version manifest is primarily for the client to report to the server a fairly accurate state of the installed software in the instance.
{
"schema" : 1,
"container" : {
"commit" : "sha1 of the built container",
"tools" : {
"node" : "output of node --version",
"npm" : "output of npm --version",
"java" : "output of java -version"
}
},
"client" : {
"version" : "version of evergreen-client"
},
"jenkins" : {
"core" : "jenkins.war embedded version",
"plugins" : {
"git" : "git.hpi embedded version",
"workflow-aggregator" : "workflow-aggregator.hpi embedded version"
}
}
}The client should also report container information, which is informational
rather than critical to the operation of the update lifecycle. This will be
used at a future point in time to better understand the runtime environments
for the Jenkins and evergreen-client processes.
In order for client to receive new Update Levels, an automated backend
process should generating an ingest.yaml to be sent to the Evergreen backend
service layer.
The ingest.yaml file should be machine-generated from the essentials.yaml
with URLs and checksums for artifacts at specific point in time. This provides
the raw data which the Update service should use to create
Update Levels. The file should be checked into source control and managed
via pull requests and automated changes to allow for thorough testing of the
set time-based snapshot of artifacts.
---
# This is an example of the output expected for incremental build information
# for consumption by the Evergreen backend service layer
##############################################################
# ISO-8601 timestmap for when this document was generated. This is to be used
# by the upload to the Evergreen backend services to understand when the ingest
# manifest was actually created (rather than commmitted to source control, for
# example)
timestamp: '2018-05-21T21:40:17+00:00'
# Core defines the latest incremental jenkins.war artifact.
core:
# The URL referenced doesn't need to be sourced through a CDN, Artifactory is
# suitable. Future versions of the Evergreen backend will need to point to a
# CDN automatically anyways.
urL: 'https://get.jenkins.io/war/latest/jenkins.war'
# The checksum is important for the Evergreen backend services, and client,
# to verify the artifact but also to distinguish effectively between two
# files which might be referenced in multiple ingest manifests which are in
# fact the same.
checksum:
# The type of supported checksum will need to be negotiated with the
# client-side support. Currently only sha256 is supported.
type: 'sha256'
signature: '246c298e9f9158f21b931e9781555ae83fcd7a46e509522e3770b9d5bdc88628'
# Plugins is an array of plugin records which represent the essential group of
# plugins to be distributed.
plugins:
- groupId: 'org.jenkins-ci.plugins'
artifactId: 'buildtriggerbadge'
url: 'https://updates.jenkins.io/download/plugins/buildtriggerbadge/2.9/buildtriggerbadge.hpi'
checksum:
type: 'sha256'
signature: '246c298e9f9158f21b931e9781555ae83fcd7a46e509522e3770b9d5bdc88628'
# Plugins defined under `environments` are expected to come from the
# `environments` key in the essentials.yaml. These plugins will follow the same
# structure as above and are intended to be joined with the "essential" group
# of plugins above for clients.
environments:
aws:
plugins:
- groupId: 'org.jenkins-ci.plugins'
artifactId: 'ec2'
url: 'https://updates.jenkins.io/download/plugins/ec2/1.39/ec2.hpi'
checksum:
type: 'sha256'
signature: '246c298e9f9158f21b931e9781555ae83fcd7a46e509522e3770b9d5bdc88628' update backend
service automation
| |
| [load ingest.yaml]
| |
| [convert to JSON]
| |
| PUT /update |
| with ingest JSON |
|<--------------------------o
o------/200 with JSON/----->|
| |{
"commit" : "0xdeadbeef",
"manifest" : "<ingest JSON>"
}{
"id" : 4,
"channel" : "general",
"tainted" : false,
"createdAt" : "<ISO 8601 timestamp>",
"manifest" : "<ingest JSON>",
"commit" : "<sha1>"
}The motivation for the Jenkins Evergreen distribution using this update lifecycle is largely driven by the goal for Jenkins Evergreen to be self-updating, which necessitates a different approach to code distribution compared to the conventional Update Center process.
The design described above is intended to be succinct enough to drive updates to Jenkins Evergreen, of which all instances are expected to be running the same approximate set of software. Contrasted to the Jenkins "Update Center" which provides much more metadata to provide user-visible information.
As Jenkins Evergreen is intended to update automatically, the metadata (Update Manifest), only needs to contain the URLs for packages and a checksum for validation. There are additional Security concerns and reasoning discussed below.
The Update Levels are a consideration to ensure that clients which have differing levels of connectivity consistency can be safely updated. Considering the following problem posed by Olivier:
.
Do you consider all updates as 'safe'? What happened if a client didn’t connect to the update service for month? Is it an information that would be useful in the update manifest?
One of the challenges for Jenkins Evergreen is determining how to handle updates for clients which are not consistently connected. If for example, a client is only connected to the Evergreen backend services layer once a week due to network misconfiguration, outages in the Evergreen services layer, or infrequent internet access, these instances should still be capable of safely updating their software.
Consider two instances, Alpha and Bravo. They both are created at the same time, at Update Level (UL) 1. Alpha stays online, and connected, for the next 14 days, while Bravo is disconnected until day 14.
Our state is now:
Alpha: UL14 Bravo: UL1
The first idea was to dry to have Bravo jump from UL1 → UL14 but with Jenkins Evergreen' testing process, this would effectively be a completely untested upgrade jump. This approach was considered too risky.
Another idea which was discussed was to use a git-bisect(1) type approach, trying UL14, if that fails, try UL7, and so on. This was also discarded as it would result in instances using completely untested upgrade paths, therefore too risky.
(contrary to what the JEP presently describes), and staggar the upgrade logic Bravo to where it can successfully go from UL1→UL2, then UL2→UL3, etc.
While there ome user experience concerns with downloading updates and restarting, at the present stage of development, this is considered an acceptable trade-off, safety rather than performance.
When considering security for Update Manifests, much of the research which was considered was around how traditional package managers consider their security challenges, such as the paper "A Look In the Mirror: Attacks on Package Managers" [1] and the design work done as part of "The Update Framework." [2]
The two major areas of concern for security with the update lifecycle are ensuring:
-
Update Manifests retrieved by the clients are themselves deemed authentic.
-
Packages suggested for the client to download are valid and legitimate.
For Update Manifests to be deemed authentic they must only be served over
TLS encrypted HTTP connections. Relying on the
Let’s Encrypt certificates provisioned for all
jenkins.io services.
To provide additional security, and protect against poisoned or fraudulent
jenkins.io certificates being used to distribute false Update Manifests, the
Jenkins Evergreen container will have a restricted set of trusted root
certificates. Trusting only the
root certificates used by Let’s
Encrypt, which are presently:
-
DST_Root_CA_X3.crt -
IdenTrust_Public_Sector_Root_CA_1.crt -
IdenTrust_Commercial_Root_CA_1.crt
(provided by the ca-certificates package on Debian 9 "Stretch")
The second concern is remedied by providing checksums from the distribution site in the Update Manifest. By ensuring that the client can trust the authenticity of the Update Manifest, the checksums will be trustworthy even in cases where the packages themselves are served through a CDN or mirror network.
The initial thinking relied on Public Key Pinning (PKP, also referred to as "pinning leaf certificates") in the client for the Update services. After cursory amounts of research, it is apparent that this approach is falling out of favor with leaders in this space such as Chromium moving away from PKP.
Another, related approach is referred to as HTTP-PKP. Which while it is possible to implement HTTP-PKP with Let’s Encrypt (also see this blog post). This approach was discarded as unnecessarily complex considering the client environment which is under control by Jenkins Evergreen.
GPG key exchange is a common approach used by package managers such as Apt and Yum. This approach was not strongly considered as the tooling for managing GPG keys from Node.js is lacking, and the use of such keys would add non-trivial amounts of complexity to the client/server design to accommodate proper key rotation and revocation.
Nothing additional outside of the existing requirements already for the Evergreen hosted service layer.
The prototype and actual implementation of this work is being performed in the jenkins-infra/evergreen repository.