Switch branches/tags
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
README.adoc

README.adoc

JEP-309: Bill of Materials

Abstract

There are multiple ways of defining packaging and what combinations of core, modules, libraries and plugins to test from multiple repositories, forks and PRs. The Bill of Materials (BoM) aims to set a common language that can be reused across them.

The BoM is a cornerstone to enable these other goals:

  • Ensure value gets delivered while quality is continuously improved

  • Do not go down the rabbit hole of spending weeks without delivering value

  • Increase quality of non-LTS builds

  • Provide an answer to “Is this code ready to be delivered to Jenkins Evergreen?”

  • Avoid complex matrix testing that is the objective of Evergreen tracing and monitoring

  • Feedback delivered to PR author as soon as possible

  • Automated releases - at least from the perspective of the engineer

  • Reduce code review overheads so that maintainers can process more. AKA “I am a plugin maintainer, not a FindBugs robot. Automate checks for me”

Specification

What are we missing

  • Builds/Tests from master, needed for Evergreen

  • Builds/Tests from core, libraries and plugins PRs

These cases are addressed by a Custom WAR Packager for custom WAR Building and testing use-cases. That format is narrow focused, and it does not address use-cases like support of custom environments. We want to unify the description format across projects within Jenkins (e.g. Jenkins Evergreen, Jenkins X, Docker Packaging, etc.).

Bill of Materials (BoM)

The BoM defines what commits of core, bundled libraries (stapler, remoting) and plugins are delivered and tested together. They key part is the usage of commit ids and not releases. The usage of the maven release plugin does not fit this requirement.

The BoM needs to be trackable in Git.

The BoM needs to have the concept of environments for Evergreen. Ie. install different plugins in AWS vs Azure vs…​

A requirement for Evergreen is that parts of the BoM need to be downloadable. Evergreen needs to enable selective downloads of core, bundled libraries and plugins for incremental upgrades. Pushing into maven repo is fine, although indexing time is big. Will need to mirror into Azure CDN at some point.

The BoM needs to be translated to take advantage of existing infrastructure.

  • Evergreen needs to generate a diff for upgrades and download the bits

  • Existing ATH uses a war binary that can be built from BoM

Packages should be trackable back to a commit id, but during development the BoM should point to branches or tags, so we would have a BoM as input and a effective BoM as output with all the git refs resolved to specific commit ids.

Workflow

Core, libraries and plugins could have a BoM in the master branch. This BoM should point to master of core, master of libraries. There is an implicit dependency to the current branch of the current git repository.

The BoM should be processed to update the Maven pom on change events.

Changes to Core

Developer needs some changes to core or Jenkins internal library or component for downstream use.

  • Create PR against core/library.

    • PR builder in core/library will publish the artifacts using the git commit id as part of the identifier, for downstream consumption.

    • Would also publish the result BoM with all git references converted to ids.

  • Create PR against a plugin

    • This PR would include a modified BoM pointing to the PR branch of core/library.

  • PR builder for plugin/library would process the BoM, creating as output:

    • A Jenkins package (WAR and/or Docker image) with the dependencies stated in the BoM.

    • The realized BoM, with all git references converted to ids.

    • Dependencies would not need to be rebuilt as they were already published.

  • Tests run against this output

  • Output artifact is published to repository, for downstream consumption in a way that can be fetched by git commmit id.

Changes to Plugins

Same as above but instead of pointing to core/library PR, pointing to master.

Continuous Delivery

Changes in master of core and libraries should trigger downstream rebuilds of plugins. Given the amount of work involved we recommend doing it in phases, targeting core plugins first, as defined in Evergreen.

BoM Format

A new yaml format based on essentials and Custom WAR packager using the Kubernetes format.

YAML example

version: 1.0
metadata:
  # labels and annotations are key: value string attributes
  labels:
    name: myplugin
    groupId: ...
    artifactId: ...
    version: ...
  annotations:
    io.jenkins.x: y
spec:
  core:
    # version OR version + ref (version just to keep Maven happy about version order)
    ref: master
    version: 1.0
  components:
    - groupId: org.acme
      artifactId: acme-component
      ref: master
      version: 1.0
  plugins:
    - groupId: org.acme
      artifactId: acme-plugin
      ref: master
      # version: 1.0
  environments:
    # environments get added to the other section when enabled
    - name: aws
      plugins:
        - groupId: org.acme
          artifactId: acme2-plugin
          ref: master
          # version: 1.0
      components: ...
  # other sections can be added and ignored by default
# the realized BoM after refs are resolved
status:
  core:
    ref: aaabbb
    # version: 1.0
  plugins:
    - groupId: org.acme
      artifactId: acme-plugin
      ref: bbbccc
      # version: 1.0
  environments:
    - name: aws
      plugins:
        - groupId: org.acme
          artifactId: acme2-plugin
          ref: cccddd
          # version: 1.0

Version

version field defines version of the BOM format. This JEP defines the version 1.0, new formats may be introduced in subsequent JEPs.

The specification version follows the Semantic Versioning 2.0.0 approach, but defines a 2-digit specification since there is no "bugfixes" planned. Incompatible changes will be always done along with a major version change.

Field: metadata

Metadata includes 2 key-value maps: labels and annotations. Both sections are optional. More new keys may be added in future BOM specification versions (see BOM Versioning), and implementations should ignore them if they hit unsupported metadata entries.

Field: labels

Labels implement fields which will be used during the build. All labels are optional from the BOM specification standpoint, but implementations may define special requirements.

There are following recommended labels:

  • name - Short string description of the bundle

  • description - Longer text description of the bundle

  • groupId - Maven Group ID of the bundle

  • artifactId - Maven Artifact ID of the bundle

  • vendor - Short description of the bundle’s vendor (e.g. Jenkins project)

All label values must be Strings.

Field: annotation

Additional metadata, which is not used during the build directly.

  • All metadata entries are optional.

  • Metadata keys should use the dot-separated strings, e.g. io.jenkins.demo.mybundle.notForProduction.

  • Metadata values should be always plain strings

The expectation from BOM packaging implementations is that they take annotations and somehow make it available to users.

Field: specification

Specification defines contents of the bundle. It consists of the following sections:

  • core - Defines the core, this is a mandatory section

  • plugins - Jenkins plugins in the bundle

  • components - Defines a component (anything excepting a plugin)

  • environments - Environment-specific components

Dependency type

Fields below use a similar dependency format.

  • groupId - Maven group ID

  • artifactId - Maven artifact ID

  • type - Type of the packaging. It may be implied or required depending on the field

  • version - Version to be used

Field: core

Defines source of the Jenkins core to be used. Depending on the BOM packaging implementation, it may be referring WAR or other packaging type.

Implied type: war

Field: plugins

Implied type: hpi

Field: components

This section defines all other components which may be used in the package. The components are classified by the type field values, and these field may be interpreted differently by implementations. Type examples:

  • jar - JAR library

    • This type may be used to define extra libraries which are included into the package

  • jenkins-module - Jenkins core module

    • Modules represent parts of the Jenkins core which have their own release cycle. They are always bundled into the core, but they are not used by the core directly.

  • groovy-hook - A package of Groovy Hook Scripts

  • jcasc - A package of

If a BOM packager implementation support modifying the WAR file, these fields may be used to define components to be included/replaced. Other implementations may ignore the section, fail or handle components differently.

Implied type: jar

Field: environments

Environments allow defining various packaging approaches for different targets. For example, BOM may define different environment specific plugins to be bundled for AWS and Azure: Agent provisioners, artifact manager implementations, etc.

Fields:

  • name - Name of the environment, e.g. aws or k8s. This field is mandatory

  • plugins - Jenkins plugins in the bundle

  • modules - Jenkins core modules to be bundled

  • components - Defines extra components to be bundled

BOM packagers are expected to support passing environments as a build argument. The following rules apply:

  • Environments can add new plugins/components

  • Environments can update plugin/component versions defined in the main BOM section, e.g. they may require newer plugin version

  • Environments cannot remove plugins/components

  • Environments cannot downgrade plugin/component versions

Dependency resolution

Plugins, modules and components may declare dependencies on other components. There is no strict requirement for BOM to provide a full list of dependencies, although it is recommended for BOM usages in production packages.

Dependency resolution logic is not specified in this BOM format version, it is up to the implementations. Similarly, implementations are responsible for upper bounds dependency resolution and checks if they support transitive dependencies. Packaging implementations may also refuse to support transitive dependencies.

Field: status

This field represents the resulting BOM generated by a BOM packager implementation. This field is equivalent to specification, but there are additional requirements:

  • All dependencies must be resolved in a reproducible way. There is no dir specifications pointing to sources not packaged with the resulting distribution

  • All transitive dependencies must be resolved and added to status

Particular BOM packager implementations may squash the environments section if they build a package targeting a single environment.

Motivation

The current approach to make changes in core, libraries and plugins is too cumbersome, far from Continuous Integration and complicated for contributors, due to the usage of multiple repositories.

Changes typically span more than one repository, causing contributors to manually combine different PRs together. The goal of this proposal is to move towards a master based delivery process, ensure that core changes don’t break plugins and that core changes needed by plugins can be quickly and safely adopted.

This proposal builds on the goals of Evergreen. We want to ensure that the Evergreen distribution is continuously delivered, off master, and is done safely with a set of checks that run automatically.

Reasoning

The chosen YAML format is just picked due to the similarities with Kubernetes model objects and has no importance. Since the format does not support all required features, it has been extended to support them.

Plugin and component sources

The implementation should support pulling in components by version from different sources. It includes version-based dependencies and also path definitions for local builds.

For version naming there are other options:

  • Use standard releases in Jenkins repository

  • Use Incremental releases (JEP-305)

  • Use Maven SNAPSHOTS

    • Automatically deploy snapshots using commit ids (ie. jenkins-core:aabbcc-SNAPSHOT)

    • Ensure the commit ids are included in the packaging and visible during builds

  • Use git modules to point to the master and PR commits

    • And build everything every time

    • This would not work for Evergreen as the components need to be individually downloadable.

For path-based naming an dir field will be used. This field will support defining absolute and relative paths to sources of the component. Build of these sources is a responsibility of BOM implementations, support of such paths is not mandatory, implementations may reject the field.

Format versioning

According to the BOM discussion feedback, BOM format may change in the future in an incompatible way. In order to support that, a version field is introduced in the format.

Tooling

During the JEP discussion, it was proposed to introduce some tools in order to simplify usage of the model. It includes:

  • Utility Java library, which allows parsing and generating BOM

  • YAML schema, which will allow verifying the BOM formats

As a part of this JEP, it was decided to NOT include separate tooling to the scope of this JEP. There is a Jenkins Custom WAR Packager library which offers BOM Model and utility methods. It can be used as a utility library for BOM format 1.0. More tooling can be implemented on-demand.

Hierarchical objects in annotations

There was a comment that annotations should support hierarchical objects to be more flexible. As a part of format 1.0, it was decided to NOT do that in order to keep YAML processing implementations simple. It can be implemented in new format versions.

Backwards Compatibility

This proposal aims to add new functionality and reuse existing tooling by generating Maven poms and other formats in use today. BOM format versioning is supported by the version field in YAML.

Security

There are no security risks related to this proposal.

Infrastructure Requirements

There are no new infrastructure requirements related to this proposal.

Testing

There are no testing issues related to this proposal. Custom WAR Packager and Evergreen have test automation which verifies support of YAML formats.

Prototype implementation