Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
149 lines (105 sloc) 4.97 KB

IEP-9: Incremental builds Maven repository

Table 1. Metadata

IEP

9

Title

Incremental builds Maven repository

Author

R Tyler Croy

Status

💬 In-process

Type

Architecture

Created

2018-03-19

Discussions-To

infra@lists.jenkins-ci.org

Abstract

In order to support more rapid continuous delivery models, such as that described by Jenkins Essentials, Jenkins core and plugin builds must be deployed into a Maven repository much more incrementally rather than waiting for a developer to manually deploy a release to the existing releases [1] repository.

Specification

For a more incremental developer workflow, rather than attempting to make sense of potentially thousands of SNAPSHOT versions of artifacts in a repository. this document proposes a specific and tightly controlled Maven repository for such builds.

This would mean Artifactory would have an incrementals repository just for these kinds of pre-release builds. For example, rather than git-client-2.4.0-SNAPSHOT.jar the repository would contain git-client-2.4.0-a3dbf.jar, assuming a3dbf is the Git short-commit for the build.

Garbage Collection

Artifacts in the incrementals repository must be garbage collected to the most recent 5 artifacts or the most recent 30 days of artifacts. For example, if a plugin (io.jenkins.plugins.retrocean) has no new commits in a 30 day period, the io/jenkins/plugins/retrocean/ directory should have no more than the five most recent artifacts stored. More active plugins should have no more than 30 days worth of artifacts stored.

ℹ️

Plugin and core tooling should use a consistent format for defining incremental built versions, such as: <artifactId>-<major>.<minor>.<patch>-<sha1>.jar. The specific format is not required for this document.

Motivation

By adding this new, somewhat ephemeral, Maven repository, we can support newer development workflows in the Jenkins project without adversely affecting the existing "mainstream" development workflow of Jenkins plugins.

As mentioned briefly in Costs, overhead is sufficiently low to adding that adding a new Maven repository, even if it’s experimental and eventually abandoned, is worth the exploration.

Rationale

A Maven repository hosted in Artifactory, alongside the releases and a number of other repositories, results in the simplest to deploy, and simplest to consume "bucket of artifacts" we currently have at our disposal.

In order to avoid over-reliance on these "incremental builds", the artifacts themselves should be expected to be deleted after the specified "garbage collection" period.

Alternate Approaches

Azure Storage Container

One alternative approach which was briefly discussed in person with Carlos Sanchez was to drop these "incremental build" artifacts directly into an Azure storage container.

This approach was rejected as it would require additional non-native tooling to be used in the development workflow. By adopting a Maven repository, existing tools for grabbing Maven dependencies can be utilized by developers wishing to incorporate "incremental builds" into their build and test workflows.

Existing Maven Repository

Re-using the existing releases Maven repository was explicitly not considered as an alternative approach as the Jenkins infrastructure team has had multiple performance issues [2] with that repository over the past couple of months. Most notably, the indexing times in the repository have varied between 10 minutes and over 60 minutes due to opaque server-side issues. The consequence of adding more load, and many more artifacts, to the repository would severely affect the indexing time and adversely affect the ability for Jenkins users to receive new plugin updates in the Update Center [3].

Costs

As Artifactory is a hosted service provided by JFrog, it is not expected that any additional financial cost will be involved in this proposal.

Additionally, since this document proposes an additional repository, it is not expected to incur any substantial runtime/performance cost on the existing Update Center and releases repository indexing process.

Reference implementation

Since we have no staging implementation of Artifactory, there is no reference implementation, acceptance of this IEP document would result in a new repository being created on repo.jenkins-ci.org.

You can’t perform that action at this time.