Skip to content

Continuous Delivery Details

Jump to bottom
Szczepan Faber edited this page Apr 10, 2017 · 7 revisions

Mockito champions true continuous delivery in the Open Source.

Introduction

For motivation see Continuous Delivery Overview page.

In early 2017 the team decided to change the release model, for more details see Continuous Delivery Pipeline 2.0.

Since Mockito moved to github circa 2012, the team wanted to automate the release process. At that time the build script was based on ant, after the repository migration the build script was progressively migrated to gradle. Things got interesting in 2014 as Szczepan started to experiment continuous delivery of Mockito.

Continuous Delivery allows interested parties to access to the latest versions including betas, thus allowing a drastically shortened feedback cycle. Examples of quick feedback cycles are issues like #360, #335 or #237. All these issues pointed to specific beta versions which allowed us to track down very quickly which pull request or commit was the culprit.

Continuous Delivery

We call it Continuous Delivery Drone. You can view all source code of the drone in the buildSrc and other gradle scripts.

The script enabled the automatic generation a release and propagate it on the bintray and maven distribution channels. That included the release note generation as well ; at first it was based on commits but now it uses the GitHub API to get Issues / Pull Request data.

The first releases to benefit from continuous delivery were the 1.10.x. For each new change on master, either a bunch of commits pushed or PR merge, the drone kicked in and performed the release and incremented the version number.

Versioning

Before the introduction of the continuous delivery drone, the team intuitively followed the semver scheme. Upon the the introduction of the introduction of the continuous delivery, the version scheme was the same. The 1.10.x line, could follow semver x being the build number :

1.10.<build number>

That leads to the following qestion...

Why artifact 2.0.0 does not exist and 2.1.0 is the first release of Mockito 2.x ?

tldr: the first release of Mockito 2.X is 2.1.0 and you can download this artifact from maven central.

Origin of the problem

To answer this question, we need to go back in December 2014. With this commit the version scheme didn't change, the beta version was just marked with the beta marker (label) to indicate Mockito 2.x beta builds had started.

This lead to versions like :

2.0.<build number>-beta

Indeed the official semver definition also states that

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

While this notion is seemingly innocent, we were now unable to publish Mockito 2.0.0: maven and gradle with version ranges would fetch the very latest beta release instead of our official release! Moreover, maven-central indicated that, while 2.0.0 was published later, it was superseded in various patch updates. e.g.

1.10.18 < 1.10.19 < 2.0.0-beta < 2.0.0 < 2.0.1-beta < 2.0.111-beta

It took us quite some time to address that problem during the 2.x beta phase. But we did :

Resolution

There were multiple workarounds to this problem (see #595).

  • The first workaround considered was unpublishing the beta releases of Mockito. However, the recent problems of such an invasive action with left-pad in the NPM ecosystem (read more) made clear that this was not a reasonable solution. Removing beta releases would break a significant number of projects, which is unacceptable.

  • The second solution considered was to publish 2.0.0 anyways and describe the version caveat clearly in the release notes and in the upgrade guide. While this is certainly possible, the issue of maven central indicating the incorrect latest version remains. Moreover, end-users might not read the notes or guide and download the incorrect beta version thinking of it as next verion after 2.0.0.

  • Lastly we considered skipping releasing Mockito 2.0.0 and instead jump to 2.1.0 immediately.

    This is the solution Mockito team chose.

    While this is certainly confusing at first, it does not break applications of end-users and fixes the issue with maven central.

The chosen solution also tweaked the version to append the build number after the beta label. e.g.

1.10.19 < 2.0.0-beta < 2.0.1-beta < 2.0.111-beta < 2.1.0-beta.112 < 2.1.0 < 2.2.0.beta.1 < 2.2.0 < ...

Lessons learned

  • If you are employing continuous delivery and publishing beta builds or wishes to set it up on your project, make sure to not increment the patch version. Instead, append the increment after the tag: 2.0.0-beta.15. If you realized at this point you are using a problematic version scheme, do not remove the published artifacts. Look for a solution that is the least invasive to the end-users. This might differ from package-manager ecosystem and individual libraries/frameworks.

  • When you get the version scheme and continuous delivery correct, the benefits are very valuable. The feedback cycle is greatly shortened and debugging an issue can be done a lot quicker. End-users or contributors can easily bisect an issue by switching artifacts rather easily, instead of building source and using git bisect. This lowers the bar for end-users resulting in more and precise feedback.

Too many beta versions!

Publishing every single beta version caused confusion. Due to life priorities the active work on Mockito 2 was unfortunately intermittent. This caused prolonged beta cycle with hundreds of beta versions in Maven Central. This caused confusion and issue #595 discusses this problem at length. The issue will addressed by #618.

Clone this wiki locally