Mockito Continuous Delivery Pipeline 2.0 #911

Open
szczepiq opened this Issue Jan 29, 2017 · 9 comments

Projects

None yet

5 participants

@szczepiq
Member
szczepiq commented Jan 29, 2017 edited

Proposed changes on easy-to-grasp diagrams are documented on Mockito blog article.

Current release model

Every code change results in a new version in Central repository (not snapshot, a real version). More information on Continuous Delivery in Mockito.

Given high rate of new versions, the community and current tooling is not comfortable in taking in new Mockito versions at fast pace. More feedback and the discussion in issue #618.

Proposed changes

Release every change but not all releases go to Central. Push to standard repository (JCenter / Maven Central) only substantial releases at ~monthly cadence. Push remaining versions to less prominent, but still public repository for early adopters.

Glossary: What are major, minor, patch versions? Explanatory example: Version 2.6.5 has major=2, minor=6, patch=5.

🚀 Feedback Needed 🚀

  1. Every version lands in Bintray repo called “community”. That repository is not automatically linked to JCenter / Maven Central (e.g. 2.7.1, 2.7.2).
  2. Every minor or major version change lands in both “community” and “production” (e.g. 2.8.0, 2.9.0, 3.0.0).
  3. For critical bugfixes, patch version change can be published to “production” when commit message contains "[production release]”. This way we can push 2.8.1 to “production” repo if needed.
  4. Every month, last patch version is automatically promoted from community to production. Example: “community” repo has 2.8.1, 2.8.2, 2.8.3, at the end of the month, 2.8.3 is promoted to production.

Note that above design has not yet been prototyped so it can change based on the available implementation options. I would love some early feedback about the plan above. I'm not completely happy with repository naming "community" and "production" as it seems to indicate that community versions are not ready for production (which is not true).

Action list

  • - release notes indicate the repository the artifact is published (mockito/mockito-release-tools#4)
  • - release notes have clean format (mockito/mockito-release-tools#3)
  • - notable release (target repo) can be forced by commit message
  • - procedure of promoting to notable release is documented
  • - the new release model is comprehensively documented
  • - release plugin automatically publishes to correct repo
  • - TravisCI cron job is configured to promote notable release on monthly basis

Implementation details

This section documents the current behavior of the release process.

New version is published under following conditions:

  • There is a commit(s) on main dev branch. Currently the main branch is "release/2.x”.
  • Travis CI build is triggered (if commit message contains “[ci skip]”, the build is not triggered)
  • There is a change to the “code” that is part of the binary. For example, following changes will _not_trigger a release: a change to the README.md, adding new test class. Following changes will trigger a release: code change in production class, adding a space to Javadoc in some java class, changing project version in “version.properties” file.
  • Environmental variable "SKIP_RELEASE" is not present
  • Commit message does not contain "[ci skip-release]"
@szczepiq szczepiq changed the title from Document and change release cadence to Change and document release cadence Jan 29, 2017
@szczepiq szczepiq self-assigned this Jan 29, 2017
@szczepiq szczepiq changed the title from Change and document release cadence to Change Mockito release cadence Jan 31, 2017
@TimvdLippe
Contributor

Every minor or major version change lands in both

I do agree for major, but for minor I am not sure. I think we should not immediately release a new minor version when 1 function was added, most notably because we might want to add a range of functions in all separate PRs (since they are separate logical units). Therefore I would say to not do it automatically, but presumably faster than with patch releases. E.g. minor every week published, patch every month.

Example: “community” repo has 2.8.1, 2.8.2, 2.8.3, at the end of the month, 2.8.3 is promoted to production.

This means that 2.81 and 2.8.2 are never available on production? I do think we should push this for 1. consistency in release versions and 2. if someone finds an issue with a version in production, he/she can bisect using the in-between patch releases to see which version contained the error.

I'm not completely happy with repository naming "community" and "production"

Maybe trunk and production?

New version is published under following conditions

I would like to add an override here for to be able to push a new version. E.g. not pushing an existing version to production, but publishing a new version without manually editing javadoc to artificially trigger the release mechanism

Overall I agree with the proposal, just some small comments. Glad that we are taking action in this direction 👍

@szczepiq
Member

Great feedback thank you!

I do agree for major, but for minor I am not sure. I think we should not immediately release a new minor version when 1 function was added, most notably because we might want to add a range of functions in all separate PRs (since they are separate logical units).

"minor" version (e.g. middle number :) is not automatically bumped by the system. It's us (or the PR submitter) that makes a judgement call to explicitly bump minor version. This decision is based on the gravity of the changes. We don't bump minor version unless it has substantial changes. If it does have substantial changes, it probably should be released to mass audience. At least this is my reasoning. If we don't automatically publish minor versions we can end up with versioning scheme where "community" and "prod" repo have pretty much random versions. If we publish minor versions automatically, "prod" contains announcable, significant versions that add substantial value, e.g. 3.1.0, 3.2.0, 3.3.0 and occasional patch release like 3.3.1. "community" repo would have all changes 3.1.0, 3.1.1, 3.1.2, 3.1.3, 3.2.0, etc. Thoughts?

This means that 2.8.1 and 2.8.2 are never available on production? I do think we should push this for 1. consistency in release versions and 2. if someone finds an issue with a version in production, he/she can bisect using the in-between patch releases to see which version contained the error.

The idea is to only include substantial versions in official repo. If the user wants to bisect, he needs configure "community" repo explicitly in pom / build.gradle. In short, the answer to your question is yes, 2.8.1 and 2.8.2 will never be in "production" repo.

Maybe trunk and production?

Let's keep thinking. Some other ideas: all-versions/announcable-versions, all-artifacts/significant-artifacts, early-adopters/slow-adopters.

Given that "minor" version (e.g. middle one) is updated explicitly by the PR submitter, he controls

I would like to add an override here for to be able to push a new version.

Good idea. I will add it to the design.

Really nice feedback. Thanks! We can tweak/iterate the process as we learn more, too.

@TimvdLippe
Contributor

We don't bump minor version unless it has substantial changes.

Makes sense, 👍

In short, the answer to your question is yes, 2.8.1 and 2.8.2 will never be in "production" repo.

I think for clarity's sake, they should be pushed. It would be weird to browse maven central and see missing versions. Let's see what the others think.

Let's keep thinking.

Latest - Stable?

@arend-von-reinersdorff

In short, the answer to your question is yes, 2.8.1 and 2.8.2 will never be in "production" repo.

I think for clarity's sake, they should be pushed. It would be weird to browse maven central and see missing versions. Let's see what the others think.

I can think of two popular projects which sometines skip versions:

  • Maven, missing versions eg 3.3.2, 3.3.4-3.3.8 (and upcoming 3.5.0 will skip 3.4.0)
  • Tomcat, missing versions eg 8.5.10, 8.5.7, 8.0.40
@szczepiq
Member
szczepiq commented Feb 1, 2017

@arend-von-reinersdorff, thank you for feedback and examples!
@TimvdLippe, currently, I most like following naming scheme for repositories: "all-versions" and "notable-versions".

@szczepiq szczepiq changed the title from Change Mockito release cadence to Mockito Continuous Delivery Pipeline 2.0 Feb 2, 2017
@tinkerware

This looks good! I agree with the model where minor versions are in release channel (notable-versions) and patch versions are in beta channel (all-versions), modulo hot fixes. If patch versions are pushed to notable-versions at the time they are released, you defeat the purpose of the release channel; your "notable versions" channel would churn multiple times daily again.

Maybe Mockito can push all interleaved patch versions available only on the beta channel to the release channel when the release channel is updated? For example:

Before:

release channel – 2.3.0
beta – 2.3.0, 2.3.1, 2.3.2

After:

release channel – 2.3.0, 2.3.1, 2.3.2, 2.3.3 (hotfix)
beta – 2.3.0, 2.3.1, 2.3.2, 2.3.3

This may go against the principle of least surprise, since someone browsing the version history of release channel will notice that suddenly not one but three new versions became visible. This may be acceptable if bisecting and keeping full version history on the release channel are important for the project. As I said before, I don't have a strong preference for this model, except to say that patch versions should not appear in the release channel at the time of their release unless they are hot fixes.

@mattnelson

I have an alternative opinion from everything proposed so far. I propose introducing the usage of the classifier attribute to differentiate between community and production releases. With this proposal the current CD procedure would be left unchanged and consumers who want to subscribe to the latest releases can continue to do so with no changes. For those consumers who want to limit the thrashing on taking updates they can do so by declaring a weekly classifier. The only change to the current release process would to be to release the most recent tag with the additional classifier attached. It could be weekly or monthly.

<!-- Current usage, unchanged! -->
<dependency>
    <groupId>org.mockito</groupId>
    <artifactId>mockito-core</artifactId>
    <version>2.7.5</version>
</dependency>

<!-- Using the `weekly` classifier -->
<dependency>
    <groupId>org.mockito</groupId>
    <artifactId>mockito-core</artifactId>
    <classifier>weekly</classifier>
    <version>2.7.1</version>
</dependency>

This could also be inverted to restore the delayed release process to the majority of consumers and allow opting in to the daily/continuous releases.

<dependency>
    <groupId>org.mockito</groupId>
    <artifactId>mockito-core</artifactId>
    <classifier>daily</classifier>
    <version>2.7.5</version>
</dependency>

<dependency>
    <groupId>org.mockito</groupId>
    <artifactId>mockito-core</artifactId>
    <version>2.7.1</version>
</dependency>
@TimvdLippe
Contributor

@mattnelson That is an interesting concept I had not heard of! I am not sure though if it would be valid for our usecase, but definitely worth investigating.

@szczepiq
Member

@mattnelson, interesting idea! At this point I want avoid using classifiers because they would complicate things. If you get this working for some project or there is a reference project that uses this method, we would like to know the experiences.

@tinkerware, thanks for feedback!

@szczepiq szczepiq referenced this issue in mockito/mockito-release-tools Feb 12, 2017
Open

Separate concise release notes file #9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment