Mockito continuous delivery model

[mockitoguy]

This issue discusses:

• Mockito continuous delivery model (e.g. publishing every PR: wiki link)
• How to prevent prolonged Beta phases as it happened for v2

[TimvdLippe]

Strict set of issues we want to solve and stick to that. Once they are resolved, publish the version and define a new strict set.

We should not add issues to the current milestone, unless they are absolutely n1 priority if it is a huge bug. But this probably rarely happens.

E.g. for 2.2.0 we are already set, I think this list is long enough: https://github.com/mockito/mockito/milestone/3. This milestone is very clear in its goal: improve the ArgumentCaptor API. That we have a release focused on an issue. For major releases, we can of course adopt more than 1 focus. That's my 2 cents.

[mockitoguy]

Good feedback. I'm not worried about v2 releases. Since they are compatible, we can simply iterate minor or patch version and release without any beta versions.

However, 3.0.0-Beta.x worries me. What can we do to avoid prolonged v3 beta phase.

[bric3]

I think the same would apply e.g. define a set of goals for 3.0. Currently 3.0 scope is a bit fuzzy, we know that we want Java 8, maybe JUnit 5.

Also note that the prolonged 2.0 phase was also due to our life. You had to move to LinkedIn, I was overwhelmed by own professional agenda. That is somehow outside our control. I don't know about your plans about mockito in the future but on my side I have more spare time recently, but that may change.

[szpak]

I think the same would apply e.g. define a set of goals for 3.0.

Having set a minimal scope with should be done before 3.0 is a good idea. However, I would not limit accepted changes to just those things. Any sensible PR should be merged (as 2.x will be in the maintenance mode). In general, a lot of versions (here beta) is a side effect of continuous delivery. On the other hand I prefer it over working with snapshots/jitpack just to have given fresh feature available in my tests.

[TimvdLippe]

If there is a PR which can be merged as-is, yes of course we should merge it. However, I think the current open PRs regarding ArgumentCaptor is what I meant with the strict set. Since this class was not included in 2.0.0, we decided to move all these PRs to 2.2.0. If there is a lot of work tight to a certain feature (or set of features), I do not think we should blatantly accept it for 3.0.0. That way we can deliver 3.0.0 on time, while also have a solid plan for later releases.

Of course we have to keep in mind the semver scheme. If we know there are breaking changes required, we might have to do those together in 3.0.0 because we can't introduce them in 3.1.0.

[mockitoguy]

Feedback from users: #595 (comment)

[mockitoguy]

Perhaps we should avoid pushing beta versions to central / jcenter and only push them to internal bintray repo.

[bric3]

Bintray can serve as a maven repo I believe, https://bintray.com/docs/usermanual/formats/formats_mavenrepositories.html#_working_with_maven

So that may be a solution.

[mockitoguy]

Options to consider:

1. don't publish Betas at all
2. don't publish to central, publish to some less prominent public repo
3. publish some betas to central, manually chosen
4. publish some betas to central, automatically at certain cadence / when specific release milestones are completed

I hope 1 year from now we won't be publishing 2.1.0-RC.123 :)

[TimvdLippe]

On master I bumped the version properties to 3.0.0-beta.1 and disabled the release procedure for now until we have developed a stable 3.X beta and resolved this issue.

[bric3]

I think we should address this issue before any further development.

[TimvdLippe]

I think the cleanest solution to have a separate bintray project mockito-beta which has all beta releases. This should be fairly easy to build in our release process, is minimally impactful for our beta testers (it's just 1 word change) and does not clutter the official releases.

[ChristianSchwarz]

Options to consider:

1. don't publish Betas at all
2. don't publish to central, publish to some less prominent public repo
3. publish some betas to central, manually chosen
4. publish some betas to central, automatically at certain cadence / when specific release milestones are completed

5 . release beta's as different artifact e.g. mockito-core-beta to central?

[TimvdLippe]

@ChristianSchwarz Yes that was the option I described in #618 (comment)

[bric3]

I like the idea of beta specific maven coordinate. However not sure we need another bintray account. Though it's not an issue if we have one. Anyway we will have to create one mockito account anyway see #631

[raphw]

Or we do it the official Maven way and publish SNAPSHOT versions to a snapshot repository.

[szpak]

SNAPSHOTs were published in the past. I personally don't like and don't trust them. With beta releases it is the package that will not change and will be available "forever". Snapshots (even those with timestamp) can are cleaned up after some time.

A separate repo could be a solution if we really plan to have a plethora of beta versions again :).

[mockitoguy]

Hearty -1 to different artifact name. It's very inconventional and confusing. It could lead problems with conflict resolution (both beta and normal artifact on the same classpath).

Separate bintray repo is the way to go.

However, it feels that separate bintray repo is not enough. We also need to plan releases better and drive them to resolution faster than in 1.5 years :)

[TimvdLippe]

I will set up a Bintray account this weekend for #631 and then we can configure it 😄

[TimvdLippe]

Oh forgot to add regarding the number of releases: The issue for Mockito 2 was that no strict feature set was defined. If we define a strict set and work towards finishing it, it should be possible to publish releases every ~2 months. This is a lot better than the 1.5 years :P. I think we already defined what Mockito 2.2.0 must incorporate, so we can start working on that in the release/2.x branch.

[mockitoguy]

Mockito 2 continously publishes. Hence, there is no risk of prolonged beta :)

[mockitoguy]

Hiding betas from default repositories and having strict feature set for milestones is not good, nor sufficient solution. We no longer continuously deliver which means harder releases, more decision points, more project management, deferred QA, slower to get features out, and risk when contributors deprioritize half finished milestones. When we continuously deliver we get effortless, cheap, seamless releases so that we can focus on features.

Effortless releases is not a negotiable feature - it's required for sustainability of OS projects.

The problem with 2.0 beta was that I pulled out the beta version and then deprioritized Mockito for next x months.

Let's make a thought experiment, what happens if core contributors deprioritize OS development today. What is the desired state of Mockito project, so that such scenario has least impact on community and Mockito itself?

Desired state is: contributors can easily ship features in current Mockito (2.x)

[bric3]

And we introduce buggy release or instable API, this has already happened on mockito.

Effortless releases is not a negotiable feature - it's required for sustainability of OS projects.

Agreed, CI and CD is one of key to success for any project, from the smallest to the biggest. Nobody denies that. However each project has a constraints and different goals.

• If the project is simple a website, something's wrong, let's modify the HTML/PHP webpage and we're good
• If the project is a huge website, then maybe we'll do some QA tests before
• If the project is the software that control a car engine, then I will sure make proper tests...
• If the project is used by 10 people that follow closely the thing then hell yay it's possible to introduce incompatibilities
• If the project is used by hundred thousand, then maybe it's my job to avoid include poor release because a PR has been merged, and to polish the API.
• If project is javascript, then well, things break

Also...

Developer productivity is a nice thing, if their environment is stable and intuitive they work on their own project so that the project is successful and make money. If the environment is not stable, changing versions all the time, API change in minor, then they focus less on what matter.

Let's not forget that we do not want to make things easy if it's to make them harder of users.

The point is that we like the CI-CD that you are building, but we are not convinced by the one commit = one release pace.

The problem with 2.0 beta was that I pulled out the beta version and then deprioritized Mockito for next x months.

Version 2 delay is a problem I share as well, I was pretty sick too and as such had other priorities in life. Then as usual work kicked in as well.

Let's make a thought experiment, what happens if core contributors deprioritize OS development today. What is the desired state of Mockito project, so that such scenario has least impact on community and Mockito itself?

Right now I'm more concerned by other things, the team don't have credentials to anything related to infrastructure, so we cannot improve them.

• ❌ DNS
• ❌ bintray
• ❌ maven oss
• ❌ control over mailing-lists
• ❗ continuous delivery bot mail account (although it's possible to make another)

What happens if for some reason the team loose the only member that have those ?

Desired state is: contributors can easily ship features in current Mockito (2.x)

Yes we know that and we want it flawless as well, but we also want time to polish features.
A pull request is a nice unit of work but not enough, mockito is not a webpage project.

(EDITED : added mailing-list, and added bold)

[TimvdLippe]

My opinion is equal to the comment above of Brice.

[mockitoguy]

And we introduce buggy release or instable API, this has already happened on mockito.

The solution to this is making sure that every PR is high quality.

Yes we know that and we want it flawless as well, but we also want time to polish features.

Polish them as long as you want in a developer branch ;)

---

Thanks guys for your feedback! I'm afraid we're not making progress so let's agree that we don't agree.

1. I'm for full CD pipeline and I'm executing towards that unless new arguments are brought up to the discussion. So far, there are no new arguments.
2. I have the infra pieces on my TODO list ;)
3. We will do traditional release model for Mockito 3.0.

[TimvdLippe]

I'm for full CD pipeline and I'm executing towards that unless new arguments are brought up to the discussion. So far, there are no new arguments.

You are implying with "I'm executing" that the whole Mockito development team must act this way? I think we (the other three core developers) have brought enough new arguments to not use the full CD workflow (e.g. 1 commit is 1 minor release). I would kindly ask you to not force any ideas on us if we keep pointing out why we do not like them. Therefore I would prefer to have an agreement on our development workflow, which is essential for working together as a team.

[mockitoguy]

You are implying with "I'm executing" that the whole Mockito development team must act this way?

Of course not! If you don't want to release every push, please work on developer branch.

I will continue to improve the CD pipeline and make changes to support it unless new arguments are brought to the table.

[TimvdLippe]

That means that whenever a contributor submits a pullrequest, we need to
request him/her to point it to our development branch. This would be a huge
burden for contributors.

Op di 11 okt. 2016 om 16:25 schreef Szczepan Faber <notifications@github.com

:

You are implying with "I'm executing" that the whole Mockito development
team must act this way?

Of course not! If you don't want to release every push, please work on
developer branch.

I will continue to improve the CD pipeline and make changes to support it
unless new arguments are brought to the table.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#618 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFrDb5oZMpYcri5DazYTZ7wvommr8BFzks5qy5xygaJpZM4JytNn
.

[szpak]

While we could have slightly bigger releases. I'm not sure there's great value for developers to release Copyright headers or Improved exception message.

@bric3 Probably most of us agree on that. New features and bug fixes should be considered as a candidate for the new releases. In other cases a person making a merge should add a tag a merge message to skip the release process. If it would be too hard to follow I personally prefer the opposite way - when a person merging a PR considers it as something important to be release a "do release" tag is added to a merge message and only then a new version is released (automatically, but on demand).

[TimvdLippe]

The amount of production releases (2.X.X) currently published. E.g. we are already at 2.2.14.

[TimvdLippe]

@szpak actually the manual release was implemented with including [ci-release 2.3.0] in the commit message. This has been reverted later.

[szpak]

I thought that [ci-release 2.3.0] was only to override the next version (and by default a release with automatic increment had been performed automatically for every commit). However, I didn't analyze that code, so you may be right @TimvdLippe .

[dnozay]

I strongly dislike "one commit = one release":

• Changelogs are now meaningless, it's more of a challenge (actually: more tedious) if you need to figure out what changed between two versions.
• Making a release for doc / comment fixes is plain retarded; while users looking at the docs will appreciate, product managers and tech leads that actually track dependencies will hate you.
• There are product managers and tech leads, whose responsibilities involve actually tracking dependencies and making sure the technological debt is low; they will hate you; mockito is mostly a test-level dependency, so it may be a low-priority item on their radar. But imagine a scenario if there were a bunch of iOS developers working on a product, and they had to deal with Xcode releases for every commit.
• Changing a dependency may mean that you have to trigger many different QA cycles; not everything everywhere is automated. This is an expensive activity, and is a driver for not keeping up-to-date often.

I totally get the idea that releasing often is great to push new fixes / features out, but there is version fatigue. The opposite of this "one commit = one release" are LTS versions. People like them. Businesses like those; I know a few that will only update from LTS to LTS.

[TimvdLippe]

@dnozay Thanks a lot for voicing your opinion! I agree with you and I think this situation is actually more harmful than the benefit of continuous delivery.

Thus far we have been unable to come to an agreement or concession regarding the current practice of continuous delivery 😢 Hopefully we can come up with a solution that prevents your situation from happening.

[mockitoguy]

@dnozay, thanks for feedback! I fully share your passion and enthusiasm to release automation, while having different opinion on the way :)

What version of Mockito do you use? How often do you upgrade? How is version fatigue impacting you?

[dnozay]

Yes, I love release automation.

What version of Mockito do you use? How often do you upgrade? How is version fatigue impacting you?

I don't use Mockito directly. I got to this page based on some email thread.

But let me share some of my experience doing release management. Maybe you know about this thing called Jenkins that helps people do continuous integration (here is their Governance document). They are very successful at releasing often, while being also very good for "scaredy-cat" customers: they do provide LTS versions.

How much do I do with it? Not much, but I used to maintain a few Jenkins plugins (e.g. build-flow-plugin), and the release model follows much of the Jenkins documentation for plugins (see also the Extend Jenkins page); it uses the Maven Release plugin. We can summarize that to:

• select a bunch of changes, "Cut the changelog",
• perform all kinds of testing on it, regression, backwards-compat, integration, end-to-end, etc.
• do a mvn release:prepare release:perform -Dusername=... -Dpassword=...

How often do you upgrade?

The main factors are

• Time and money : upgrading is not free.
• Urgency : is this a security related release?
• Is this a minor version / is this backwards-compatible?
• Do I need a full qualification round, or can I save time (or $) doing smoke tests?
• Am I myself in a release cycle, doing my own testing? Is that going to invalidate prior results?

I typically will NOT upgrade unless security fixes or I intend to pull new features, or just came back from vacation and need to catch up or have extra time to spend on conflict resolution.

Upgrading costs time and money.
If you think about your dependencies, it is a graph of dependencies, not a tree. You can have conflicting packages. I mean there is a reason behind excluding transitive dependencies in gradle. Sometimes different implementations of the same classes end up on the classpath. Sometimes the order matters and you spent time down the rabbit-hole trying to figure out classpath issues.

Have you worked for a company that releases an OS? Have you seen build-to-build upgrade testing? - I have. When you make the assumption that there is some common base / thing that should work (e.g. your OS is the same), you may think testing is fine. Now imagine that you cannot trust the OS; that you need to re-install it - just because that's the thing you are testing.

Too many releases is just noise
Your consumers probably won't care if you are releasing more than once a day.
If it takes 2 days to test; they can't physically handle upgrading more than every 2 days unless they have multiple teams / staggered release cycles.

How is version fatigue impacting you?

I only have so much time I can allocate.

• Updating dependencies takes time (compile time, test time, ...) which either makes me busy, or ties resources that I can't use for other activities; e.g. I can't disconnect my laptop from the network and hop on a commuter bus.
• The problem scales non-linearly with more dependencies.
• Sometimes I have to resolve conflicts due to API changes, that's extra.

I just want to code my own stuff, not waste time on "upgrade" activities, so on a personal level, I hate it with a passion words cannot fully describe. Things are usually fine when the upgrade is smooth and silent; but that's not the common case for me.

Why are things not smooth?
That's a question that was missing. Because the bar for quality for a library/tool is low in general (I am not talking about Mockito here, so don't get mad at me).

• lack of unit tests / poor coverage,
• lack of integration tests,
• shifting responsibility of testing to the consumer,
• unstable APIs; lack of semantic versioning,
• not understanding your consumers' use cases.

Some cool things to look at:

• https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+Artifactory+Plugin+-+Release+Management
• https://wiki.jenkins-ci.org/display/JENKINS/Artifactory+Plugin
• https://wiki.jenkins-ci.org/display/JENKINS/M2+Release+Plugin
• https://wiki.jenkins-ci.org/display/JENKINS/ArtifactPromotionPlugin
• https://wiki.jenkins-ci.org/display/JENKINS/Promoted+Builds+Plugin

[mattnelson]

That's why Mockito is completely backwards compatible and rigorously follows semantic versioning. This should streamline managing Mockito dependency and if it does not, we would love hear about the use cases. Also, managing traditional releases is more work for us == less time for working on features.

Coming in a little late to this discussion as we just uplifted to Mockito 2 last week and found some quirks with the continuous aspects of this project in the first week. We have a CI job that runs every week and creates a PR to auto update all of our dependencies. When the script ran this week we got prompted to update to 2.3.7[1]. When we attempted to validate the build, it failed with a transient mockito error. We kicked off the build again and it went away. But in parallel to the build running, we attempted to look at the changes for 2.3.7 to see what changed and if we needed to make code changes, the tag/release was no where to be found. 2.3.8 was missing as well. In that time 2.3.9 and 2.3.10 were tagged, but are not available in maven central at the time of this posting.

That left us with the following options:

• Lock the dependency to our current known working version (2.2.29)
• Work back from 2.3.6 until we find a working version
• Wait for 2.3.9+ to be available on maven central and see if the build passes.

I don't want to discourage the release automation process especially if it makes it easier for maintainers and encourages contributors as they will be able to use their changes quicker.
But I agree with some of the other sentiments in this thread that less frequent and more vetted releases would be preferable.

Transient error that occurred with 2.3.7. Looks related to #638, but we are not using the internal APIs.

java.lang.NoClassDefFoundError: Could not initialize class org.mockito.internal.util.MockUtil

[1] http://search.maven.org/#artifactdetails%7Corg.mockito%7Cmockito-core%7C2.3.7%7Cjar
[2] https://github.com/mockito/mockito/tags

[TimvdLippe]

@mattnelson Sorry for the trouble that we caused you! There was a problem with releasing 2.3.7 as we are preparing the release procedure for the new mockito-android. The plugin we use to push artifacts to Bintray had a bug which was uploading corrupt jars to maven central. We identified the error quickly and retracted the corrupt versions, which was probably what you encountered too.

The release procedure should be fixed once again, but I think we can see this is a learning point that if we update the release procedure we must first test on our Bintray test project. Once we can install correctly ourselves, we push out to maven central. @szczepiq WDYT?

[mockitoguy]

@mattnelson, very sorry for problems you encountered. We had some hiccups recently as described by @TimvdLippe. Do you happen to have the stack trace of the transient issue? Can you gist it for us?

@TimvdLippe, let's chat on slack channel. Can you formulate the release model that you have in mind when you say "must first test on our Bintray test project".

@mattnelson - thanks so much for taking the time and reporting the issue back to us. This feedback helps us a lot make decisions on how to release safely.

[drewhannay]

I agree with pretty much everything @dnozay said. Upgrading Mockito is now very painful. There's no good release notes or upgrade instructions, everything is spread out across many many versions. I don't have the time or desire to look at every commit individually to find out what changed.

I understand that following semantic versioning means that we should be able to "trust" and drop in any newer compatible version. Now I may trust you all to stick to that, but my team of 50+ developers certainly does not. They want to see a list of what has changed and what benefits we get from upgrading that make it worth the risk. The current release cycle means we're much less likely to upgrade until there's a meaningful reason to do so. But with so many versions, how will I ever notice that there's a good reason to upgrade?!?

Right now my method is to follow @szczepiq on LinkedIn and Twitter to find out when there's a cool new feature and then use my own discretion to decide if it's worth spending half a day of my time to upgrade and validate a new version of Mockito. This seems....less than optimal 😉

[mockitoguy]

@drewhannay, fantastic feedback. Thank you for writing this up. You raised valid points and we will address them.

half a day of my time to upgrade and validate a new version of Mockito

Is updating minor version of Mockito half day of your work? Why?

[drewhannay]

Mainly because Android tests are slow and we have a lot of them 😉

In the best case, I end up doing at least two full test runs (~45 minutes each), plus time getting a code review and time for importing the new version of Mockito into our internal artifact server.

[mockitoguy]

I documented draft plan for new release cadence at issue #911 (interesting number, huh? coincidence? :). I would love to get feedback!

[tinkerware]

As an anecdote, I want to talk about what I had to do just today when updating dependencies on a project using both Mockito and ByteBuddy.

I had Mockito on a recent enough 2.5.4 and a separate internal library had ByteBuddy on 1.6.0. When I upgrade Mockito to its latest as of today, I end up with Mockito on 2.7.5. ByteBuddy dependency from Mockito is at 1.6.5, however. Additionally, ByteBuddy 1.6.5 breaks public API with 1.6.0 by renaming at least one interface and one method (lest you think this is unusual, ByteBuddy has always broken public API in its patch versions; arguably, that's the root cause). Now suddenly, I have two compile-incompatible versions of the same library when I upgrade from Mockito 2.5.4 to 2.7.5.

I'm not bringing this particular example up to assign blame (I love ByteBuddy to pieces, even with its painful unpromise of public API compatibility; I have accepted that it's a high-maintenance library to keep up with) or to suggest changes to Mockito (though, in retrospect, Mockito can shade ByteBuddy as a private dependency to avoid passing down the pain of keeping up with it). I only wanted to point out that maintaining dependencies can be work, even hard work. Predictable, compatible releases are valuable because they make this work easier.

[mockitoguy]

@tinkerware, this is extremely useful feedback. Thank you very much for reporting. I will start an internal discussion about the idea of shading the bytebuddy jar.

In the early days of Mockito, we used to ship with direct dependency on cglib. However, it was causing similar problems to our users and we started shading cglib. Bytebuddy is getting popular (rightfully so!) and will be used more more often. I want to make sure that mockito is always very convenient to use, and avoids potential incompatibilities that cause users to troubleshoot classpaths / dependency graphs.

[raphw]

@tinkerware Out of curiosity. What API of Byte Buddy do you rely on that broke since then? Typically, Byte Buddy changes its public API because it has to what happens with features that were recently added and had to prove their use in practice. This is a bit of a catch 22 where I consider anything new to be incubating without naming a maturity date or version as I simply do not know.

If there is any particuar API that broke for you, I would be curious to know!

[tinkerware]

Out of curiosity. What API of Byte Buddy do you rely on that broke since then?

I had code that gathered annotations from multiple "levels" (e.g. declaring type, enclosing types, package) so I used AnnotatedCodeElement as the common type across functions that traversed declarations. That got renamed to AnnotationSource. Another one is the isGenericDeclaration method, renamed to isGenerified; that was part of a matcher predicate (to be clear, I like the new names better).

Typically, Byte Buddy changes its public API because it has to what happens with features that were recently added and had to prove their use in practice. This is a bit of a catch 22 where I consider anything new to be incubating without naming a maturity date or version as I simply do not know.

I genuinely sympathize with this, and I think shoehorning multiple API's, each in a different point in their lifecycle, into one artifact version number is eventually self-defeating. The late Pieter Hintjens had a good handle on defining this problem and one possible way forward, written up in End of Software Versions.

In general, I find the current state of tooling to manage API lifecycle miserable, but there is some low-hanging fruit. I have two suggestions: (1) You can mark new API that has to prove itself with a draft/beta API annotation, and (2) deprecate all API before removal. Deprecated API does not have to hang on for ages; I'm okay if beta API disappears in a minor version after a short period of deprecation. Only stable API would be relevant for semantic versioning.

I don't want to hijack this thread with non-Mockito discussions, so feel free to point me to another place to continue if you think it's worth it.

Thanks for ByteBuddy and Mockito!

[raphw]

Thanks for your feedback. True, I had to change this with respect to making type annotation resolution lazy which is was not before. Its rather the exception than the rule but in this case, it was a customer-driven innovation which is partly a requirement for keeping the library alive. I agree that I should have not done this in a minor update but the incompatibility was difficult to avoid.

[TimvdLippe]

Would like to give everyone a heads up who is subscribed to this issue: @szczepiq did some great work to implement the new workflow. You can read a detailed description in the (WIP) wiki at https://github.com/mockito/mockito/wiki/Continuous-Delivery-Details#cd

We are planning to ship this new implementation soon, which means you will receive fewer Mockito updates. If you want to use the latest version all of the time, follow the instructions in the wiki to get listed for all updates 😄

[TimvdLippe]

Over the past half year, we have been using Shipkit to enable our continuous delivery story. We published our rationale in https://github.com/mockito/mockito/wiki/Continuous-Delivery-Details and described how we tackled this issue. All versions are still automatically published to Bintray to enable users to opt-in frequent updates. Other users can rely on the Maven Central release, which is published roughly every month (this is not strict, but based on numbers on https://mvnrepository.com/artifact/org.mockito/mockito-core).

All in all, we have not received any more feedback regarding too frequent published versions. We are also happy that we can still receive quick feedback and bisect based on Bintray binaries to find out which version (and thus PR) introduced the regression.

I would like to thank everyone that participated in this discussion and worked with us to tackle this issue. If you have any more specific feedback regarding our release mechanism, please file your feedback at https://github.com/mockito/shipkit