Mockito continuous delivery model #618

Open
szczepiq opened this Issue Sep 1, 2016 · 80 comments

Projects

None yet
@szczepiq
Member
szczepiq commented Sep 1, 2016 edited

This issue discusses:

  • Mockito continuous delivery model (e.g. publishing every PR: wiki link)
  • How to prevent prolonged Beta phases as it happened for v2
@TimvdLippe
Contributor

Strict set of issues we want to solve and stick to that. Once they are resolved, publish the version and define a new strict set.

We should not add issues to the current milestone, unless they are absolutely n1 priority if it is a huge bug. But this probably rarely happens.

E.g. for 2.2.0 we are already set, I think this list is long enough: https://github.com/mockito/mockito/milestone/3. This milestone is very clear in its goal: improve the ArgumentCaptor API. That we have a release focused on an issue. For major releases, we can of course adopt more than 1 focus. That's my 2 cents.

@szczepiq
Member
szczepiq commented Sep 2, 2016

Good feedback. I'm not worried about v2 releases. Since they are compatible, we can simply iterate minor or patch version and release without any beta versions.

However, 3.0.0-Beta.x worries me. What can we do to avoid prolonged v3 beta phase.

@bric3
Contributor
bric3 commented Sep 2, 2016 edited

I think the same would apply e.g. define a set of goals for 3.0. Currently 3.0 scope is a bit fuzzy, we know that we want Java 8, maybe JUnit 5.

Also note that the prolonged 2.0 phase was also due to our life. You had to move to LinkedIn, I was overwhelmed by own professional agenda. That is somehow outside our control. I don't know about your plans about mockito in the future but on my side I have more spare time recently, but that may change.

@szpak
Member
szpak commented Sep 3, 2016

I think the same would apply e.g. define a set of goals for 3.0.

Having set a minimal scope with should be done before 3.0 is a good idea. However, I would not limit accepted changes to just those things. Any sensible PR should be merged (as 2.x will be in the maintenance mode). In general, a lot of versions (here beta) is a side effect of continuous delivery. On the other hand I prefer it over working with snapshots/jitpack just to have given fresh feature available in my tests.

@TimvdLippe
Contributor

If there is a PR which can be merged as-is, yes of course we should merge it. However, I think the current open PRs regarding ArgumentCaptor is what I meant with the strict set. Since this class was not included in 2.0.0, we decided to move all these PRs to 2.2.0. If there is a lot of work tight to a certain feature (or set of features), I do not think we should blatantly accept it for 3.0.0. That way we can deliver 3.0.0 on time, while also have a solid plan for later releases.

Of course we have to keep in mind the semver scheme. If we know there are breaking changes required, we might have to do those together in 3.0.0 because we can't introduce them in 3.1.0.

@szczepiq
Member
szczepiq commented Sep 9, 2016

Feedback from users: #595 (comment)

@szczepiq
Member
szczepiq commented Sep 9, 2016

Perhaps we should avoid pushing beta versions to central / jcenter and only push them to internal bintray repo.

@bric3
Contributor
bric3 commented Sep 9, 2016 edited

Bintray can serve as a maven repo I believe, https://bintray.com/docs/usermanual/formats/formats_mavenrepositories.html#_working_with_maven

So that may be a solution.

@szczepiq
Member

Options to consider:

  1. don't publish Betas at all
  2. don't publish to central, publish to some less prominent public repo
  3. publish some betas to central, manually chosen
  4. publish some betas to central, automatically at certain cadence / when specific release milestones are completed

I hope 1 year from now we won't be publishing 2.1.0-RC.123 :)

@TimvdLippe
Contributor

On master I bumped the version properties to 3.0.0-beta.1 and disabled the release procedure for now until we have developed a stable 3.X beta and resolved this issue.

@bric3
Contributor
bric3 commented Sep 10, 2016

I think we should address this issue before any further development.

@TimvdLippe
Contributor

I think the cleanest solution to have a separate bintray project mockito-beta which has all beta releases. This should be fairly easy to build in our release process, is minimally impactful for our beta testers (it's just 1 word change) and does not clutter the official releases.

@ChristianSchwarz
Contributor
ChristianSchwarz commented Sep 15, 2016 edited

Options to consider:

  1. don't publish Betas at all
  2. don't publish to central, publish to some less prominent public repo
  3. publish some betas to central, manually chosen
  4. publish some betas to central, automatically at certain cadence / when specific release milestones are completed

5 . release beta's as different artifact e.g. mockito-core-beta to central?

@TimvdLippe
Contributor

@ChristianSchwarz Yes that was the option I described in #618 (comment)

@bric3
Contributor
bric3 commented Sep 16, 2016 edited

I like the idea of beta specific maven coordinate. However not sure we need another bintray account. Though it's not an issue if we have one. Anyway we will have to create one mockito account anyway see #631

@raphw
Member
raphw commented Sep 16, 2016

Or we do it the official Maven way and publish SNAPSHOT versions to a snapshot repository.

@szpak
Member
szpak commented Sep 16, 2016 edited

SNAPSHOTs were published in the past. I personally don't like and don't trust them. With beta releases it is the package that will not change and will be available "forever". Snapshots (even those with timestamp) can are cleaned up after some time.

A separate repo could be a solution if we really plan to have a plethora of beta versions again :).

@szczepiq
Member

Hearty -1 to different artifact name. It's very inconventional and confusing. It could lead problems with conflict resolution (both beta and normal artifact on the same classpath).

Separate bintray repo is the way to go.

However, it feels that separate bintray repo is not enough. We also need to plan releases better and drive them to resolution faster than in 1.5 years :)

@TimvdLippe
Contributor

I will set up a Bintray account this weekend for #631 and then we can configure it ๐Ÿ˜„

@TimvdLippe
Contributor

Oh forgot to add regarding the number of releases: The issue for Mockito 2 was that no strict feature set was defined. If we define a strict set and work towards finishing it, it should be possible to publish releases every ~2 months. This is a lot better than the 1.5 years :P. I think we already defined what Mockito 2.2.0 must incorporate, so we can start working on that in the release/2.x branch.

@szczepiq
Member

Mockito 2 continously publishes. Hence, there is no risk of prolonged beta :)

@bric3 bric3 added a commit that referenced this issue Oct 3, 2016
@bric3 bric3 Didable release again and update version to 2.2.0-beta.2 even if won'โ€ฆ
โ€ฆt distribute other beta until #618
0a6ae5d
@szczepiq
Member
szczepiq commented Oct 9, 2016

Hiding betas from default repositories and having strict feature set for milestones is not good, nor sufficient solution. We no longer continuously deliver which means harder releases, more decision points, more project management, deferred QA, slower to get features out, and risk when contributors deprioritize half finished milestones. When we continuously deliver we get effortless, cheap, seamless releases so that we can focus on features.

Effortless releases is not a negotiable feature - it's required for sustainability of OS projects.

The problem with 2.0 beta was that I pulled out the beta version and then deprioritized Mockito for next x months.

Let's make a thought experiment, what happens if core contributors deprioritize OS development today. What is the desired state of Mockito project, so that such scenario has least impact on community and Mockito itself?

Desired state is: contributors can easily ship features in current Mockito (2.x)

@bric3
Contributor
bric3 commented Oct 10, 2016 edited

And we introduce buggy release or instable API, this has already happened on mockito.

Effortless releases is not a negotiable feature - it's required for sustainability of OS projects.

Agreed, CI and CD is one of key to success for any project, from the smallest to the biggest. Nobody denies that. However each project has a constraints and different goals.

  • If the project is simple a website, something's wrong, let's modify the HTML/PHP webpage and we're good
  • If the project is a huge website, then maybe we'll do some QA tests before
  • If the project is the software that control a car engine, then I will sure make proper tests...
  • If the project is used by 10 people that follow closely the thing then hell yay it's possible to introduce incompatibilities
  • If the project is used by hundred thousand, then maybe it's my job to avoid include poor release because a PR has been merged, and to polish the API.
  • If project is javascript, then well, things break

Also...

Developer productivity is a nice thing, if their environment is stable and intuitive they work on their own project so that the project is successful and make money. If the environment is not stable, changing versions all the time, API change in minor, then they focus less on what matter.

Let's not forget that we do not want to make things easy if it's to make them harder of users.

The point is that we like the CI-CD that you are building, but we are not convinced by the one commit = one release pace.

The problem with 2.0 beta was that I pulled out the beta version and then deprioritized Mockito for next x months.

Version 2 delay is a problem I share as well, I was pretty sick too and as such had other priorities in life. Then as usual work kicked in as well.

Let's make a thought experiment, what happens if core contributors deprioritize OS development today. What is the desired state of Mockito project, so that such scenario has least impact on community and Mockito itself?

Right now I'm more concerned by other things, the team don't have credentials to anything related to infrastructure, so we cannot improve them.

  • โŒ DNS
  • โŒ bintray
  • โŒ maven oss
  • โŒ control over mailing-lists
  • โ—๏ธ continuous delivery bot mail account (although it's possible to make another)

What happens if for some reason the team loose the only member that have those ?

Desired state is: contributors can easily ship features in current Mockito (2.x)

Yes we know that and we want it flawless as well, but we also want time to polish features.
A pull request is a nice unit of work but not enough, mockito is not a webpage project.

(EDITED : added mailing-list, and added bold)

@TimvdLippe
Contributor

My opinion is equal to the comment above of Brice.

@szczepiq
Member

And we introduce buggy release or instable API, this has already happened on mockito.

The solution to this is making sure that every PR is high quality.

Yes we know that and we want it flawless as well, but we also want time to polish features.

Polish them as long as you want in a developer branch ;)


Thanks guys for your feedback! I'm afraid we're not making progress so let's agree that we don't agree.

  1. I'm for full CD pipeline and I'm executing towards that unless new arguments are brought up to the discussion. So far, there are no new arguments.
  2. I have the infra pieces on my TODO list ;)
  3. We will do traditional release model for Mockito 3.0.
@TimvdLippe
Contributor

I'm for full CD pipeline and I'm executing towards that unless new arguments are brought up to the discussion. So far, there are no new arguments.

You are implying with "I'm executing" that the whole Mockito development team must act this way? I think we (the other three core developers) have brought enough new arguments to not use the full CD workflow (e.g. 1 commit is 1 minor release). I would kindly ask you to not force any ideas on us if we keep pointing out why we do not like them. Therefore I would prefer to have an agreement on our development workflow, which is essential for working together as a team.

@szczepiq
Member

You are implying with "I'm executing" that the whole Mockito development team must act this way?

Of course not! If you don't want to release every push, please work on developer branch.

I will continue to improve the CD pipeline and make changes to support it unless new arguments are brought to the table.

@TimvdLippe
Contributor

That means that whenever a contributor submits a pullrequest, we need to
request him/her to point it to our development branch. This would be a huge
burden for contributors.

Op di 11 okt. 2016 om 16:25 schreef Szczepan Faber <notifications@github.com

:

You are implying with "I'm executing" that the whole Mockito development
team must act this way?

Of course not! If you don't want to release every push, please work on
developer branch.

I will continue to improve the CD pipeline and make changes to support it
unless new arguments are brought to the table.

โ€”
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#618 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFrDb5oZMpYcri5DazYTZ7wvommr8BFzks5qy5xygaJpZM4JytNn
.

@TimvdLippe
Contributor

Okay since the previous arguments did not convince you yet, I would like to
sketch the scenario which depicts why we are uncomfortable with the
workflow.

First of all there is probably some misunderstanding regarding the naming.
We are all for continuous delivery, we are against continuous deployment.
With delivery we have always a working version available as snapshots (or
beta whatever you want to call it). With deployment you immediately publish
it as an official version.

The scenario which has very undesirable results is when multiple PRs are
merged in a short period of time; lets say a couple of hours. With
continuous delivery, the snapshots are generated and available. However
with continuous deployment they are ALL published to production. This means
that in a matter of a couple of hours, we generated a number of minor
versions. This is confusing for end users, a huge burden on updating the
Mockito dependency and as maintainer annoying to manage.

For these reasons, we dislike continuous deployment. We are all for
delivery. I hope this makes clear the standpoint.

Denoting that we can use separate workflows at the same time will not work.
Thats taking the bad sides of both solutions together.

On Tue, 11 Oct 2016, 16:39 Tim van der Lippe, tvanderlippe@gmail.com
wrote:

That means that whenever a contributor submits a pullrequest, we need to
request him/her to point it to our development branch. This would be a huge
burden for contributors.

Op di 11 okt. 2016 om 16:25 schreef Szczepan Faber <
notifications@github.com>:

You are implying with "I'm executing" that the whole Mockito development
team must act this way?

Of course not! If you don't want to release every push, please work on
developer branch.

I will continue to improve the CD pipeline and make changes to support it
unless new arguments are brought to the table.

โ€”
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#618 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFrDb5oZMpYcri5DazYTZ7wvommr8BFzks5qy5xygaJpZM4JytNn
.

@szczepiq
Member

That means that whenever a contributor submits a pullrequest, we need to
request him/her to point it to our development branch. This would be a huge
burden for contributors.

On the contrary, it's much easier for contributors of high quality PRs to get their changes published quickly.

Okay since the previous arguments did not convince you yet

This argument has been brought up before. I don't discount it I have a different point of view. It's mockito mission to pioneer continuous delivery for Java libraries. Consumers don't have to update to the latest version every day. It's up to them.

You brought up the point of decision making in Mockito team. There is no formal process, we make decisions organically. There is no democracy, there is no despotism, either. The more contributions one has over all-time (and how engaged he is currently), the stronger his opinion matters. There is no mathematical formula, I can give an example: If I deprioritize mockito and you don't hear from me for 2 months, @bric3 becomes most senior engineer (level of contributions over time and current engagement) and his opinion weighs most. @TimvdLippe, you're next :)

@jeantil
jeantil commented Oct 12, 2016

To sum up the disagreement from an external point of view :
why should the frequency of releases be linked to commit frequency ?

continuous != on every commit ( or at least not necessarily )
continuous == regularly unless there is nothing to deliver

Fixed timings could work too :

  • daily for release for beta/snapshot whatever
  • bi-monthly for releases to official repo

Whatever timing you want to use, you can go for a mixed strategy:
tag a release after 10 commits or 2 weeks (this way even if contributor activity is low, a user never has to wait more than 2 weeks to get his merged feature in the central repo)

These strategies can still be considered as continuous since they don't depend on a manual action. Yet

  • they allow for an opportunity window to fix unavoidable human mistake,
  • they allow for predictable release cycles for the users (includes reduced confusion for non hardcore users)
  • they don't preclude polishing in developer branch

2cents, have fun.

@TimvdLippe
Contributor

@jeantil Thanks a lot for joining the discussion and expression your opinion. It is greatly appreciated!

@tinkerware

This is an interesting discussion; thanks for having it in the open.

I've been using the 2.x versions of Mockito for the last six months. I will admit to being mildly annoyed at the pace new beta versions have appeared, and I was cross at VersionEye for incorrectly marking Mockito as out-of-date after I picked the latest stable release.

On the other hand, I am a lot more upset with a library if it does not make stable releases every few weeks (though this depends on pace of development) so that recent fixes and PR's make their way as soon as they can into the hands of users. For a library with the development pace of Mockito, I would want to see a new stable release at least once every few weeks. And further, if I'm contributing to the library or waiting on a fix, I appreciate greatly being able to access a release immediately so that I don't have to go to the trouble of publishing to an internal repo or vendoring the library.

Looking at the space for browsers, they seem to have come up with a workable model to balance these two needs with release channels. Users who value stable versions that do not move every day can use the stable channel to get access to reasonably often updates with fixes and features, and others can subscribe to the beta channel to get the full artifact-per-commit stream.

Thanks for the great work!

@TimvdLippe
Contributor

@tinkerware I worked out a release model where I got my inspiration from the browser release process, you can read it at #681 (comment). Was this a model you had in mind?

@bric3
Contributor
bric3 commented Oct 13, 2016

On the contrary, it's much easier for contributors of high quality PRs to get their changes published quickly.

That's the thing, while I'm definitely pushing for high-quality PR, all are not hight quality for various reasons :

  • user don't know git well e.g. contributor do X non meaningful commits, history is unreadable and reason of change is obscure or split across various commits
  • Test coverage
  • API design is wrong or in-progress
  • Contributor don't reply

Or PR represents internal reworks without value, to prepare something bigger. Sometime we may find flaws in previous PR. Once an API goes in the wild it's hard to go back.

There is no mathematical formula, I can give an example: If I deprioritize mockito and you don't hear from me for 2 months, @bric3 becomes most senior engineer (level of contributions over time and current engagement) and his opinion weighs most. @TimvdLippe, you're next :)

While I appreciate that my opinions may matter because I've contributing, I think that regardless of level of contributions, any opinion can be just as good as anyone's.

It's mockito mission to pioneer continuous delivery for Java libraries.

No it is not! The mockito's mission is to make a quality library that allows to write good tests with mocks. Pioneering CI/CD allows to accomplish that main mission with ease, but it is not the main goal. Having the delivery upon each commit (or not) isn't pioneering or even innovation, it is a choice that is decided to match how a projects model its release pace.

@jeantil @tinkerware Interesting point of view, browser release model is indeed interesting.

@szczepiq
Member

Thanks guys for feedback!

That's the thing, while I'm definitely pushing for high-quality PR, all are not hight quality for various reasons

@bric3, regardless of the release model, we never want to merge low quality PRs to main dev branch

Mockito mission is to excel in quality of all: impact on quality of tests and code, feature set, extensibility, release model. Continuous delivery model allows most sustainable, effortless release capability which is part of software quality.

Guys, I'm curious about 1) what is your exposure to continuous delivery with production systems / libraries? 2) Did you read continuous delivery book?

To give you my background: every change publishes new version of libraries at LinkedIn since 2011. In 2015 at LinkedIn we built 3x3 model (3 hours max time commit-to-production, 3 releases per day), my friend wrote an article about 3x3 for mobile.

@jeantil
jeantil commented Oct 14, 2016

Fascinating read, but I feel like the model for a library is closer to that of a mobile app than that of a web app

For true continuous integration, we ship every commit directly to production โ€“ but this is clearly not realistic in the world of native apps, where binaries must be published in the App Store (with Appleโ€™s week-long review process) and Google Play, then downloaded by members.

It's the same as browsers or IDEs or anything which requires an action on the user part to install.They have an EAP/beta channel where they release up to multiple times a day if they want, and a normal release channel where identified stable builds are promoted in a timely manner.

Continuous delivery works well when you can be sure that if something broken reaches a user, you can push a fix to them quickly. It is an excellent model for : websites where a refresh fixes the app, and more generally self updating software where update doesn't require a restart or where restart time is very slow.

For a java library, as far as I know, the version update process tends to be a manual edit in a file (pom, gradle, sbt,etc) by a developper. That's even less user friendly than an app store update push notification.
Why would you want to make your users do that more often ?

And for the few advanced users using version ranges they would risk random build failures which are always such a pain to investigate...

@szczepiq
Member

For a java library, as far as I know, the version update process tends to be a manual edit in a file (pom, gradle, sbt,etc) by a developper. That's even less user friendly than an app store update push notification.
Why would you want to make your users do that more often?

It's entirely up to the consumers how often they want to update. I give them the option to update often, they don't have that option in traditional model.

And for the few advanced users using version ranges they would risk random build failures which are always such a pain to investigate...

Automated ways of updating dependencies is exactly the way forward. For this model to succeed, semantic versioning needs to be adopted, high quality changes, comprehensive test suite that never breaks the customers. This is entirely different discussion - it's about consumers not producers. In traditional release model, people can still use version ranges and get exposed to problems if the producers are not high quality.

Thanks for joining the conversation!

@TimvdLippe
Contributor

Automated ways of updating dependencies is exactly the way forward.

As a user of dependencies, I disagree. Recently I have resorted to fixed version dependencies instead of ranges to have control. Mostly to resolve conflicts at PR time instead of having bugs creep into the system without me knowing. Therefore this statement is invalid in my eyes.

I give them the option to update often, they don't have that option in traditional model.

Please be reminded that you are not the only Mockito developer and currently the only developer advocating for a new model.

@szczepiq
Member

As a user of dependencies, I disagree. Recently I have resorted to fixed version dependencies instead of ranges to have control. Mostly to resolve conflicts at PR time instead of having bugs creep into the system without me knowing. Therefore this statement is invalid in my eyes.

I respect your POV.

Please be reminded that you are not the only Mockito developer and currently the only developer advocating for a new model.

I'm the creator of Mockito, it's the best mocking framework for Java and currently it has the best release model of all OS libraries of similar scale. It's not bragging - it's a fact.

It's not a new model, it's the model we had since 2014.

@szczepiq
Member

I wrote an article this weekend on continuous delivery VS traditional release model.

Join the continuous delivery movement!!! :)

@szpak
Member
szpak commented Oct 17, 2016

Automated ways of updating dependencies is exactly the way forward.

As a user of dependencies, I disagree. Recently I have resorted to fixed version dependencies instead of ranges to have control. Mostly to resolve conflicts at PR time instead of having bugs creep into the system without me knowing. Therefore this statement is invalid in my eyes.

@TimvdLippe Using ranges (which is not deterministic over time) is not the only possible option here. For me, updating dependencies in most cases is just a boring duty that you need to perform occasionally to not leave with ancient versions as your dependencies. Some time ago I was playing with a "dirty hack" based on uptodate-gradle-plugin to automatically update dependencies in my project if new minor/patch version is available. If all tests pass the code the new fixed version(s) could be committed to my repo (and in case of CD the new version could be released). As a person who don't like to do boring repeatable things I like that approach very much!

When dependencies are bumped manually there is usually nothing more that you do in addition before a commit/release. Unfortunately the process/tool has to be more sophisticated (e.g. different rules for different tools, blacklists, narrowing a reason of failure if more than one library has been updated and tests failed, ...), but there is still a change that one day I will so much spare time to implemented it as reusable component which could be reused across different projects :).

@tinkerware
tinkerware commented Oct 17, 2016 edited

If you want an additional data point or two as to how Mockito's release model works out for others, I have found it to be one of the worst offenders in terms of outdated dependencies from an open-source project. Maybe this is due to the two-year old beta label on the 2.0 release, or maybe it is "upgrade fatigue" from compulsively bumping to the newest release; in any case, it is a real problem that needs addressing in my opinion.

I'm not against a continuous release model, which I've used for internal code, but I think that it works much better when you are in the position to fix your clients in case of breakage or when you are behind a service boundary so your clients always interact with your latest deployed version. High quality PR's and automated testing are mandatory, but the amount of problems they can make visible do not compare to the diversity of usage that a library with Mockito's reach enables; your users will always find more interesting failure cases in your releases.

The problem here is that with a diverse set of users you start to have groups with different needs. Following the release stream is work for your users, and having Mockito show up as pretty much always "outdated" quickly teaches them that it's pointless to keep bumping versions; most projects I've seen settle into one version that works for them and ignore updates after that. Only a minority will follow up on the release stream more or less in a timely manner; some teams are just more pro-active in managing their dependencies.

Grouping releases by (stable, dev) channels would explicitly recognize that these two sets of users have differing needs when it comes to managing dependencies. The reality is that public releases just have a longer shelf-life for widely-used libraries, and disregarding that fact actively discourages slow-adopters from using new releases.

A few suggestions on mechanics:

  • The "beta" signifier is, I think, unneeded and has a negative effect on upgrades. If Mockito is continuously delivering, it does not need to make a beta/release distinction.
  • dev channel releases should be a superset of the stable channel, with stable releases appearing in both channels.
  • To ease the semver versioning in the build process, I know of siom79/japicmp project to compare two binary versions and figure out the version bump that's needed based on the resulting diff; that may be useful.
@bric3
Contributor
bric3 commented Oct 18, 2016 edited

Difficult to argue over blogs posts, but drawing bicycle and rocket are not arguments, also traditional word is kinda vague. The board in this blog post may apply to some specific context in a specific project. But it doesn't on mockito e.g. the red coloured column don't even apply to mockito.

continuous-vs traditional

In the CONTEXT of mockito, i.e. not a web application

Users and other contributors are not arguing about keeping traditional model, they just don't like this, I don't :

one commit = one release

However I believe that everyone accepts that

one commit can be released any time

From this we can build a release model where we can resolve issues related to OSS, quality, release, continuous delivery. Propositions by @jeantil or @tinkerware are great in that regard.

While the refused new arguments close these opportunities.

@szczepiq szczepiq changed the title from Prevent Beta version overload in the future to Mockito continuous delivery model Oct 19, 2016
@szczepiq
Member

@tinkerware, @bric3, Thanks for feedback!

I have found it to be one of the worst offenders in terms of outdated dependencies from an open-source project. Maybe this is due to the two-year old beta label on the 2.0 release, or maybe it is "upgrade fatigue" from compulsively bumping to the newest release; in any case, it is a real problem that needs addressing in my opinion.

Let's ignore the beta version for now (we all agree that it was problematic). @tinkerware, can you elaborate on the "real problem" part? Is it because Mockito library constantly shows up as "outdated".

Following the release stream is work for your users

Why is it work? Upgrading is not mandatory and it is compatible should someone decide to upgrade.

and having Mockito show up as pretty much always "outdated"

Where does it show up as "outdated"?

quickly teaches them that it's pointless to keep bumping versions; most projects I've seen settle into one version that works for them and ignore updates after that.

This is true regardless of the release model, e.g. with daily/monthly/yearly releases there will always be consumers that are happy with some version and will not upgrade.

@tinkerware

Following the release stream is work for your users

Why is it work? Upgrading is not mandatory and it is compatible should someone decide to upgrade.

Managing dependencies has always been work; I did not think this would be a controversial statement. If you do not have or are not informed about tool support such as VersionEye or version:display-dependency-updates, it is usually tedious, busy work. It's not always optional either; transitive dependencies mean that you need to stay on top of your whole dependency tree in order to avoid time-consuming build (or, worse, runtime, depending on how repeatable your builds are) problems. Depending on how big the bump is, you may have to set up exclusions for transitive dependencies so your build succeeds, and you may have to rework code to avoid deprecated API's.

and having Mockito show up as pretty much always "outdated"

Where does it show up as "outdated"?

If you have any tooling that checks your dependencies against latest releases, you can see it as outdated. Another way is to look at duplicate artifacts coming through transitive dependencies.

quickly teaches them that it's pointless to keep bumping versions; most projects I've seen settle into one version that works for them and ignore updates after that.

This is true regardless of the release model, e.g. with daily/monthly/yearly releases there will always be consumers that are happy with some version and will not upgrade.

I agree that there will always be consumers that do not upgrade. My point was that there's a psychological component to their decision. Ask yourself this: does having a single artifact-per-commit release channel act as an incentive or deterrent for the slow-adopters? I'm arguing for the latter when I said "quickly teaches them that it's pointless to keep bumping versions"; the volume of the artifact-per-commit releases is more like a fire hose so most consumers tune it out. There's only so much attention to go around when tending to the garden, so to speak, and when you publish two releases within an hour on your only release channel (see 2.2.4 and 2.2.5), you are effectively asking for apathy.

If you want to incentivize timely adoption, I'm suggesting that it would help to provide a release channel that's more predictable and has lower pace. When slow adopters are finding it difficult to stay on a regular update schedule, giving them a gentler moving walkway instead of expecting them to jump on the express-speed one has a better chance of keeping your downstream consumers up-to-date.

@szczepiq
Member

Really good arguments. Thank you for taking the time and writing this up!!!

I'll get back :)

@TimvdLippe
Contributor

Keep in mind that if a critical bug fix has been merged, a manual release by a developer is still possible.
Other than that, all users I talked to and myself have the same viewpoint as described by @tinkerware

@philwebb

Just to add another data-point to this discussion, it's worth thinking about how libraries that extended Mockito would deal with a continuous delivery style approach. I work on Spring Boot, and we have deep Mockito/Spring integration that we offer for our users. We also provide a managed dependency so that our users can consume a "blessed" Mockito version.

I'd have to agree with @tinkerware's comments above. I feel continually pushing releases to Maven central is likely to cause version fatigue and actually have the opposite effect than the one your trying to achieve. It's also going to make it quite hard for us to know what is actually the best Mockito release to use.

I'm a big fan of continuous delivery and continuous deployment, but I think continuously deploying every commit to a new Maven Central release would be a mistake.

Here's how Spring Boot does things (in case you're interested).

  • We maintain branches for the upcoming release and the last major release. Bug fixes go on the maintenance branch and new features go on master. Bug fixes are also forward merged to master.
  • Each commit to the repo results in a CI build and binaries being published to repo.spring.io/snapshot.
  • We use the suffix -BUILD-SNAPSHOT to indicate that we're working towards a specific version.
  • For major releases we'll periodically release milestones to repo.spring.io/milestone with the suffix M1, M2 etc. We'll also put "release candidates" in the repo.
  • For an actual real release, we publish to repo.spring.io/release, Bintray and Maven Central. We then bump the version number to the next -SNAPSHOT and the cycle continues.

For a number of reasons, I quite like this process. It generally seems to work quite well for our users. People don't get overwhelmed with Maven Central releases, but they can still try the bleeding edge easily if they want to.

Here's a few typical scenarios and how we deal with them:

  1. A user reports a bug on the stable branch.
  • We commit a fix to maintenance branch in git
  • CI builds and publishes it to repo.spring.io/snapshot
  • We ask the user to test the fix. To do this they add repo.spring.io/snapshot to their pom.xml or build.gradle and update the version that they use
  • Sometimes we mess up, and need to try to fix things again and this loop continues

Mostly users are then happy enough to wait until the next blessed release. If they're really in a hole they may decided to use the SNAPSHOT in production. Since SNAPSHOTS for maintenance branches are fairly stable this usually isn't a problem.

  1. A user wants to contribute or try a new feature
  • We commit the new feature to master
  • CI builds and publishes the SNAPSHOT
  • The user builds against the SNAPSHOT and plays with the feature
  • We iterate on things, often changing the design. People are aware that by using the bleeding edge SNAPSHOT nothing is set in stone.
  1. A user is interested in ensure that they can upgrade smoothly
  • We release a milestone (or releases candidate) and announce it
  • Users add repo.spring.io/milestone to their pom.xml or build.gradle
  • They try upgrading their project and report issues
  1. A user is just on the happy path
  • They use a tool like version eye or the maven versions plugin or just watch announcements
  • They upgrade the version in their POM and get a bunch of fixes

Hope that's useful information and good luck with whatever approach you finally take!

@TimvdLippe
Contributor

@philwebb Thanks a lot for sharing your release model with us! I think it would fit Mockito too to adopt such a model. It seems that a lot of developers have also upvoted the comment of @tinkerware

@tinkerware

@philwebb Thanks for the details of your release model. I like how you gave thought to common scenarios and how the model helps each one out.

I think Mockito could do with a simpler model. In particular, I think long-lived maintenance branches are more useful if you have a long shelf-life "platform" release that others can target when building tools and middleware, which makes a lot of sense for Spring Boot. Keeping all development on master including fixes and feature work is simpler; and even if you had to do it otherwise, I think it's safer to keep fixes flowing one-way from master to branches by cherry-picking.

On the other hand, I see that there's a release/2.x branch in addition to master (for Mockito 3?) on this project, and no clear way to choose between the two branches when looking at the active PR's, so I'm not sure about the actual/intended development model; the contributor guide was not much help. Is there some CI magic that verifies and merges approved PR's into both branches at the same time? Both branches seem very active, which is interesting. Maybe it makes more sense to say that 2.x and master branches have their own independent release models and leave it at that.

@TimvdLippe
Contributor

@tinkerware master is for 3 and the release branch for 2. We periodically merge release into master but not vice versa

@bric3
Contributor
bric3 commented Oct 21, 2016 edited

And we introduce buggy release or instable API, this has already happened on mockito.

The solution to this is making sure that every PR is high quality.

A goal but in practice there still things that slips through, just like yesterday.


@philwebb @tinkerware @jeantil

About the release model we elaborated a few months ago, before the one commit=one release model came back to wipe out these plans is described in #483 (comment)

My thinking is :

  1. Release 2.0.0
    1. Branch out to release/2.x
    2. Forget about 2.1.x
    3. Adapt release script to perform and deploy release on this branch
    4. Publish critical fixes only with versions 2.0.x
  2. Upgrade master to 3.0.0-beta-<build number>
    1. Start mockito 3.0.0 progress there (like java 8 only)

For historical purpose : https://groups.google.com/d/msg/mockito-dev/dG2KltOz-gQ/CiNAZkNBcLoJ

EDIT : See #594

So at that point the idea was that we will do only the minimal stuff in mockito 2, only add the remaining items we couldn't include in 2.1.0. So to clarify a bit more with a graph that would have been something like. (tags and numbers are for example purpose)

                o--o--o      <- PR #73 targets master
               /               New awesome API, maybe with hidden bugs
--o-- -o-o-  --o--o--o--o-   <- master (next version eg v3) -> release/deploy 3.0.0-build
  \                    /
   o-- - -o- - --o-- -o-     <- release/2.x ()
    \     ^      ^
     \    |       fixes something on 2.1.0 -> release/deploy 2.1.x
      \    release of 2.1.0
       o--o--o--o            <- PR #132 targets release/2.x
                                New small feature for 2.2 (because unsure for 2.1)

At some point or regularly there would be a merge from release/2.x to master. Current workflow is manual though.
Of course release/2x should not stay active for a long time, so PRs like 132 would be small enough and not many.


I'm pretty sure the model described above and the continuous delivery can fit together. Just removing the beta tag is not a fix for #595 which is somehow what is the one commit=one release model.

@bric3
Contributor
bric3 commented Oct 21, 2016

@philwebb @tinkerware @jeantil

Currently the default branch on GH has been changed to be release/2.x instead of master. Indeed @ChristianSchwarz noticed the contributing guide and pr/issue template don't mention the different which branch to target, I will update those.

@szczepiq
Member

Thank you guys!, the feedback is really amazing!!!

Just to clarify: releasing every commit != releasing every change. Mockito does the latter.

@tinkerware, thank you for sharing the versioneye use case - this is exactly the feedback that we need. Releasing every couple of weeks, plus some extra critical bugfixes, can still create users annoyed by outdated dependencies reported by tools like versioneye. Optimizing Mockito release model for tools like versioneye is does not seem right. I'd rather versioneye supported better libraries that move fast and adopted continuous delivery.

@tinkerware, I agree dependency management is work. That's why Mockito is completely backwards compatible and rigorously follows semantic versioning. This should streamline managing Mockito dependency and if it does not, we would love hear about the use cases. Also, managing traditional releases is more work for us == less time for working on features.

@philwebb, thank you for taking the time and describe the spring boot release model! I think Mockito has sufficiently different profile than spring boot that we can have simpler release model (e.g. I agree with @tinkerware's reply).

@philwebb, blessed version of Mockito is always the latest. We ensure 100% backwards compatibility, always. I understand that for spring boot it would be preferable to integrate with library that moves at similar pace.

@jeantil, your idea about automated, cadence releases is super interesting. I want to explore this idea further: #708

@szpak, I really like the techniques and tools that help engineers be up-to-date with their dependencies automatically. Thanks for the pointers!

@bric3, thank you for clearly describing your desired release model! Also, well spotted bug in the test code I submitted yesterday!

@TimvdLippe, thanks for upvotes and looking forward to your code reviews!

All, I wrote a wiki page combining my motivation and the feedback gathered in this ticket: Continuous Delivery Overview.

Next steps:

  • I'd like us maintain continuous delivery model this quarter, revisit it early next year as we keep collecting great feedback
  • scope @jeantil's idea #708

THANK YOU for feedback. This is extremely valuable discussion.

@bric3
Contributor
bric3 commented Oct 25, 2016

Just to clarify: releasing every commit != releasing every change. Mockito does the latter.

It's a play of word, a commit represents changes and it applies to normal commits as well as on merge commits.
Let's just rephrase the controversial release model here to every reflog change.

We ensure 100% backwards compatibility, always.

Errors happened before in this every change model. This is something we cannot guarantee this.

@bric3
Contributor
bric3 commented Nov 3, 2016

Just to spill oil on the fire : Here's what can happen on every change : https://twitter.com/yanncebron/status/794167202303930368

linkedin-null-tweet

With this model I am not enticed to do any important change.

@bric3
Contributor
bric3 commented Nov 10, 2016

In my opinion like the opinion of others, we want to avoid is this case : 4 release in 2 days

2.2.15 (2016-11-08 20:43 UTC)

Authors: 2
Commits: 4
3: Brice Dutheil
1: Szczepan Faber
Improvements: 1
Enhancements: 1
Allow fluent usage of Mockito.framework() listeners methods (#747)
2.2.14 (2016-11-08 20:11 UTC)

Authors: 1
Commits: 1
1: Dmitriy Zaitsev
Improvements: 1
Add missing copyright headers (#746)
2.2.13 (2016-11-07 22:59 UTC)

Authors: 1
Commits: 1
1: Brice Dutheil
Improvements: 1
Removes deprecated way of documenting package by package-info.java (#745)
2.2.12 (2016-11-07 04:29 UTC)

Authors: 1
Commits: 11
11: Szczepan Faber
Improvements: 1
Enhancements: 1
Exposed existing verification API so that it no longer leaks internal API (#735)

While we could have slightly bigger releases. I'm not sure there's great value for developers to release Copyright headers or Improved exception message.

@TimvdLippe
Contributor

The past few days I have received a lot of questions regarding the overload of Mockito releases. After I explain the reasoning for this model, there has not been any developer that agrees with this model ๐Ÿ˜ข Developer response is therefore unanimously negative according to my knowledge.

@szpak
Member
szpak commented Nov 10, 2016 edited

@TimvdLippe Are you referring to Mockito 2.0-beta (too many releases) or CD in FOSS projects in general?

@szpak
Member
szpak commented Nov 10, 2016 edited

While we could have slightly bigger releases. I'm not sure there's great value for developers to release Copyright headers or Improved exception message.

@bric3 Probably most of us agree on that. New features and bug fixes should be considered as a candidate for the new releases. In other cases a person making a merge should add a tag a merge message to skip the release process. If it would be too hard to follow I personally prefer the opposite way - when a person merging a PR considers it as something important to be release a "do release" tag is added to a merge message and only then a new version is released (automatically, but on demand).

@TimvdLippe
Contributor

The amount of production releases (2.X.X) currently published. E.g. we are already at 2.2.14.

@TimvdLippe
Contributor

@szpak actually the manual release was implemented with including [ci-release 2.3.0] in the commit message. This has been reverted later.

@szpak
Member
szpak commented Nov 10, 2016 edited

I thought that [ci-release 2.3.0] was only to override the next version (and by default a release with automatic increment had been performed automatically for every commit). However, I didn't analyze that code, so you may be right @TimvdLippe .

@dnozay
dnozay commented Nov 28, 2016

I strongly dislike "one commit = one release":

  • Changelogs are now meaningless, it's more of a challenge (actually: more tedious) if you need to figure out what changed between two versions.
  • Making a release for doc / comment fixes is plain retarded; while users looking at the docs will appreciate, product managers and tech leads that actually track dependencies will hate you.
  • There are product managers and tech leads, whose responsibilities involve actually tracking dependencies and making sure the technological debt is low; they will hate you; mockito is mostly a test-level dependency, so it may be a low-priority item on their radar. But imagine a scenario if there were a bunch of iOS developers working on a product, and they had to deal with Xcode releases for every commit.
  • Changing a dependency may mean that you have to trigger many different QA cycles; not everything everywhere is automated. This is an expensive activity, and is a driver for not keeping up-to-date often.

I totally get the idea that releasing often is great to push new fixes / features out, but there is version fatigue. The opposite of this "one commit = one release" are LTS versions. People like them. Businesses like those; I know a few that will only update from LTS to LTS.

@TimvdLippe
Contributor

@dnozay Thanks a lot for voicing your opinion! I agree with you and I think this situation is actually more harmful than the benefit of continuous delivery.

Thus far we have been unable to come to an agreement or concession regarding the current practice of continuous delivery ๐Ÿ˜ข Hopefully we can come up with a solution that prevents your situation from happening.

@szczepiq
Member
szczepiq commented Dec 2, 2016

@dnozay, thanks for feedback! I fully share your passion and enthusiasm to release automation, while having different opinion on the way :)

What version of Mockito do you use? How often do you upgrade? How is version fatigue impacting you?

@dnozay
dnozay commented Dec 7, 2016

Yes, I love release automation.

What version of Mockito do you use? How often do you upgrade? How is version fatigue impacting you?

I don't use Mockito directly. I got to this page based on some email thread.

But let me share some of my experience doing release management. Maybe you know about this thing called Jenkins that helps people do continuous integration (here is their Governance document). They are very successful at releasing often, while being also very good for "scaredy-cat" customers: they do provide LTS versions.

How much do I do with it? Not much, but I used to maintain a few Jenkins plugins (e.g. build-flow-plugin), and the release model follows much of the Jenkins documentation for plugins (see also the Extend Jenkins page); it uses the Maven Release plugin. We can summarize that to:

  • select a bunch of changes, "Cut the changelog",
  • perform all kinds of testing on it, regression, backwards-compat, integration, end-to-end, etc.
  • do a mvn release:prepare release:perform -Dusername=... -Dpassword=...

How often do you upgrade?

The main factors are

  • Time and money : upgrading is not free.
  • Urgency : is this a security related release?
  • Is this a minor version / is this backwards-compatible?
  • Do I need a full qualification round, or can I save time (or $) doing smoke tests?
  • Am I myself in a release cycle, doing my own testing? Is that going to invalidate prior results?

I typically will NOT upgrade unless security fixes or I intend to pull new features, or just came back from vacation and need to catch up or have extra time to spend on conflict resolution.

Upgrading costs time and money.
If you think about your dependencies, it is a graph of dependencies, not a tree. You can have conflicting packages. I mean there is a reason behind excluding transitive dependencies in gradle. Sometimes different implementations of the same classes end up on the classpath. Sometimes the order matters and you spent time down the rabbit-hole trying to figure out classpath issues.

Have you worked for a company that releases an OS? Have you seen build-to-build upgrade testing? - I have. When you make the assumption that there is some common base / thing that should work (e.g. your OS is the same), you may think testing is fine. Now imagine that you cannot trust the OS; that you need to re-install it - just because that's the thing you are testing.

Too many releases is just noise
Your consumers probably won't care if you are releasing more than once a day.
If it takes 2 days to test; they can't physically handle upgrading more than every 2 days unless they have multiple teams / staggered release cycles.

How is version fatigue impacting you?

I only have so much time I can allocate.

  • Updating dependencies takes time (compile time, test time, ...) which either makes me busy, or ties resources that I can't use for other activities; e.g. I can't disconnect my laptop from the network and hop on a commuter bus.
  • The problem scales non-linearly with more dependencies.
  • Sometimes I have to resolve conflicts due to API changes, that's extra.

I just want to code my own stuff, not waste time on "upgrade" activities, so on a personal level, I hate it with a passion words cannot fully describe. Things are usually fine when the upgrade is smooth and silent; but that's not the common case for me.

Why are things not smooth?
That's a question that was missing. Because the bar for quality for a library/tool is low in general (I am not talking about Mockito here, so don't get mad at me).

  • lack of unit tests / poor coverage,
  • lack of integration tests,
  • shifting responsibility of testing to the consumer,
  • unstable APIs; lack of semantic versioning,
  • not understanding your consumers' use cases.

Some cool things to look at:

@mattnelson

That's why Mockito is completely backwards compatible and rigorously follows semantic versioning. This should streamline managing Mockito dependency and if it does not, we would love hear about the use cases. Also, managing traditional releases is more work for us == less time for working on features.

Coming in a little late to this discussion as we just uplifted to Mockito 2 last week and found some quirks with the continuous aspects of this project in the first week. We have a CI job that runs every week and creates a PR to auto update all of our dependencies. When the script ran this week we got prompted to update to 2.3.7[1]. When we attempted to validate the build, it failed with a transient mockito error. We kicked off the build again and it went away. But in parallel to the build running, we attempted to look at the changes for 2.3.7 to see what changed and if we needed to make code changes, the tag/release was no where to be found. 2.3.8 was missing as well. In that time 2.3.9 and 2.3.10 were tagged, but are not available in maven central at the time of this posting.

That left us with the following options:

  • Lock the dependency to our current known working version (2.2.29)
  • Work back from 2.3.6 until we find a working version
  • Wait for 2.3.9+ to be available on maven central and see if the build passes.

I don't want to discourage the release automation process especially if it makes it easier for maintainers and encourages contributors as they will be able to use their changes quicker.
But I agree with some of the other sentiments in this thread that less frequent and more vetted releases would be preferable.

Transient error that occurred with 2.3.7. Looks related to #638, but we are not using the internal APIs.

java.lang.NoClassDefFoundError: Could not initialize class org.mockito.internal.util.MockUtil

[1] http://search.maven.org/#artifactdetails%7Corg.mockito%7Cmockito-core%7C2.3.7%7Cjar
[2] https://github.com/mockito/mockito/tags

@TimvdLippe
Contributor

@mattnelson Sorry for the trouble that we caused you! There was a problem with releasing 2.3.7 as we are preparing the release procedure for the new mockito-android. The plugin we use to push artifacts to Bintray had a bug which was uploading corrupt jars to maven central. We identified the error quickly and retracted the corrupt versions, which was probably what you encountered too.

The release procedure should be fixed once again, but I think we can see this is a learning point that if we update the release procedure we must first test on our Bintray test project. Once we can install correctly ourselves, we push out to maven central. @szczepiq WDYT?

@szczepiq
Member

@mattnelson, very sorry for problems you encountered. We had some hiccups recently as described by @TimvdLippe. Do you happen to have the stack trace of the transient issue? Can you gist it for us?

@TimvdLippe, let's chat on slack channel. Can you formulate the release model that you have in mind when you say "must first test on our Bintray test project".

@mattnelson - thanks so much for taking the time and reporting the issue back to us. This feedback helps us a lot make decisions on how to release safely.

@drewhannay

I agree with pretty much everything @dnozay said. Upgrading Mockito is now very painful. There's no good release notes or upgrade instructions, everything is spread out across many many versions. I don't have the time or desire to look at every commit individually to find out what changed.

I understand that following semantic versioning means that we should be able to "trust" and drop in any newer compatible version. Now I may trust you all to stick to that, but my team of 50+ developers certainly does not. They want to see a list of what has changed and what benefits we get from upgrading that make it worth the risk. The current release cycle means we're much less likely to upgrade until there's a meaningful reason to do so. But with so many versions, how will I ever notice that there's a good reason to upgrade?!?

Right now my method is to follow @szczepiq on LinkedIn and Twitter to find out when there's a cool new feature and then use my own discretion to decide if it's worth spending half a day of my time to upgrade and validate a new version of Mockito. This seems....less than optimal ๐Ÿ˜‰

@szczepiq
Member
szczepiq commented Jan 5, 2017

@drewhannay, fantastic feedback. Thank you for writing this up. You raised valid points and we will address them.

half a day of my time to upgrade and validate a new version of Mockito

Is updating minor version of Mockito half day of your work? Why?

@drewhannay

Mainly because Android tests are slow and we have a lot of them ๐Ÿ˜‰

In the best case, I end up doing at least two full test runs (~45 minutes each), plus time getting a code review and time for importing the new version of Mockito into our internal artifact server.

@szczepiq szczepiq referenced this issue Jan 29, 2017
Open

Mockito Continuous Delivery Pipeline 2.0 #911

0 of 7 tasks complete
@szczepiq
Member

I documented draft plan for new release cadence at issue #911 (interesting number, huh? coincidence? :). I would love to get feedback!

@tinkerware
tinkerware commented Feb 8, 2017 edited

As an anecdote, I want to talk about what I had to do just today when updating dependencies on a project using both Mockito and ByteBuddy.

I had Mockito on a recent enough 2.5.4 and a separate internal library had ByteBuddy on 1.6.0. When I upgrade Mockito to its latest as of today, I end up with Mockito on 2.7.5. ByteBuddy dependency from Mockito is at 1.6.5, however. Additionally, ByteBuddy 1.6.5 breaks public API with 1.6.0 by renaming at least one interface and one method (lest you think this is unusual, ByteBuddy has always broken public API in its patch versions; arguably, that's the root cause). Now suddenly, I have two compile-incompatible versions of the same library when I upgrade from Mockito 2.5.4 to 2.7.5.

I'm not bringing this particular example up to assign blame (I love ByteBuddy to pieces, even with its painful unpromise of public API compatibility; I have accepted that it's a high-maintenance library to keep up with) or to suggest changes to Mockito (though, in retrospect, Mockito can shade ByteBuddy as a private dependency to avoid passing down the pain of keeping up with it). I only wanted to point out that maintaining dependencies can be work, even hard work. Predictable, compatible releases are valuable because they make this work easier.

@szczepiq
Member

@tinkerware, this is extremely useful feedback. Thank you very much for reporting. I will start an internal discussion about the idea of shading the bytebuddy jar.

In the early days of Mockito, we used to ship with direct dependency on cglib. However, it was causing similar problems to our users and we started shading cglib. Bytebuddy is getting popular (rightfully so!) and will be used more more often. I want to make sure that mockito is always very convenient to use, and avoids potential incompatibilities that cause users to troubleshoot classpaths / dependency graphs.

@raphw
Member
raphw commented Feb 12, 2017 edited

@tinkerware Out of curiosity. What API of Byte Buddy do you rely on that broke since then? Typically, Byte Buddy changes its public API because it has to what happens with features that were recently added and had to prove their use in practice. This is a bit of a catch 22 where I consider anything new to be incubating without naming a maturity date or version as I simply do not know.

If there is any particuar API that broke for you, I would be curious to know!

@tinkerware

Out of curiosity. What API of Byte Buddy do you rely on that broke since then?

I had code that gathered annotations from multiple "levels" (e.g. declaring type, enclosing types, package) so I used AnnotatedCodeElement as the common type across functions that traversed declarations. That got renamed to AnnotationSource. Another one is the isGenericDeclaration method, renamed to isGenerified; that was part of a matcher predicate (to be clear, I like the new names better).

Typically, Byte Buddy changes its public API because it has to what happens with features that were recently added and had to prove their use in practice. This is a bit of a catch 22 where I consider anything new to be incubating without naming a maturity date or version as I simply do not know.

I genuinely sympathize with this, and I think shoehorning multiple API's, each in a different point in their lifecycle, into one artifact version number is eventually self-defeating. The late Pieter Hintjens had a good handle on defining this problem and one possible way forward, written up in End of Software Versions.

In general, I find the current state of tooling to manage API lifecycle miserable, but there is some low-hanging fruit. I have two suggestions: (1) You can mark new API that has to prove itself with a draft/beta API annotation, and (2) deprecate all API before removal. Deprecated API does not have to hang on for ages; I'm okay if beta API disappears in a minor version after a short period of deprecation. Only stable API would be relevant for semantic versioning.

I don't want to hijack this thread with non-Mockito discussions, so feel free to point me to another place to continue if you think it's worth it.

Thanks for ByteBuddy and Mockito!

@jeantil jeantil referenced this issue in olafurpg/scalafmt Feb 14, 2017
Open

Automate releases #721

0 of 5 tasks complete
@raphw
Member
raphw commented Feb 14, 2017

Thanks for your feedback. True, I had to change this with respect to making type annotation resolution lazy which is was not before. Its rather the exception than the rule but in this case, it was a customer-driven innovation which is partly a requirement for keeping the library alive. I agree that I should have not done this in a minor update but the incompatibility was difficult to avoid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment