New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mockito continuous delivery model #618

Closed
mockitoguy opened this Issue Sep 1, 2016 · 82 comments

Comments

Projects
None yet
@mockitoguy
Member

mockitoguy commented Sep 1, 2016

This issue discusses:

  • Mockito continuous delivery model (e.g. publishing every PR: wiki link)
  • How to prevent prolonged Beta phases as it happened for v2
@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Sep 1, 2016

Contributor

Strict set of issues we want to solve and stick to that. Once they are resolved, publish the version and define a new strict set.

We should not add issues to the current milestone, unless they are absolutely n1 priority if it is a huge bug. But this probably rarely happens.

E.g. for 2.2.0 we are already set, I think this list is long enough: https://github.com/mockito/mockito/milestone/3. This milestone is very clear in its goal: improve the ArgumentCaptor API. That we have a release focused on an issue. For major releases, we can of course adopt more than 1 focus. That's my 2 cents.

Contributor

TimvdLippe commented Sep 1, 2016

Strict set of issues we want to solve and stick to that. Once they are resolved, publish the version and define a new strict set.

We should not add issues to the current milestone, unless they are absolutely n1 priority if it is a huge bug. But this probably rarely happens.

E.g. for 2.2.0 we are already set, I think this list is long enough: https://github.com/mockito/mockito/milestone/3. This milestone is very clear in its goal: improve the ArgumentCaptor API. That we have a release focused on an issue. For major releases, we can of course adopt more than 1 focus. That's my 2 cents.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Sep 2, 2016

Member

Good feedback. I'm not worried about v2 releases. Since they are compatible, we can simply iterate minor or patch version and release without any beta versions.

However, 3.0.0-Beta.x worries me. What can we do to avoid prolonged v3 beta phase.

Member

mockitoguy commented Sep 2, 2016

Good feedback. I'm not worried about v2 releases. Since they are compatible, we can simply iterate minor or patch version and release without any beta versions.

However, 3.0.0-Beta.x worries me. What can we do to avoid prolonged v3 beta phase.

@bric3

This comment has been minimized.

Show comment
Hide comment
@bric3

bric3 Sep 2, 2016

Contributor

I think the same would apply e.g. define a set of goals for 3.0. Currently 3.0 scope is a bit fuzzy, we know that we want Java 8, maybe JUnit 5.

Also note that the prolonged 2.0 phase was also due to our life. You had to move to LinkedIn, I was overwhelmed by own professional agenda. That is somehow outside our control. I don't know about your plans about mockito in the future but on my side I have more spare time recently, but that may change.

Contributor

bric3 commented Sep 2, 2016

I think the same would apply e.g. define a set of goals for 3.0. Currently 3.0 scope is a bit fuzzy, we know that we want Java 8, maybe JUnit 5.

Also note that the prolonged 2.0 phase was also due to our life. You had to move to LinkedIn, I was overwhelmed by own professional agenda. That is somehow outside our control. I don't know about your plans about mockito in the future but on my side I have more spare time recently, but that may change.

@szpak

This comment has been minimized.

Show comment
Hide comment
@szpak

szpak Sep 3, 2016

Member

I think the same would apply e.g. define a set of goals for 3.0.

Having set a minimal scope with should be done before 3.0 is a good idea. However, I would not limit accepted changes to just those things. Any sensible PR should be merged (as 2.x will be in the maintenance mode). In general, a lot of versions (here beta) is a side effect of continuous delivery. On the other hand I prefer it over working with snapshots/jitpack just to have given fresh feature available in my tests.

Member

szpak commented Sep 3, 2016

I think the same would apply e.g. define a set of goals for 3.0.

Having set a minimal scope with should be done before 3.0 is a good idea. However, I would not limit accepted changes to just those things. Any sensible PR should be merged (as 2.x will be in the maintenance mode). In general, a lot of versions (here beta) is a side effect of continuous delivery. On the other hand I prefer it over working with snapshots/jitpack just to have given fresh feature available in my tests.

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Sep 3, 2016

Contributor

If there is a PR which can be merged as-is, yes of course we should merge it. However, I think the current open PRs regarding ArgumentCaptor is what I meant with the strict set. Since this class was not included in 2.0.0, we decided to move all these PRs to 2.2.0. If there is a lot of work tight to a certain feature (or set of features), I do not think we should blatantly accept it for 3.0.0. That way we can deliver 3.0.0 on time, while also have a solid plan for later releases.

Of course we have to keep in mind the semver scheme. If we know there are breaking changes required, we might have to do those together in 3.0.0 because we can't introduce them in 3.1.0.

Contributor

TimvdLippe commented Sep 3, 2016

If there is a PR which can be merged as-is, yes of course we should merge it. However, I think the current open PRs regarding ArgumentCaptor is what I meant with the strict set. Since this class was not included in 2.0.0, we decided to move all these PRs to 2.2.0. If there is a lot of work tight to a certain feature (or set of features), I do not think we should blatantly accept it for 3.0.0. That way we can deliver 3.0.0 on time, while also have a solid plan for later releases.

Of course we have to keep in mind the semver scheme. If we know there are breaking changes required, we might have to do those together in 3.0.0 because we can't introduce them in 3.1.0.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Sep 9, 2016

Member

Feedback from users: #595 (comment)

Member

mockitoguy commented Sep 9, 2016

Feedback from users: #595 (comment)

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Sep 9, 2016

Member

Perhaps we should avoid pushing beta versions to central / jcenter and only push them to internal bintray repo.

Member

mockitoguy commented Sep 9, 2016

Perhaps we should avoid pushing beta versions to central / jcenter and only push them to internal bintray repo.

@bric3

This comment has been minimized.

Show comment
Hide comment
@bric3

bric3 Sep 9, 2016

Contributor

Bintray can serve as a maven repo I believe, https://bintray.com/docs/usermanual/formats/formats_mavenrepositories.html#_working_with_maven

So that may be a solution.

Contributor

bric3 commented Sep 9, 2016

Bintray can serve as a maven repo I believe, https://bintray.com/docs/usermanual/formats/formats_mavenrepositories.html#_working_with_maven

So that may be a solution.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Sep 10, 2016

Member

Options to consider:

  1. don't publish Betas at all
  2. don't publish to central, publish to some less prominent public repo
  3. publish some betas to central, manually chosen
  4. publish some betas to central, automatically at certain cadence / when specific release milestones are completed

I hope 1 year from now we won't be publishing 2.1.0-RC.123 :)

Member

mockitoguy commented Sep 10, 2016

Options to consider:

  1. don't publish Betas at all
  2. don't publish to central, publish to some less prominent public repo
  3. publish some betas to central, manually chosen
  4. publish some betas to central, automatically at certain cadence / when specific release milestones are completed

I hope 1 year from now we won't be publishing 2.1.0-RC.123 :)

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Sep 10, 2016

Contributor

On master I bumped the version properties to 3.0.0-beta.1 and disabled the release procedure for now until we have developed a stable 3.X beta and resolved this issue.

Contributor

TimvdLippe commented Sep 10, 2016

On master I bumped the version properties to 3.0.0-beta.1 and disabled the release procedure for now until we have developed a stable 3.X beta and resolved this issue.

@bric3

This comment has been minimized.

Show comment
Hide comment
@bric3

bric3 Sep 10, 2016

Contributor

I think we should address this issue before any further development.

Contributor

bric3 commented Sep 10, 2016

I think we should address this issue before any further development.

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Sep 11, 2016

Contributor

I think the cleanest solution to have a separate bintray project mockito-beta which has all beta releases. This should be fairly easy to build in our release process, is minimally impactful for our beta testers (it's just 1 word change) and does not clutter the official releases.

Contributor

TimvdLippe commented Sep 11, 2016

I think the cleanest solution to have a separate bintray project mockito-beta which has all beta releases. This should be fairly easy to build in our release process, is minimally impactful for our beta testers (it's just 1 word change) and does not clutter the official releases.

@ChristianSchwarz

This comment has been minimized.

Show comment
Hide comment
@ChristianSchwarz

ChristianSchwarz Sep 15, 2016

Contributor

Options to consider:

  1. don't publish Betas at all
  2. don't publish to central, publish to some less prominent public repo
  3. publish some betas to central, manually chosen
  4. publish some betas to central, automatically at certain cadence / when specific release milestones are completed

5 . release beta's as different artifact e.g. mockito-core-beta to central?

Contributor

ChristianSchwarz commented Sep 15, 2016

Options to consider:

  1. don't publish Betas at all
  2. don't publish to central, publish to some less prominent public repo
  3. publish some betas to central, manually chosen
  4. publish some betas to central, automatically at certain cadence / when specific release milestones are completed

5 . release beta's as different artifact e.g. mockito-core-beta to central?

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Sep 15, 2016

Contributor

@ChristianSchwarz Yes that was the option I described in #618 (comment)

Contributor

TimvdLippe commented Sep 15, 2016

@ChristianSchwarz Yes that was the option I described in #618 (comment)

@bric3

This comment has been minimized.

Show comment
Hide comment
@bric3

bric3 Sep 16, 2016

Contributor

I like the idea of beta specific maven coordinate. However not sure we need another bintray account. Though it's not an issue if we have one. Anyway we will have to create one mockito account anyway see #631

Contributor

bric3 commented Sep 16, 2016

I like the idea of beta specific maven coordinate. However not sure we need another bintray account. Though it's not an issue if we have one. Anyway we will have to create one mockito account anyway see #631

@raphw

This comment has been minimized.

Show comment
Hide comment
@raphw

raphw Sep 16, 2016

Member

Or we do it the official Maven way and publish SNAPSHOT versions to a snapshot repository.

Member

raphw commented Sep 16, 2016

Or we do it the official Maven way and publish SNAPSHOT versions to a snapshot repository.

@szpak

This comment has been minimized.

Show comment
Hide comment
@szpak

szpak Sep 16, 2016

Member

SNAPSHOTs were published in the past. I personally don't like and don't trust them. With beta releases it is the package that will not change and will be available "forever". Snapshots (even those with timestamp) can are cleaned up after some time.

A separate repo could be a solution if we really plan to have a plethora of beta versions again :).

Member

szpak commented Sep 16, 2016

SNAPSHOTs were published in the past. I personally don't like and don't trust them. With beta releases it is the package that will not change and will be available "forever". Snapshots (even those with timestamp) can are cleaned up after some time.

A separate repo could be a solution if we really plan to have a plethora of beta versions again :).

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Sep 17, 2016

Member

Hearty -1 to different artifact name. It's very inconventional and confusing. It could lead problems with conflict resolution (both beta and normal artifact on the same classpath).

Separate bintray repo is the way to go.

However, it feels that separate bintray repo is not enough. We also need to plan releases better and drive them to resolution faster than in 1.5 years :)

Member

mockitoguy commented Sep 17, 2016

Hearty -1 to different artifact name. It's very inconventional and confusing. It could lead problems with conflict resolution (both beta and normal artifact on the same classpath).

Separate bintray repo is the way to go.

However, it feels that separate bintray repo is not enough. We also need to plan releases better and drive them to resolution faster than in 1.5 years :)

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Sep 17, 2016

Contributor

I will set up a Bintray account this weekend for #631 and then we can configure it 😄

Contributor

TimvdLippe commented Sep 17, 2016

I will set up a Bintray account this weekend for #631 and then we can configure it 😄

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Sep 17, 2016

Contributor

Oh forgot to add regarding the number of releases: The issue for Mockito 2 was that no strict feature set was defined. If we define a strict set and work towards finishing it, it should be possible to publish releases every ~2 months. This is a lot better than the 1.5 years :P. I think we already defined what Mockito 2.2.0 must incorporate, so we can start working on that in the release/2.x branch.

Contributor

TimvdLippe commented Sep 17, 2016

Oh forgot to add regarding the number of releases: The issue for Mockito 2 was that no strict feature set was defined. If we define a strict set and work towards finishing it, it should be possible to publish releases every ~2 months. This is a lot better than the 1.5 years :P. I think we already defined what Mockito 2.2.0 must incorporate, so we can start working on that in the release/2.x branch.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Sep 17, 2016

Member

Mockito 2 continously publishes. Hence, there is no risk of prolonged beta :)

Member

mockitoguy commented Sep 17, 2016

Mockito 2 continously publishes. Hence, there is no risk of prolonged beta :)

bric3 added a commit that referenced this issue Oct 3, 2016

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Oct 9, 2016

Member

Hiding betas from default repositories and having strict feature set for milestones is not good, nor sufficient solution. We no longer continuously deliver which means harder releases, more decision points, more project management, deferred QA, slower to get features out, and risk when contributors deprioritize half finished milestones. When we continuously deliver we get effortless, cheap, seamless releases so that we can focus on features.

Effortless releases is not a negotiable feature - it's required for sustainability of OS projects.

The problem with 2.0 beta was that I pulled out the beta version and then deprioritized Mockito for next x months.

Let's make a thought experiment, what happens if core contributors deprioritize OS development today. What is the desired state of Mockito project, so that such scenario has least impact on community and Mockito itself?

Desired state is: contributors can easily ship features in current Mockito (2.x)

Member

mockitoguy commented Oct 9, 2016

Hiding betas from default repositories and having strict feature set for milestones is not good, nor sufficient solution. We no longer continuously deliver which means harder releases, more decision points, more project management, deferred QA, slower to get features out, and risk when contributors deprioritize half finished milestones. When we continuously deliver we get effortless, cheap, seamless releases so that we can focus on features.

Effortless releases is not a negotiable feature - it's required for sustainability of OS projects.

The problem with 2.0 beta was that I pulled out the beta version and then deprioritized Mockito for next x months.

Let's make a thought experiment, what happens if core contributors deprioritize OS development today. What is the desired state of Mockito project, so that such scenario has least impact on community and Mockito itself?

Desired state is: contributors can easily ship features in current Mockito (2.x)

@bric3

This comment has been minimized.

Show comment
Hide comment
@bric3

bric3 Oct 10, 2016

Contributor

And we introduce buggy release or instable API, this has already happened on mockito.

Effortless releases is not a negotiable feature - it's required for sustainability of OS projects.

Agreed, CI and CD is one of key to success for any project, from the smallest to the biggest. Nobody denies that. However each project has a constraints and different goals.

  • If the project is simple a website, something's wrong, let's modify the HTML/PHP webpage and we're good
  • If the project is a huge website, then maybe we'll do some QA tests before
  • If the project is the software that control a car engine, then I will sure make proper tests...
  • If the project is used by 10 people that follow closely the thing then hell yay it's possible to introduce incompatibilities
  • If the project is used by hundred thousand, then maybe it's my job to avoid include poor release because a PR has been merged, and to polish the API.
  • If project is javascript, then well, things break

Also...

Developer productivity is a nice thing, if their environment is stable and intuitive they work on their own project so that the project is successful and make money. If the environment is not stable, changing versions all the time, API change in minor, then they focus less on what matter.

Let's not forget that we do not want to make things easy if it's to make them harder of users.

The point is that we like the CI-CD that you are building, but we are not convinced by the one commit = one release pace.

The problem with 2.0 beta was that I pulled out the beta version and then deprioritized Mockito for next x months.

Version 2 delay is a problem I share as well, I was pretty sick too and as such had other priorities in life. Then as usual work kicked in as well.

Let's make a thought experiment, what happens if core contributors deprioritize OS development today. What is the desired state of Mockito project, so that such scenario has least impact on community and Mockito itself?

Right now I'm more concerned by other things, the team don't have credentials to anything related to infrastructure, so we cannot improve them.

  • DNS
  • bintray
  • maven oss
  • control over mailing-lists
  • ❗️ continuous delivery bot mail account (although it's possible to make another)

What happens if for some reason the team loose the only member that have those ?

Desired state is: contributors can easily ship features in current Mockito (2.x)

Yes we know that and we want it flawless as well, but we also want time to polish features.
A pull request is a nice unit of work but not enough, mockito is not a webpage project.

(EDITED : added mailing-list, and added bold)

Contributor

bric3 commented Oct 10, 2016

And we introduce buggy release or instable API, this has already happened on mockito.

Effortless releases is not a negotiable feature - it's required for sustainability of OS projects.

Agreed, CI and CD is one of key to success for any project, from the smallest to the biggest. Nobody denies that. However each project has a constraints and different goals.

  • If the project is simple a website, something's wrong, let's modify the HTML/PHP webpage and we're good
  • If the project is a huge website, then maybe we'll do some QA tests before
  • If the project is the software that control a car engine, then I will sure make proper tests...
  • If the project is used by 10 people that follow closely the thing then hell yay it's possible to introduce incompatibilities
  • If the project is used by hundred thousand, then maybe it's my job to avoid include poor release because a PR has been merged, and to polish the API.
  • If project is javascript, then well, things break

Also...

Developer productivity is a nice thing, if their environment is stable and intuitive they work on their own project so that the project is successful and make money. If the environment is not stable, changing versions all the time, API change in minor, then they focus less on what matter.

Let's not forget that we do not want to make things easy if it's to make them harder of users.

The point is that we like the CI-CD that you are building, but we are not convinced by the one commit = one release pace.

The problem with 2.0 beta was that I pulled out the beta version and then deprioritized Mockito for next x months.

Version 2 delay is a problem I share as well, I was pretty sick too and as such had other priorities in life. Then as usual work kicked in as well.

Let's make a thought experiment, what happens if core contributors deprioritize OS development today. What is the desired state of Mockito project, so that such scenario has least impact on community and Mockito itself?

Right now I'm more concerned by other things, the team don't have credentials to anything related to infrastructure, so we cannot improve them.

  • DNS
  • bintray
  • maven oss
  • control over mailing-lists
  • ❗️ continuous delivery bot mail account (although it's possible to make another)

What happens if for some reason the team loose the only member that have those ?

Desired state is: contributors can easily ship features in current Mockito (2.x)

Yes we know that and we want it flawless as well, but we also want time to polish features.
A pull request is a nice unit of work but not enough, mockito is not a webpage project.

(EDITED : added mailing-list, and added bold)

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Oct 10, 2016

Contributor

My opinion is equal to the comment above of Brice.

Contributor

TimvdLippe commented Oct 10, 2016

My opinion is equal to the comment above of Brice.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Oct 11, 2016

Member

And we introduce buggy release or instable API, this has already happened on mockito.

The solution to this is making sure that every PR is high quality.

Yes we know that and we want it flawless as well, but we also want time to polish features.

Polish them as long as you want in a developer branch ;)


Thanks guys for your feedback! I'm afraid we're not making progress so let's agree that we don't agree.

  1. I'm for full CD pipeline and I'm executing towards that unless new arguments are brought up to the discussion. So far, there are no new arguments.
  2. I have the infra pieces on my TODO list ;)
  3. We will do traditional release model for Mockito 3.0.
Member

mockitoguy commented Oct 11, 2016

And we introduce buggy release or instable API, this has already happened on mockito.

The solution to this is making sure that every PR is high quality.

Yes we know that and we want it flawless as well, but we also want time to polish features.

Polish them as long as you want in a developer branch ;)


Thanks guys for your feedback! I'm afraid we're not making progress so let's agree that we don't agree.

  1. I'm for full CD pipeline and I'm executing towards that unless new arguments are brought up to the discussion. So far, there are no new arguments.
  2. I have the infra pieces on my TODO list ;)
  3. We will do traditional release model for Mockito 3.0.
@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Oct 11, 2016

Contributor

I'm for full CD pipeline and I'm executing towards that unless new arguments are brought up to the discussion. So far, there are no new arguments.

You are implying with "I'm executing" that the whole Mockito development team must act this way? I think we (the other three core developers) have brought enough new arguments to not use the full CD workflow (e.g. 1 commit is 1 minor release). I would kindly ask you to not force any ideas on us if we keep pointing out why we do not like them. Therefore I would prefer to have an agreement on our development workflow, which is essential for working together as a team.

Contributor

TimvdLippe commented Oct 11, 2016

I'm for full CD pipeline and I'm executing towards that unless new arguments are brought up to the discussion. So far, there are no new arguments.

You are implying with "I'm executing" that the whole Mockito development team must act this way? I think we (the other three core developers) have brought enough new arguments to not use the full CD workflow (e.g. 1 commit is 1 minor release). I would kindly ask you to not force any ideas on us if we keep pointing out why we do not like them. Therefore I would prefer to have an agreement on our development workflow, which is essential for working together as a team.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Oct 11, 2016

Member

You are implying with "I'm executing" that the whole Mockito development team must act this way?

Of course not! If you don't want to release every push, please work on developer branch.

I will continue to improve the CD pipeline and make changes to support it unless new arguments are brought to the table.

Member

mockitoguy commented Oct 11, 2016

You are implying with "I'm executing" that the whole Mockito development team must act this way?

Of course not! If you don't want to release every push, please work on developer branch.

I will continue to improve the CD pipeline and make changes to support it unless new arguments are brought to the table.

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Oct 11, 2016

Contributor

That means that whenever a contributor submits a pullrequest, we need to
request him/her to point it to our development branch. This would be a huge
burden for contributors.

Op di 11 okt. 2016 om 16:25 schreef Szczepan Faber <notifications@github.com

:

You are implying with "I'm executing" that the whole Mockito development
team must act this way?

Of course not! If you don't want to release every push, please work on
developer branch.

I will continue to improve the CD pipeline and make changes to support it
unless new arguments are brought to the table.


You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#618 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFrDb5oZMpYcri5DazYTZ7wvommr8BFzks5qy5xygaJpZM4JytNn
.

Contributor

TimvdLippe commented Oct 11, 2016

That means that whenever a contributor submits a pullrequest, we need to
request him/her to point it to our development branch. This would be a huge
burden for contributors.

Op di 11 okt. 2016 om 16:25 schreef Szczepan Faber <notifications@github.com

:

You are implying with "I'm executing" that the whole Mockito development
team must act this way?

Of course not! If you don't want to release every push, please work on
developer branch.

I will continue to improve the CD pipeline and make changes to support it
unless new arguments are brought to the table.


You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#618 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFrDb5oZMpYcri5DazYTZ7wvommr8BFzks5qy5xygaJpZM4JytNn
.

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Oct 12, 2016

Contributor

Okay since the previous arguments did not convince you yet, I would like to
sketch the scenario which depicts why we are uncomfortable with the
workflow.

First of all there is probably some misunderstanding regarding the naming.
We are all for continuous delivery, we are against continuous deployment.
With delivery we have always a working version available as snapshots (or
beta whatever you want to call it). With deployment you immediately publish
it as an official version.

The scenario which has very undesirable results is when multiple PRs are
merged in a short period of time; lets say a couple of hours. With
continuous delivery, the snapshots are generated and available. However
with continuous deployment they are ALL published to production. This means
that in a matter of a couple of hours, we generated a number of minor
versions. This is confusing for end users, a huge burden on updating the
Mockito dependency and as maintainer annoying to manage.

For these reasons, we dislike continuous deployment. We are all for
delivery. I hope this makes clear the standpoint.

Denoting that we can use separate workflows at the same time will not work.
Thats taking the bad sides of both solutions together.

On Tue, 11 Oct 2016, 16:39 Tim van der Lippe, tvanderlippe@gmail.com
wrote:

That means that whenever a contributor submits a pullrequest, we need to
request him/her to point it to our development branch. This would be a huge
burden for contributors.

Op di 11 okt. 2016 om 16:25 schreef Szczepan Faber <
notifications@github.com>:

You are implying with "I'm executing" that the whole Mockito development
team must act this way?

Of course not! If you don't want to release every push, please work on
developer branch.

I will continue to improve the CD pipeline and make changes to support it
unless new arguments are brought to the table.


You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#618 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFrDb5oZMpYcri5DazYTZ7wvommr8BFzks5qy5xygaJpZM4JytNn
.

Contributor

TimvdLippe commented Oct 12, 2016

Okay since the previous arguments did not convince you yet, I would like to
sketch the scenario which depicts why we are uncomfortable with the
workflow.

First of all there is probably some misunderstanding regarding the naming.
We are all for continuous delivery, we are against continuous deployment.
With delivery we have always a working version available as snapshots (or
beta whatever you want to call it). With deployment you immediately publish
it as an official version.

The scenario which has very undesirable results is when multiple PRs are
merged in a short period of time; lets say a couple of hours. With
continuous delivery, the snapshots are generated and available. However
with continuous deployment they are ALL published to production. This means
that in a matter of a couple of hours, we generated a number of minor
versions. This is confusing for end users, a huge burden on updating the
Mockito dependency and as maintainer annoying to manage.

For these reasons, we dislike continuous deployment. We are all for
delivery. I hope this makes clear the standpoint.

Denoting that we can use separate workflows at the same time will not work.
Thats taking the bad sides of both solutions together.

On Tue, 11 Oct 2016, 16:39 Tim van der Lippe, tvanderlippe@gmail.com
wrote:

That means that whenever a contributor submits a pullrequest, we need to
request him/her to point it to our development branch. This would be a huge
burden for contributors.

Op di 11 okt. 2016 om 16:25 schreef Szczepan Faber <
notifications@github.com>:

You are implying with "I'm executing" that the whole Mockito development
team must act this way?

Of course not! If you don't want to release every push, please work on
developer branch.

I will continue to improve the CD pipeline and make changes to support it
unless new arguments are brought to the table.


You are receiving this because you commented.

Reply to this email directly, view it on GitHub
#618 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFrDb5oZMpYcri5DazYTZ7wvommr8BFzks5qy5xygaJpZM4JytNn
.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Oct 12, 2016

Member

That means that whenever a contributor submits a pullrequest, we need to
request him/her to point it to our development branch. This would be a huge
burden for contributors.

On the contrary, it's much easier for contributors of high quality PRs to get their changes published quickly.

Okay since the previous arguments did not convince you yet

This argument has been brought up before. I don't discount it I have a different point of view. It's mockito mission to pioneer continuous delivery for Java libraries. Consumers don't have to update to the latest version every day. It's up to them.

You brought up the point of decision making in Mockito team. There is no formal process, we make decisions organically. There is no democracy, there is no despotism, either. The more contributions one has over all-time (and how engaged he is currently), the stronger his opinion matters. There is no mathematical formula, I can give an example: If I deprioritize mockito and you don't hear from me for 2 months, @bric3 becomes most senior engineer (level of contributions over time and current engagement) and his opinion weighs most. @TimvdLippe, you're next :)

Member

mockitoguy commented Oct 12, 2016

That means that whenever a contributor submits a pullrequest, we need to
request him/her to point it to our development branch. This would be a huge
burden for contributors.

On the contrary, it's much easier for contributors of high quality PRs to get their changes published quickly.

Okay since the previous arguments did not convince you yet

This argument has been brought up before. I don't discount it I have a different point of view. It's mockito mission to pioneer continuous delivery for Java libraries. Consumers don't have to update to the latest version every day. It's up to them.

You brought up the point of decision making in Mockito team. There is no formal process, we make decisions organically. There is no democracy, there is no despotism, either. The more contributions one has over all-time (and how engaged he is currently), the stronger his opinion matters. There is no mathematical formula, I can give an example: If I deprioritize mockito and you don't hear from me for 2 months, @bric3 becomes most senior engineer (level of contributions over time and current engagement) and his opinion weighs most. @TimvdLippe, you're next :)

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil Oct 12, 2016

To sum up the disagreement from an external point of view :
why should the frequency of releases be linked to commit frequency ?

continuous != on every commit ( or at least not necessarily )
continuous == regularly unless there is nothing to deliver

Fixed timings could work too :

  • daily for release for beta/snapshot whatever
  • bi-monthly for releases to official repo

Whatever timing you want to use, you can go for a mixed strategy:
tag a release after 10 commits or 2 weeks (this way even if contributor activity is low, a user never has to wait more than 2 weeks to get his merged feature in the central repo)

These strategies can still be considered as continuous since they don't depend on a manual action. Yet

  • they allow for an opportunity window to fix unavoidable human mistake,
  • they allow for predictable release cycles for the users (includes reduced confusion for non hardcore users)
  • they don't preclude polishing in developer branch

2cents, have fun.

jeantil commented Oct 12, 2016

To sum up the disagreement from an external point of view :
why should the frequency of releases be linked to commit frequency ?

continuous != on every commit ( or at least not necessarily )
continuous == regularly unless there is nothing to deliver

Fixed timings could work too :

  • daily for release for beta/snapshot whatever
  • bi-monthly for releases to official repo

Whatever timing you want to use, you can go for a mixed strategy:
tag a release after 10 commits or 2 weeks (this way even if contributor activity is low, a user never has to wait more than 2 weeks to get his merged feature in the central repo)

These strategies can still be considered as continuous since they don't depend on a manual action. Yet

  • they allow for an opportunity window to fix unavoidable human mistake,
  • they allow for predictable release cycles for the users (includes reduced confusion for non hardcore users)
  • they don't preclude polishing in developer branch

2cents, have fun.

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Oct 12, 2016

Contributor

@jeantil Thanks a lot for joining the discussion and expression your opinion. It is greatly appreciated!

Contributor

TimvdLippe commented Oct 12, 2016

@jeantil Thanks a lot for joining the discussion and expression your opinion. It is greatly appreciated!

@tinkerware

This comment has been minimized.

Show comment
Hide comment
@tinkerware

tinkerware Oct 13, 2016

This is an interesting discussion; thanks for having it in the open.

I've been using the 2.x versions of Mockito for the last six months. I will admit to being mildly annoyed at the pace new beta versions have appeared, and I was cross at VersionEye for incorrectly marking Mockito as out-of-date after I picked the latest stable release.

On the other hand, I am a lot more upset with a library if it does not make stable releases every few weeks (though this depends on pace of development) so that recent fixes and PR's make their way as soon as they can into the hands of users. For a library with the development pace of Mockito, I would want to see a new stable release at least once every few weeks. And further, if I'm contributing to the library or waiting on a fix, I appreciate greatly being able to access a release immediately so that I don't have to go to the trouble of publishing to an internal repo or vendoring the library.

Looking at the space for browsers, they seem to have come up with a workable model to balance these two needs with release channels. Users who value stable versions that do not move every day can use the stable channel to get access to reasonably often updates with fixes and features, and others can subscribe to the beta channel to get the full artifact-per-commit stream.

Thanks for the great work!

This is an interesting discussion; thanks for having it in the open.

I've been using the 2.x versions of Mockito for the last six months. I will admit to being mildly annoyed at the pace new beta versions have appeared, and I was cross at VersionEye for incorrectly marking Mockito as out-of-date after I picked the latest stable release.

On the other hand, I am a lot more upset with a library if it does not make stable releases every few weeks (though this depends on pace of development) so that recent fixes and PR's make their way as soon as they can into the hands of users. For a library with the development pace of Mockito, I would want to see a new stable release at least once every few weeks. And further, if I'm contributing to the library or waiting on a fix, I appreciate greatly being able to access a release immediately so that I don't have to go to the trouble of publishing to an internal repo or vendoring the library.

Looking at the space for browsers, they seem to have come up with a workable model to balance these two needs with release channels. Users who value stable versions that do not move every day can use the stable channel to get access to reasonably often updates with fixes and features, and others can subscribe to the beta channel to get the full artifact-per-commit stream.

Thanks for the great work!

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Oct 13, 2016

Contributor

@tinkerware I worked out a release model where I got my inspiration from the browser release process, you can read it at #681 (comment). Was this a model you had in mind?

Contributor

TimvdLippe commented Oct 13, 2016

@tinkerware I worked out a release model where I got my inspiration from the browser release process, you can read it at #681 (comment). Was this a model you had in mind?

@bric3

This comment has been minimized.

Show comment
Hide comment
@bric3

bric3 Oct 13, 2016

Contributor

On the contrary, it's much easier for contributors of high quality PRs to get their changes published quickly.

That's the thing, while I'm definitely pushing for high-quality PR, all are not hight quality for various reasons :

  • user don't know git well e.g. contributor do X non meaningful commits, history is unreadable and reason of change is obscure or split across various commits
  • Test coverage
  • API design is wrong or in-progress
  • Contributor don't reply

Or PR represents internal reworks without value, to prepare something bigger. Sometime we may find flaws in previous PR. Once an API goes in the wild it's hard to go back.

There is no mathematical formula, I can give an example: If I deprioritize mockito and you don't hear from me for 2 months, @bric3 becomes most senior engineer (level of contributions over time and current engagement) and his opinion weighs most. @TimvdLippe, you're next :)

While I appreciate that my opinions may matter because I've contributing, I think that regardless of level of contributions, any opinion can be just as good as anyone's.

It's mockito mission to pioneer continuous delivery for Java libraries.

No it is not! The mockito's mission is to make a quality library that allows to write good tests with mocks. Pioneering CI/CD allows to accomplish that main mission with ease, but it is not the main goal. Having the delivery upon each commit (or not) isn't pioneering or even innovation, it is a choice that is decided to match how a projects model its release pace.

@jeantil @tinkerware Interesting point of view, browser release model is indeed interesting.

Contributor

bric3 commented Oct 13, 2016

On the contrary, it's much easier for contributors of high quality PRs to get their changes published quickly.

That's the thing, while I'm definitely pushing for high-quality PR, all are not hight quality for various reasons :

  • user don't know git well e.g. contributor do X non meaningful commits, history is unreadable and reason of change is obscure or split across various commits
  • Test coverage
  • API design is wrong or in-progress
  • Contributor don't reply

Or PR represents internal reworks without value, to prepare something bigger. Sometime we may find flaws in previous PR. Once an API goes in the wild it's hard to go back.

There is no mathematical formula, I can give an example: If I deprioritize mockito and you don't hear from me for 2 months, @bric3 becomes most senior engineer (level of contributions over time and current engagement) and his opinion weighs most. @TimvdLippe, you're next :)

While I appreciate that my opinions may matter because I've contributing, I think that regardless of level of contributions, any opinion can be just as good as anyone's.

It's mockito mission to pioneer continuous delivery for Java libraries.

No it is not! The mockito's mission is to make a quality library that allows to write good tests with mocks. Pioneering CI/CD allows to accomplish that main mission with ease, but it is not the main goal. Having the delivery upon each commit (or not) isn't pioneering or even innovation, it is a choice that is decided to match how a projects model its release pace.

@jeantil @tinkerware Interesting point of view, browser release model is indeed interesting.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Oct 14, 2016

Member

Thanks guys for feedback!

That's the thing, while I'm definitely pushing for high-quality PR, all are not hight quality for various reasons

@bric3, regardless of the release model, we never want to merge low quality PRs to main dev branch

Mockito mission is to excel in quality of all: impact on quality of tests and code, feature set, extensibility, release model. Continuous delivery model allows most sustainable, effortless release capability which is part of software quality.

Guys, I'm curious about 1) what is your exposure to continuous delivery with production systems / libraries? 2) Did you read continuous delivery book?

To give you my background: every change publishes new version of libraries at LinkedIn since 2011. In 2015 at LinkedIn we built 3x3 model (3 hours max time commit-to-production, 3 releases per day), my friend wrote an article about 3x3 for mobile.

Member

mockitoguy commented Oct 14, 2016

Thanks guys for feedback!

That's the thing, while I'm definitely pushing for high-quality PR, all are not hight quality for various reasons

@bric3, regardless of the release model, we never want to merge low quality PRs to main dev branch

Mockito mission is to excel in quality of all: impact on quality of tests and code, feature set, extensibility, release model. Continuous delivery model allows most sustainable, effortless release capability which is part of software quality.

Guys, I'm curious about 1) what is your exposure to continuous delivery with production systems / libraries? 2) Did you read continuous delivery book?

To give you my background: every change publishes new version of libraries at LinkedIn since 2011. In 2015 at LinkedIn we built 3x3 model (3 hours max time commit-to-production, 3 releases per day), my friend wrote an article about 3x3 for mobile.

@jeantil

This comment has been minimized.

Show comment
Hide comment
@jeantil

jeantil Oct 14, 2016

Fascinating read, but I feel like the model for a library is closer to that of a mobile app than that of a web app

For true continuous integration, we ship every commit directly to production – but this is clearly not realistic in the world of native apps, where binaries must be published in the App Store (with Apple’s week-long review process) and Google Play, then downloaded by members.

It's the same as browsers or IDEs or anything which requires an action on the user part to install.They have an EAP/beta channel where they release up to multiple times a day if they want, and a normal release channel where identified stable builds are promoted in a timely manner.

Continuous delivery works well when you can be sure that if something broken reaches a user, you can push a fix to them quickly. It is an excellent model for : websites where a refresh fixes the app, and more generally self updating software where update doesn't require a restart or where restart time is very slow.

For a java library, as far as I know, the version update process tends to be a manual edit in a file (pom, gradle, sbt,etc) by a developper. That's even less user friendly than an app store update push notification.
Why would you want to make your users do that more often ?

And for the few advanced users using version ranges they would risk random build failures which are always such a pain to investigate...

jeantil commented Oct 14, 2016

Fascinating read, but I feel like the model for a library is closer to that of a mobile app than that of a web app

For true continuous integration, we ship every commit directly to production – but this is clearly not realistic in the world of native apps, where binaries must be published in the App Store (with Apple’s week-long review process) and Google Play, then downloaded by members.

It's the same as browsers or IDEs or anything which requires an action on the user part to install.They have an EAP/beta channel where they release up to multiple times a day if they want, and a normal release channel where identified stable builds are promoted in a timely manner.

Continuous delivery works well when you can be sure that if something broken reaches a user, you can push a fix to them quickly. It is an excellent model for : websites where a refresh fixes the app, and more generally self updating software where update doesn't require a restart or where restart time is very slow.

For a java library, as far as I know, the version update process tends to be a manual edit in a file (pom, gradle, sbt,etc) by a developper. That's even less user friendly than an app store update push notification.
Why would you want to make your users do that more often ?

And for the few advanced users using version ranges they would risk random build failures which are always such a pain to investigate...

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Oct 16, 2016

Member

For a java library, as far as I know, the version update process tends to be a manual edit in a file (pom, gradle, sbt,etc) by a developper. That's even less user friendly than an app store update push notification.
Why would you want to make your users do that more often?

It's entirely up to the consumers how often they want to update. I give them the option to update often, they don't have that option in traditional model.

And for the few advanced users using version ranges they would risk random build failures which are always such a pain to investigate...

Automated ways of updating dependencies is exactly the way forward. For this model to succeed, semantic versioning needs to be adopted, high quality changes, comprehensive test suite that never breaks the customers. This is entirely different discussion - it's about consumers not producers. In traditional release model, people can still use version ranges and get exposed to problems if the producers are not high quality.

Thanks for joining the conversation!

Member

mockitoguy commented Oct 16, 2016

For a java library, as far as I know, the version update process tends to be a manual edit in a file (pom, gradle, sbt,etc) by a developper. That's even less user friendly than an app store update push notification.
Why would you want to make your users do that more often?

It's entirely up to the consumers how often they want to update. I give them the option to update often, they don't have that option in traditional model.

And for the few advanced users using version ranges they would risk random build failures which are always such a pain to investigate...

Automated ways of updating dependencies is exactly the way forward. For this model to succeed, semantic versioning needs to be adopted, high quality changes, comprehensive test suite that never breaks the customers. This is entirely different discussion - it's about consumers not producers. In traditional release model, people can still use version ranges and get exposed to problems if the producers are not high quality.

Thanks for joining the conversation!

@tinkerware

This comment has been minimized.

Show comment
Hide comment
@tinkerware

tinkerware Oct 21, 2016

@philwebb Thanks for the details of your release model. I like how you gave thought to common scenarios and how the model helps each one out.

I think Mockito could do with a simpler model. In particular, I think long-lived maintenance branches are more useful if you have a long shelf-life "platform" release that others can target when building tools and middleware, which makes a lot of sense for Spring Boot. Keeping all development on master including fixes and feature work is simpler; and even if you had to do it otherwise, I think it's safer to keep fixes flowing one-way from master to branches by cherry-picking.

On the other hand, I see that there's a release/2.x branch in addition to master (for Mockito 3?) on this project, and no clear way to choose between the two branches when looking at the active PR's, so I'm not sure about the actual/intended development model; the contributor guide was not much help. Is there some CI magic that verifies and merges approved PR's into both branches at the same time? Both branches seem very active, which is interesting. Maybe it makes more sense to say that 2.x and master branches have their own independent release models and leave it at that.

@philwebb Thanks for the details of your release model. I like how you gave thought to common scenarios and how the model helps each one out.

I think Mockito could do with a simpler model. In particular, I think long-lived maintenance branches are more useful if you have a long shelf-life "platform" release that others can target when building tools and middleware, which makes a lot of sense for Spring Boot. Keeping all development on master including fixes and feature work is simpler; and even if you had to do it otherwise, I think it's safer to keep fixes flowing one-way from master to branches by cherry-picking.

On the other hand, I see that there's a release/2.x branch in addition to master (for Mockito 3?) on this project, and no clear way to choose between the two branches when looking at the active PR's, so I'm not sure about the actual/intended development model; the contributor guide was not much help. Is there some CI magic that verifies and merges approved PR's into both branches at the same time? Both branches seem very active, which is interesting. Maybe it makes more sense to say that 2.x and master branches have their own independent release models and leave it at that.

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Oct 21, 2016

Contributor

@tinkerware master is for 3 and the release branch for 2. We periodically merge release into master but not vice versa

Contributor

TimvdLippe commented Oct 21, 2016

@tinkerware master is for 3 and the release branch for 2. We periodically merge release into master but not vice versa

@bric3

This comment has been minimized.

Show comment
Hide comment
@bric3

bric3 Oct 21, 2016

Contributor

And we introduce buggy release or instable API, this has already happened on mockito.

The solution to this is making sure that every PR is high quality.

A goal but in practice there still things that slips through, just like yesterday.


@philwebb @tinkerware @jeantil

About the release model we elaborated a few months ago, before the one commit=one release model came back to wipe out these plans is described in #483 (comment)

My thinking is :

  1. Release 2.0.0
    1. Branch out to release/2.x
    2. Forget about 2.1.x
    3. Adapt release script to perform and deploy release on this branch
    4. Publish critical fixes only with versions 2.0.x
  2. Upgrade master to 3.0.0-beta-<build number>
    1. Start mockito 3.0.0 progress there (like java 8 only)

For historical purpose : https://groups.google.com/d/msg/mockito-dev/dG2KltOz-gQ/CiNAZkNBcLoJ

EDIT : See #594

So at that point the idea was that we will do only the minimal stuff in mockito 2, only add the remaining items we couldn't include in 2.1.0. So to clarify a bit more with a graph that would have been something like. (tags and numbers are for example purpose)

                o--o--o      <- PR #73 targets master
               /               New awesome API, maybe with hidden bugs
--o-- -o-o-  --o--o--o--o-   <- master (next version eg v3) -> release/deploy 3.0.0-build
  \                    /
   o-- - -o- - --o-- -o-     <- release/2.x ()
    \     ^      ^
     \    |       fixes something on 2.1.0 -> release/deploy 2.1.x
      \    release of 2.1.0
       o--o--o--o            <- PR #132 targets release/2.x
                                New small feature for 2.2 (because unsure for 2.1)

At some point or regularly there would be a merge from release/2.x to master. Current workflow is manual though.
Of course release/2x should not stay active for a long time, so PRs like 132 would be small enough and not many.


I'm pretty sure the model described above and the continuous delivery can fit together. Just removing the beta tag is not a fix for #595 which is somehow what is the one commit=one release model.

Contributor

bric3 commented Oct 21, 2016

And we introduce buggy release or instable API, this has already happened on mockito.

The solution to this is making sure that every PR is high quality.

A goal but in practice there still things that slips through, just like yesterday.


@philwebb @tinkerware @jeantil

About the release model we elaborated a few months ago, before the one commit=one release model came back to wipe out these plans is described in #483 (comment)

My thinking is :

  1. Release 2.0.0
    1. Branch out to release/2.x
    2. Forget about 2.1.x
    3. Adapt release script to perform and deploy release on this branch
    4. Publish critical fixes only with versions 2.0.x
  2. Upgrade master to 3.0.0-beta-<build number>
    1. Start mockito 3.0.0 progress there (like java 8 only)

For historical purpose : https://groups.google.com/d/msg/mockito-dev/dG2KltOz-gQ/CiNAZkNBcLoJ

EDIT : See #594

So at that point the idea was that we will do only the minimal stuff in mockito 2, only add the remaining items we couldn't include in 2.1.0. So to clarify a bit more with a graph that would have been something like. (tags and numbers are for example purpose)

                o--o--o      <- PR #73 targets master
               /               New awesome API, maybe with hidden bugs
--o-- -o-o-  --o--o--o--o-   <- master (next version eg v3) -> release/deploy 3.0.0-build
  \                    /
   o-- - -o- - --o-- -o-     <- release/2.x ()
    \     ^      ^
     \    |       fixes something on 2.1.0 -> release/deploy 2.1.x
      \    release of 2.1.0
       o--o--o--o            <- PR #132 targets release/2.x
                                New small feature for 2.2 (because unsure for 2.1)

At some point or regularly there would be a merge from release/2.x to master. Current workflow is manual though.
Of course release/2x should not stay active for a long time, so PRs like 132 would be small enough and not many.


I'm pretty sure the model described above and the continuous delivery can fit together. Just removing the beta tag is not a fix for #595 which is somehow what is the one commit=one release model.

@bric3

This comment has been minimized.

Show comment
Hide comment
@bric3

bric3 Oct 21, 2016

Contributor

@philwebb @tinkerware @jeantil

Currently the default branch on GH has been changed to be release/2.x instead of master. Indeed @ChristianSchwarz noticed the contributing guide and pr/issue template don't mention the different which branch to target, I will update those.

Contributor

bric3 commented Oct 21, 2016

@philwebb @tinkerware @jeantil

Currently the default branch on GH has been changed to be release/2.x instead of master. Indeed @ChristianSchwarz noticed the contributing guide and pr/issue template don't mention the different which branch to target, I will update those.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Oct 22, 2016

Member

Thank you guys!, the feedback is really amazing!!!

Just to clarify: releasing every commit != releasing every change. Mockito does the latter.

@tinkerware, thank you for sharing the versioneye use case - this is exactly the feedback that we need. Releasing every couple of weeks, plus some extra critical bugfixes, can still create users annoyed by outdated dependencies reported by tools like versioneye. Optimizing Mockito release model for tools like versioneye is does not seem right. I'd rather versioneye supported better libraries that move fast and adopted continuous delivery.

@tinkerware, I agree dependency management is work. That's why Mockito is completely backwards compatible and rigorously follows semantic versioning. This should streamline managing Mockito dependency and if it does not, we would love hear about the use cases. Also, managing traditional releases is more work for us == less time for working on features.

@philwebb, thank you for taking the time and describe the spring boot release model! I think Mockito has sufficiently different profile than spring boot that we can have simpler release model (e.g. I agree with @tinkerware's reply).

@philwebb, blessed version of Mockito is always the latest. We ensure 100% backwards compatibility, always. I understand that for spring boot it would be preferable to integrate with library that moves at similar pace.

@jeantil, your idea about automated, cadence releases is super interesting. I want to explore this idea further: #708

@szpak, I really like the techniques and tools that help engineers be up-to-date with their dependencies automatically. Thanks for the pointers!

@bric3, thank you for clearly describing your desired release model! Also, well spotted bug in the test code I submitted yesterday!

@TimvdLippe, thanks for upvotes and looking forward to your code reviews!

All, I wrote a wiki page combining my motivation and the feedback gathered in this ticket: Continuous Delivery Overview.

Next steps:

  • I'd like us maintain continuous delivery model this quarter, revisit it early next year as we keep collecting great feedback
  • scope @jeantil's idea #708

THANK YOU for feedback. This is extremely valuable discussion.

Member

mockitoguy commented Oct 22, 2016

Thank you guys!, the feedback is really amazing!!!

Just to clarify: releasing every commit != releasing every change. Mockito does the latter.

@tinkerware, thank you for sharing the versioneye use case - this is exactly the feedback that we need. Releasing every couple of weeks, plus some extra critical bugfixes, can still create users annoyed by outdated dependencies reported by tools like versioneye. Optimizing Mockito release model for tools like versioneye is does not seem right. I'd rather versioneye supported better libraries that move fast and adopted continuous delivery.

@tinkerware, I agree dependency management is work. That's why Mockito is completely backwards compatible and rigorously follows semantic versioning. This should streamline managing Mockito dependency and if it does not, we would love hear about the use cases. Also, managing traditional releases is more work for us == less time for working on features.

@philwebb, thank you for taking the time and describe the spring boot release model! I think Mockito has sufficiently different profile than spring boot that we can have simpler release model (e.g. I agree with @tinkerware's reply).

@philwebb, blessed version of Mockito is always the latest. We ensure 100% backwards compatibility, always. I understand that for spring boot it would be preferable to integrate with library that moves at similar pace.

@jeantil, your idea about automated, cadence releases is super interesting. I want to explore this idea further: #708

@szpak, I really like the techniques and tools that help engineers be up-to-date with their dependencies automatically. Thanks for the pointers!

@bric3, thank you for clearly describing your desired release model! Also, well spotted bug in the test code I submitted yesterday!

@TimvdLippe, thanks for upvotes and looking forward to your code reviews!

All, I wrote a wiki page combining my motivation and the feedback gathered in this ticket: Continuous Delivery Overview.

Next steps:

  • I'd like us maintain continuous delivery model this quarter, revisit it early next year as we keep collecting great feedback
  • scope @jeantil's idea #708

THANK YOU for feedback. This is extremely valuable discussion.

@bric3

This comment has been minimized.

Show comment
Hide comment
@bric3

bric3 Oct 25, 2016

Contributor

Just to clarify: releasing every commit != releasing every change. Mockito does the latter.

It's a play of word, a commit represents changes and it applies to normal commits as well as on merge commits.
Let's just rephrase the controversial release model here to every reflog change.

We ensure 100% backwards compatibility, always.

Errors happened before in this every change model. This is something we cannot guarantee this.

Contributor

bric3 commented Oct 25, 2016

Just to clarify: releasing every commit != releasing every change. Mockito does the latter.

It's a play of word, a commit represents changes and it applies to normal commits as well as on merge commits.
Let's just rephrase the controversial release model here to every reflog change.

We ensure 100% backwards compatibility, always.

Errors happened before in this every change model. This is something we cannot guarantee this.

@bric3

This comment has been minimized.

Show comment
Hide comment
@bric3

bric3 Nov 3, 2016

Contributor

Just to spill oil on the fire : Here's what can happen on every change : https://twitter.com/yanncebron/status/794167202303930368

linkedin-null-tweet

With this model I am not enticed to do any important change.

Contributor

bric3 commented Nov 3, 2016

Just to spill oil on the fire : Here's what can happen on every change : https://twitter.com/yanncebron/status/794167202303930368

linkedin-null-tweet

With this model I am not enticed to do any important change.

@bric3

This comment has been minimized.

Show comment
Hide comment
@bric3

bric3 Nov 10, 2016

Contributor

In my opinion like the opinion of others, we want to avoid is this case : 4 release in 2 days

2.2.15 (2016-11-08 20:43 UTC)

Authors: 2
Commits: 4
3: Brice Dutheil
1: Szczepan Faber
Improvements: 1
Enhancements: 1
Allow fluent usage of Mockito.framework() listeners methods (#747)
2.2.14 (2016-11-08 20:11 UTC)

Authors: 1
Commits: 1
1: Dmitriy Zaitsev
Improvements: 1
Add missing copyright headers (#746)
2.2.13 (2016-11-07 22:59 UTC)

Authors: 1
Commits: 1
1: Brice Dutheil
Improvements: 1
Removes deprecated way of documenting package by package-info.java (#745)
2.2.12 (2016-11-07 04:29 UTC)

Authors: 1
Commits: 11
11: Szczepan Faber
Improvements: 1
Enhancements: 1
Exposed existing verification API so that it no longer leaks internal API (#735)

While we could have slightly bigger releases. I'm not sure there's great value for developers to release Copyright headers or Improved exception message.

Contributor

bric3 commented Nov 10, 2016

In my opinion like the opinion of others, we want to avoid is this case : 4 release in 2 days

2.2.15 (2016-11-08 20:43 UTC)

Authors: 2
Commits: 4
3: Brice Dutheil
1: Szczepan Faber
Improvements: 1
Enhancements: 1
Allow fluent usage of Mockito.framework() listeners methods (#747)
2.2.14 (2016-11-08 20:11 UTC)

Authors: 1
Commits: 1
1: Dmitriy Zaitsev
Improvements: 1
Add missing copyright headers (#746)
2.2.13 (2016-11-07 22:59 UTC)

Authors: 1
Commits: 1
1: Brice Dutheil
Improvements: 1
Removes deprecated way of documenting package by package-info.java (#745)
2.2.12 (2016-11-07 04:29 UTC)

Authors: 1
Commits: 11
11: Szczepan Faber
Improvements: 1
Enhancements: 1
Exposed existing verification API so that it no longer leaks internal API (#735)

While we could have slightly bigger releases. I'm not sure there's great value for developers to release Copyright headers or Improved exception message.

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Nov 10, 2016

Contributor

The past few days I have received a lot of questions regarding the overload of Mockito releases. After I explain the reasoning for this model, there has not been any developer that agrees with this model 😢 Developer response is therefore unanimously negative according to my knowledge.

Contributor

TimvdLippe commented Nov 10, 2016

The past few days I have received a lot of questions regarding the overload of Mockito releases. After I explain the reasoning for this model, there has not been any developer that agrees with this model 😢 Developer response is therefore unanimously negative according to my knowledge.

@szpak

This comment has been minimized.

Show comment
Hide comment
@szpak

szpak Nov 10, 2016

Member

@TimvdLippe Are you referring to Mockito 2.0-beta (too many releases) or CD in FOSS projects in general?

Member

szpak commented Nov 10, 2016

@TimvdLippe Are you referring to Mockito 2.0-beta (too many releases) or CD in FOSS projects in general?

@szpak

This comment has been minimized.

Show comment
Hide comment
@szpak

szpak Nov 10, 2016

Member

While we could have slightly bigger releases. I'm not sure there's great value for developers to release Copyright headers or Improved exception message.

@bric3 Probably most of us agree on that. New features and bug fixes should be considered as a candidate for the new releases. In other cases a person making a merge should add a tag a merge message to skip the release process. If it would be too hard to follow I personally prefer the opposite way - when a person merging a PR considers it as something important to be release a "do release" tag is added to a merge message and only then a new version is released (automatically, but on demand).

Member

szpak commented Nov 10, 2016

While we could have slightly bigger releases. I'm not sure there's great value for developers to release Copyright headers or Improved exception message.

@bric3 Probably most of us agree on that. New features and bug fixes should be considered as a candidate for the new releases. In other cases a person making a merge should add a tag a merge message to skip the release process. If it would be too hard to follow I personally prefer the opposite way - when a person merging a PR considers it as something important to be release a "do release" tag is added to a merge message and only then a new version is released (automatically, but on demand).

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Nov 10, 2016

Contributor

The amount of production releases (2.X.X) currently published. E.g. we are already at 2.2.14.

Contributor

TimvdLippe commented Nov 10, 2016

The amount of production releases (2.X.X) currently published. E.g. we are already at 2.2.14.

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Nov 10, 2016

Contributor

@szpak actually the manual release was implemented with including [ci-release 2.3.0] in the commit message. This has been reverted later.

Contributor

TimvdLippe commented Nov 10, 2016

@szpak actually the manual release was implemented with including [ci-release 2.3.0] in the commit message. This has been reverted later.

@szpak

This comment has been minimized.

Show comment
Hide comment
@szpak

szpak Nov 10, 2016

Member

I thought that [ci-release 2.3.0] was only to override the next version (and by default a release with automatic increment had been performed automatically for every commit). However, I didn't analyze that code, so you may be right @TimvdLippe .

Member

szpak commented Nov 10, 2016

I thought that [ci-release 2.3.0] was only to override the next version (and by default a release with automatic increment had been performed automatically for every commit). However, I didn't analyze that code, so you may be right @TimvdLippe .

@dnozay

This comment has been minimized.

Show comment
Hide comment
@dnozay

dnozay Nov 28, 2016

I strongly dislike "one commit = one release":

  • Changelogs are now meaningless, it's more of a challenge (actually: more tedious) if you need to figure out what changed between two versions.
  • Making a release for doc / comment fixes is plain retarded; while users looking at the docs will appreciate, product managers and tech leads that actually track dependencies will hate you.
  • There are product managers and tech leads, whose responsibilities involve actually tracking dependencies and making sure the technological debt is low; they will hate you; mockito is mostly a test-level dependency, so it may be a low-priority item on their radar. But imagine a scenario if there were a bunch of iOS developers working on a product, and they had to deal with Xcode releases for every commit.
  • Changing a dependency may mean that you have to trigger many different QA cycles; not everything everywhere is automated. This is an expensive activity, and is a driver for not keeping up-to-date often.

I totally get the idea that releasing often is great to push new fixes / features out, but there is version fatigue. The opposite of this "one commit = one release" are LTS versions. People like them. Businesses like those; I know a few that will only update from LTS to LTS.

dnozay commented Nov 28, 2016

I strongly dislike "one commit = one release":

  • Changelogs are now meaningless, it's more of a challenge (actually: more tedious) if you need to figure out what changed between two versions.
  • Making a release for doc / comment fixes is plain retarded; while users looking at the docs will appreciate, product managers and tech leads that actually track dependencies will hate you.
  • There are product managers and tech leads, whose responsibilities involve actually tracking dependencies and making sure the technological debt is low; they will hate you; mockito is mostly a test-level dependency, so it may be a low-priority item on their radar. But imagine a scenario if there were a bunch of iOS developers working on a product, and they had to deal with Xcode releases for every commit.
  • Changing a dependency may mean that you have to trigger many different QA cycles; not everything everywhere is automated. This is an expensive activity, and is a driver for not keeping up-to-date often.

I totally get the idea that releasing often is great to push new fixes / features out, but there is version fatigue. The opposite of this "one commit = one release" are LTS versions. People like them. Businesses like those; I know a few that will only update from LTS to LTS.

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Nov 29, 2016

Contributor

@dnozay Thanks a lot for voicing your opinion! I agree with you and I think this situation is actually more harmful than the benefit of continuous delivery.

Thus far we have been unable to come to an agreement or concession regarding the current practice of continuous delivery 😢 Hopefully we can come up with a solution that prevents your situation from happening.

Contributor

TimvdLippe commented Nov 29, 2016

@dnozay Thanks a lot for voicing your opinion! I agree with you and I think this situation is actually more harmful than the benefit of continuous delivery.

Thus far we have been unable to come to an agreement or concession regarding the current practice of continuous delivery 😢 Hopefully we can come up with a solution that prevents your situation from happening.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Dec 2, 2016

Member

@dnozay, thanks for feedback! I fully share your passion and enthusiasm to release automation, while having different opinion on the way :)

What version of Mockito do you use? How often do you upgrade? How is version fatigue impacting you?

Member

mockitoguy commented Dec 2, 2016

@dnozay, thanks for feedback! I fully share your passion and enthusiasm to release automation, while having different opinion on the way :)

What version of Mockito do you use? How often do you upgrade? How is version fatigue impacting you?

@dnozay

This comment has been minimized.

Show comment
Hide comment
@dnozay

dnozay Dec 7, 2016

Yes, I love release automation.

What version of Mockito do you use? How often do you upgrade? How is version fatigue impacting you?

I don't use Mockito directly. I got to this page based on some email thread.

But let me share some of my experience doing release management. Maybe you know about this thing called Jenkins that helps people do continuous integration (here is their Governance document). They are very successful at releasing often, while being also very good for "scaredy-cat" customers: they do provide LTS versions.

How much do I do with it? Not much, but I used to maintain a few Jenkins plugins (e.g. build-flow-plugin), and the release model follows much of the Jenkins documentation for plugins (see also the Extend Jenkins page); it uses the Maven Release plugin. We can summarize that to:

  • select a bunch of changes, "Cut the changelog",
  • perform all kinds of testing on it, regression, backwards-compat, integration, end-to-end, etc.
  • do a mvn release:prepare release:perform -Dusername=... -Dpassword=...

How often do you upgrade?

The main factors are

  • Time and money : upgrading is not free.
  • Urgency : is this a security related release?
  • Is this a minor version / is this backwards-compatible?
  • Do I need a full qualification round, or can I save time (or $) doing smoke tests?
  • Am I myself in a release cycle, doing my own testing? Is that going to invalidate prior results?

I typically will NOT upgrade unless security fixes or I intend to pull new features, or just came back from vacation and need to catch up or have extra time to spend on conflict resolution.

Upgrading costs time and money.
If you think about your dependencies, it is a graph of dependencies, not a tree. You can have conflicting packages. I mean there is a reason behind excluding transitive dependencies in gradle. Sometimes different implementations of the same classes end up on the classpath. Sometimes the order matters and you spent time down the rabbit-hole trying to figure out classpath issues.

Have you worked for a company that releases an OS? Have you seen build-to-build upgrade testing? - I have. When you make the assumption that there is some common base / thing that should work (e.g. your OS is the same), you may think testing is fine. Now imagine that you cannot trust the OS; that you need to re-install it - just because that's the thing you are testing.

Too many releases is just noise
Your consumers probably won't care if you are releasing more than once a day.
If it takes 2 days to test; they can't physically handle upgrading more than every 2 days unless they have multiple teams / staggered release cycles.

How is version fatigue impacting you?

I only have so much time I can allocate.

  • Updating dependencies takes time (compile time, test time, ...) which either makes me busy, or ties resources that I can't use for other activities; e.g. I can't disconnect my laptop from the network and hop on a commuter bus.
  • The problem scales non-linearly with more dependencies.
  • Sometimes I have to resolve conflicts due to API changes, that's extra.

I just want to code my own stuff, not waste time on "upgrade" activities, so on a personal level, I hate it with a passion words cannot fully describe. Things are usually fine when the upgrade is smooth and silent; but that's not the common case for me.

Why are things not smooth?
That's a question that was missing. Because the bar for quality for a library/tool is low in general (I am not talking about Mockito here, so don't get mad at me).

  • lack of unit tests / poor coverage,
  • lack of integration tests,
  • shifting responsibility of testing to the consumer,
  • unstable APIs; lack of semantic versioning,
  • not understanding your consumers' use cases.

Some cool things to look at:

dnozay commented Dec 7, 2016

Yes, I love release automation.

What version of Mockito do you use? How often do you upgrade? How is version fatigue impacting you?

I don't use Mockito directly. I got to this page based on some email thread.

But let me share some of my experience doing release management. Maybe you know about this thing called Jenkins that helps people do continuous integration (here is their Governance document). They are very successful at releasing often, while being also very good for "scaredy-cat" customers: they do provide LTS versions.

How much do I do with it? Not much, but I used to maintain a few Jenkins plugins (e.g. build-flow-plugin), and the release model follows much of the Jenkins documentation for plugins (see also the Extend Jenkins page); it uses the Maven Release plugin. We can summarize that to:

  • select a bunch of changes, "Cut the changelog",
  • perform all kinds of testing on it, regression, backwards-compat, integration, end-to-end, etc.
  • do a mvn release:prepare release:perform -Dusername=... -Dpassword=...

How often do you upgrade?

The main factors are

  • Time and money : upgrading is not free.
  • Urgency : is this a security related release?
  • Is this a minor version / is this backwards-compatible?
  • Do I need a full qualification round, or can I save time (or $) doing smoke tests?
  • Am I myself in a release cycle, doing my own testing? Is that going to invalidate prior results?

I typically will NOT upgrade unless security fixes or I intend to pull new features, or just came back from vacation and need to catch up or have extra time to spend on conflict resolution.

Upgrading costs time and money.
If you think about your dependencies, it is a graph of dependencies, not a tree. You can have conflicting packages. I mean there is a reason behind excluding transitive dependencies in gradle. Sometimes different implementations of the same classes end up on the classpath. Sometimes the order matters and you spent time down the rabbit-hole trying to figure out classpath issues.

Have you worked for a company that releases an OS? Have you seen build-to-build upgrade testing? - I have. When you make the assumption that there is some common base / thing that should work (e.g. your OS is the same), you may think testing is fine. Now imagine that you cannot trust the OS; that you need to re-install it - just because that's the thing you are testing.

Too many releases is just noise
Your consumers probably won't care if you are releasing more than once a day.
If it takes 2 days to test; they can't physically handle upgrading more than every 2 days unless they have multiple teams / staggered release cycles.

How is version fatigue impacting you?

I only have so much time I can allocate.

  • Updating dependencies takes time (compile time, test time, ...) which either makes me busy, or ties resources that I can't use for other activities; e.g. I can't disconnect my laptop from the network and hop on a commuter bus.
  • The problem scales non-linearly with more dependencies.
  • Sometimes I have to resolve conflicts due to API changes, that's extra.

I just want to code my own stuff, not waste time on "upgrade" activities, so on a personal level, I hate it with a passion words cannot fully describe. Things are usually fine when the upgrade is smooth and silent; but that's not the common case for me.

Why are things not smooth?
That's a question that was missing. Because the bar for quality for a library/tool is low in general (I am not talking about Mockito here, so don't get mad at me).

  • lack of unit tests / poor coverage,
  • lack of integration tests,
  • shifting responsibility of testing to the consumer,
  • unstable APIs; lack of semantic versioning,
  • not understanding your consumers' use cases.

Some cool things to look at:

@mattnelson

This comment has been minimized.

Show comment
Hide comment
@mattnelson

mattnelson Dec 19, 2016

That's why Mockito is completely backwards compatible and rigorously follows semantic versioning. This should streamline managing Mockito dependency and if it does not, we would love hear about the use cases. Also, managing traditional releases is more work for us == less time for working on features.

Coming in a little late to this discussion as we just uplifted to Mockito 2 last week and found some quirks with the continuous aspects of this project in the first week. We have a CI job that runs every week and creates a PR to auto update all of our dependencies. When the script ran this week we got prompted to update to 2.3.7[1]. When we attempted to validate the build, it failed with a transient mockito error. We kicked off the build again and it went away. But in parallel to the build running, we attempted to look at the changes for 2.3.7 to see what changed and if we needed to make code changes, the tag/release was no where to be found. 2.3.8 was missing as well. In that time 2.3.9 and 2.3.10 were tagged, but are not available in maven central at the time of this posting.

That left us with the following options:

  • Lock the dependency to our current known working version (2.2.29)
  • Work back from 2.3.6 until we find a working version
  • Wait for 2.3.9+ to be available on maven central and see if the build passes.

I don't want to discourage the release automation process especially if it makes it easier for maintainers and encourages contributors as they will be able to use their changes quicker.
But I agree with some of the other sentiments in this thread that less frequent and more vetted releases would be preferable.

Transient error that occurred with 2.3.7. Looks related to #638, but we are not using the internal APIs.

java.lang.NoClassDefFoundError: Could not initialize class org.mockito.internal.util.MockUtil

[1] http://search.maven.org/#artifactdetails%7Corg.mockito%7Cmockito-core%7C2.3.7%7Cjar
[2] https://github.com/mockito/mockito/tags

That's why Mockito is completely backwards compatible and rigorously follows semantic versioning. This should streamline managing Mockito dependency and if it does not, we would love hear about the use cases. Also, managing traditional releases is more work for us == less time for working on features.

Coming in a little late to this discussion as we just uplifted to Mockito 2 last week and found some quirks with the continuous aspects of this project in the first week. We have a CI job that runs every week and creates a PR to auto update all of our dependencies. When the script ran this week we got prompted to update to 2.3.7[1]. When we attempted to validate the build, it failed with a transient mockito error. We kicked off the build again and it went away. But in parallel to the build running, we attempted to look at the changes for 2.3.7 to see what changed and if we needed to make code changes, the tag/release was no where to be found. 2.3.8 was missing as well. In that time 2.3.9 and 2.3.10 were tagged, but are not available in maven central at the time of this posting.

That left us with the following options:

  • Lock the dependency to our current known working version (2.2.29)
  • Work back from 2.3.6 until we find a working version
  • Wait for 2.3.9+ to be available on maven central and see if the build passes.

I don't want to discourage the release automation process especially if it makes it easier for maintainers and encourages contributors as they will be able to use their changes quicker.
But I agree with some of the other sentiments in this thread that less frequent and more vetted releases would be preferable.

Transient error that occurred with 2.3.7. Looks related to #638, but we are not using the internal APIs.

java.lang.NoClassDefFoundError: Could not initialize class org.mockito.internal.util.MockUtil

[1] http://search.maven.org/#artifactdetails%7Corg.mockito%7Cmockito-core%7C2.3.7%7Cjar
[2] https://github.com/mockito/mockito/tags

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Dec 19, 2016

Contributor

@mattnelson Sorry for the trouble that we caused you! There was a problem with releasing 2.3.7 as we are preparing the release procedure for the new mockito-android. The plugin we use to push artifacts to Bintray had a bug which was uploading corrupt jars to maven central. We identified the error quickly and retracted the corrupt versions, which was probably what you encountered too.

The release procedure should be fixed once again, but I think we can see this is a learning point that if we update the release procedure we must first test on our Bintray test project. Once we can install correctly ourselves, we push out to maven central. @szczepiq WDYT?

Contributor

TimvdLippe commented Dec 19, 2016

@mattnelson Sorry for the trouble that we caused you! There was a problem with releasing 2.3.7 as we are preparing the release procedure for the new mockito-android. The plugin we use to push artifacts to Bintray had a bug which was uploading corrupt jars to maven central. We identified the error quickly and retracted the corrupt versions, which was probably what you encountered too.

The release procedure should be fixed once again, but I think we can see this is a learning point that if we update the release procedure we must first test on our Bintray test project. Once we can install correctly ourselves, we push out to maven central. @szczepiq WDYT?

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Dec 20, 2016

Member

@mattnelson, very sorry for problems you encountered. We had some hiccups recently as described by @TimvdLippe. Do you happen to have the stack trace of the transient issue? Can you gist it for us?

@TimvdLippe, let's chat on slack channel. Can you formulate the release model that you have in mind when you say "must first test on our Bintray test project".

@mattnelson - thanks so much for taking the time and reporting the issue back to us. This feedback helps us a lot make decisions on how to release safely.

Member

mockitoguy commented Dec 20, 2016

@mattnelson, very sorry for problems you encountered. We had some hiccups recently as described by @TimvdLippe. Do you happen to have the stack trace of the transient issue? Can you gist it for us?

@TimvdLippe, let's chat on slack channel. Can you formulate the release model that you have in mind when you say "must first test on our Bintray test project".

@mattnelson - thanks so much for taking the time and reporting the issue back to us. This feedback helps us a lot make decisions on how to release safely.

@drewhannay

This comment has been minimized.

Show comment
Hide comment
@drewhannay

drewhannay Jan 3, 2017

I agree with pretty much everything @dnozay said. Upgrading Mockito is now very painful. There's no good release notes or upgrade instructions, everything is spread out across many many versions. I don't have the time or desire to look at every commit individually to find out what changed.

I understand that following semantic versioning means that we should be able to "trust" and drop in any newer compatible version. Now I may trust you all to stick to that, but my team of 50+ developers certainly does not. They want to see a list of what has changed and what benefits we get from upgrading that make it worth the risk. The current release cycle means we're much less likely to upgrade until there's a meaningful reason to do so. But with so many versions, how will I ever notice that there's a good reason to upgrade?!?

Right now my method is to follow @szczepiq on LinkedIn and Twitter to find out when there's a cool new feature and then use my own discretion to decide if it's worth spending half a day of my time to upgrade and validate a new version of Mockito. This seems....less than optimal 😉

I agree with pretty much everything @dnozay said. Upgrading Mockito is now very painful. There's no good release notes or upgrade instructions, everything is spread out across many many versions. I don't have the time or desire to look at every commit individually to find out what changed.

I understand that following semantic versioning means that we should be able to "trust" and drop in any newer compatible version. Now I may trust you all to stick to that, but my team of 50+ developers certainly does not. They want to see a list of what has changed and what benefits we get from upgrading that make it worth the risk. The current release cycle means we're much less likely to upgrade until there's a meaningful reason to do so. But with so many versions, how will I ever notice that there's a good reason to upgrade?!?

Right now my method is to follow @szczepiq on LinkedIn and Twitter to find out when there's a cool new feature and then use my own discretion to decide if it's worth spending half a day of my time to upgrade and validate a new version of Mockito. This seems....less than optimal 😉

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Jan 5, 2017

Member

@drewhannay, fantastic feedback. Thank you for writing this up. You raised valid points and we will address them.

half a day of my time to upgrade and validate a new version of Mockito

Is updating minor version of Mockito half day of your work? Why?

Member

mockitoguy commented Jan 5, 2017

@drewhannay, fantastic feedback. Thank you for writing this up. You raised valid points and we will address them.

half a day of my time to upgrade and validate a new version of Mockito

Is updating minor version of Mockito half day of your work? Why?

@drewhannay

This comment has been minimized.

Show comment
Hide comment
@drewhannay

drewhannay Jan 6, 2017

Mainly because Android tests are slow and we have a lot of them 😉

In the best case, I end up doing at least two full test runs (~45 minutes each), plus time getting a code review and time for importing the new version of Mockito into our internal artifact server.

Mainly because Android tests are slow and we have a lot of them 😉

In the best case, I end up doing at least two full test runs (~45 minutes each), plus time getting a code review and time for importing the new version of Mockito into our internal artifact server.

@mockitoguy mockitoguy referenced this issue Jan 29, 2017

Closed

Mockito Continuous Delivery Pipeline 2.0 #911

4 of 9 tasks complete
@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Jan 31, 2017

Member

I documented draft plan for new release cadence at issue #911 (interesting number, huh? coincidence? :). I would love to get feedback!

Member

mockitoguy commented Jan 31, 2017

I documented draft plan for new release cadence at issue #911 (interesting number, huh? coincidence? :). I would love to get feedback!

@tinkerware

This comment has been minimized.

Show comment
Hide comment
@tinkerware

tinkerware Feb 8, 2017

As an anecdote, I want to talk about what I had to do just today when updating dependencies on a project using both Mockito and ByteBuddy.

I had Mockito on a recent enough 2.5.4 and a separate internal library had ByteBuddy on 1.6.0. When I upgrade Mockito to its latest as of today, I end up with Mockito on 2.7.5. ByteBuddy dependency from Mockito is at 1.6.5, however. Additionally, ByteBuddy 1.6.5 breaks public API with 1.6.0 by renaming at least one interface and one method (lest you think this is unusual, ByteBuddy has always broken public API in its patch versions; arguably, that's the root cause). Now suddenly, I have two compile-incompatible versions of the same library when I upgrade from Mockito 2.5.4 to 2.7.5.

I'm not bringing this particular example up to assign blame (I love ByteBuddy to pieces, even with its painful unpromise of public API compatibility; I have accepted that it's a high-maintenance library to keep up with) or to suggest changes to Mockito (though, in retrospect, Mockito can shade ByteBuddy as a private dependency to avoid passing down the pain of keeping up with it). I only wanted to point out that maintaining dependencies can be work, even hard work. Predictable, compatible releases are valuable because they make this work easier.

tinkerware commented Feb 8, 2017

As an anecdote, I want to talk about what I had to do just today when updating dependencies on a project using both Mockito and ByteBuddy.

I had Mockito on a recent enough 2.5.4 and a separate internal library had ByteBuddy on 1.6.0. When I upgrade Mockito to its latest as of today, I end up with Mockito on 2.7.5. ByteBuddy dependency from Mockito is at 1.6.5, however. Additionally, ByteBuddy 1.6.5 breaks public API with 1.6.0 by renaming at least one interface and one method (lest you think this is unusual, ByteBuddy has always broken public API in its patch versions; arguably, that's the root cause). Now suddenly, I have two compile-incompatible versions of the same library when I upgrade from Mockito 2.5.4 to 2.7.5.

I'm not bringing this particular example up to assign blame (I love ByteBuddy to pieces, even with its painful unpromise of public API compatibility; I have accepted that it's a high-maintenance library to keep up with) or to suggest changes to Mockito (though, in retrospect, Mockito can shade ByteBuddy as a private dependency to avoid passing down the pain of keeping up with it). I only wanted to point out that maintaining dependencies can be work, even hard work. Predictable, compatible releases are valuable because they make this work easier.

@mockitoguy

This comment has been minimized.

Show comment
Hide comment
@mockitoguy

mockitoguy Feb 12, 2017

Member

@tinkerware, this is extremely useful feedback. Thank you very much for reporting. I will start an internal discussion about the idea of shading the bytebuddy jar.

In the early days of Mockito, we used to ship with direct dependency on cglib. However, it was causing similar problems to our users and we started shading cglib. Bytebuddy is getting popular (rightfully so!) and will be used more more often. I want to make sure that mockito is always very convenient to use, and avoids potential incompatibilities that cause users to troubleshoot classpaths / dependency graphs.

Member

mockitoguy commented Feb 12, 2017

@tinkerware, this is extremely useful feedback. Thank you very much for reporting. I will start an internal discussion about the idea of shading the bytebuddy jar.

In the early days of Mockito, we used to ship with direct dependency on cglib. However, it was causing similar problems to our users and we started shading cglib. Bytebuddy is getting popular (rightfully so!) and will be used more more often. I want to make sure that mockito is always very convenient to use, and avoids potential incompatibilities that cause users to troubleshoot classpaths / dependency graphs.

@raphw

This comment has been minimized.

Show comment
Hide comment
@raphw

raphw Feb 12, 2017

Member

@tinkerware Out of curiosity. What API of Byte Buddy do you rely on that broke since then? Typically, Byte Buddy changes its public API because it has to what happens with features that were recently added and had to prove their use in practice. This is a bit of a catch 22 where I consider anything new to be incubating without naming a maturity date or version as I simply do not know.

If there is any particuar API that broke for you, I would be curious to know!

Member

raphw commented Feb 12, 2017

@tinkerware Out of curiosity. What API of Byte Buddy do you rely on that broke since then? Typically, Byte Buddy changes its public API because it has to what happens with features that were recently added and had to prove their use in practice. This is a bit of a catch 22 where I consider anything new to be incubating without naming a maturity date or version as I simply do not know.

If there is any particuar API that broke for you, I would be curious to know!

@tinkerware

This comment has been minimized.

Show comment
Hide comment
@tinkerware

tinkerware Feb 13, 2017

Out of curiosity. What API of Byte Buddy do you rely on that broke since then?

I had code that gathered annotations from multiple "levels" (e.g. declaring type, enclosing types, package) so I used AnnotatedCodeElement as the common type across functions that traversed declarations. That got renamed to AnnotationSource. Another one is the isGenericDeclaration method, renamed to isGenerified; that was part of a matcher predicate (to be clear, I like the new names better).

Typically, Byte Buddy changes its public API because it has to what happens with features that were recently added and had to prove their use in practice. This is a bit of a catch 22 where I consider anything new to be incubating without naming a maturity date or version as I simply do not know.

I genuinely sympathize with this, and I think shoehorning multiple API's, each in a different point in their lifecycle, into one artifact version number is eventually self-defeating. The late Pieter Hintjens had a good handle on defining this problem and one possible way forward, written up in End of Software Versions.

In general, I find the current state of tooling to manage API lifecycle miserable, but there is some low-hanging fruit. I have two suggestions: (1) You can mark new API that has to prove itself with a draft/beta API annotation, and (2) deprecate all API before removal. Deprecated API does not have to hang on for ages; I'm okay if beta API disappears in a minor version after a short period of deprecation. Only stable API would be relevant for semantic versioning.

I don't want to hijack this thread with non-Mockito discussions, so feel free to point me to another place to continue if you think it's worth it.

Thanks for ByteBuddy and Mockito!

Out of curiosity. What API of Byte Buddy do you rely on that broke since then?

I had code that gathered annotations from multiple "levels" (e.g. declaring type, enclosing types, package) so I used AnnotatedCodeElement as the common type across functions that traversed declarations. That got renamed to AnnotationSource. Another one is the isGenericDeclaration method, renamed to isGenerified; that was part of a matcher predicate (to be clear, I like the new names better).

Typically, Byte Buddy changes its public API because it has to what happens with features that were recently added and had to prove their use in practice. This is a bit of a catch 22 where I consider anything new to be incubating without naming a maturity date or version as I simply do not know.

I genuinely sympathize with this, and I think shoehorning multiple API's, each in a different point in their lifecycle, into one artifact version number is eventually self-defeating. The late Pieter Hintjens had a good handle on defining this problem and one possible way forward, written up in End of Software Versions.

In general, I find the current state of tooling to manage API lifecycle miserable, but there is some low-hanging fruit. I have two suggestions: (1) You can mark new API that has to prove itself with a draft/beta API annotation, and (2) deprecate all API before removal. Deprecated API does not have to hang on for ages; I'm okay if beta API disappears in a minor version after a short period of deprecation. Only stable API would be relevant for semantic versioning.

I don't want to hijack this thread with non-Mockito discussions, so feel free to point me to another place to continue if you think it's worth it.

Thanks for ByteBuddy and Mockito!

@jeantil jeantil referenced this issue Feb 14, 2017

Closed

Automate releases #721

3 of 5 tasks complete
@raphw

This comment has been minimized.

Show comment
Hide comment
@raphw

raphw Feb 14, 2017

Member

Thanks for your feedback. True, I had to change this with respect to making type annotation resolution lazy which is was not before. Its rather the exception than the rule but in this case, it was a customer-driven innovation which is partly a requirement for keeping the library alive. I agree that I should have not done this in a minor update but the incompatibility was difficult to avoid.

Member

raphw commented Feb 14, 2017

Thanks for your feedback. True, I had to change this with respect to making type annotation resolution lazy which is was not before. Its rather the exception than the rule but in this case, it was a customer-driven innovation which is partly a requirement for keeping the library alive. I agree that I should have not done this in a minor update but the incompatibility was difficult to avoid.

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Apr 11, 2017

Contributor

Would like to give everyone a heads up who is subscribed to this issue: @szczepiq did some great work to implement the new workflow. You can read a detailed description in the (WIP) wiki at https://github.com/mockito/mockito/wiki/Continuous-Delivery-Details#cd

We are planning to ship this new implementation soon, which means you will receive fewer Mockito updates. If you want to use the latest version all of the time, follow the instructions in the wiki to get listed for all updates 😄

Contributor

TimvdLippe commented Apr 11, 2017

Would like to give everyone a heads up who is subscribed to this issue: @szczepiq did some great work to implement the new workflow. You can read a detailed description in the (WIP) wiki at https://github.com/mockito/mockito/wiki/Continuous-Delivery-Details#cd

We are planning to ship this new implementation soon, which means you will receive fewer Mockito updates. If you want to use the latest version all of the time, follow the instructions in the wiki to get listed for all updates 😄

@TimvdLippe

This comment has been minimized.

Show comment
Hide comment
@TimvdLippe

TimvdLippe Nov 4, 2017

Contributor

Over the past half year, we have been using Shipkit to enable our continuous delivery story. We published our rationale in https://github.com/mockito/mockito/wiki/Continuous-Delivery-Details and described how we tackled this issue. All versions are still automatically published to Bintray to enable users to opt-in frequent updates. Other users can rely on the Maven Central release, which is published roughly every month (this is not strict, but based on numbers on https://mvnrepository.com/artifact/org.mockito/mockito-core).

All in all, we have not received any more feedback regarding too frequent published versions. We are also happy that we can still receive quick feedback and bisect based on Bintray binaries to find out which version (and thus PR) introduced the regression.

I would like to thank everyone that participated in this discussion and worked with us to tackle this issue. If you have any more specific feedback regarding our release mechanism, please file your feedback at https://github.com/mockito/shipkit

Contributor

TimvdLippe commented Nov 4, 2017

Over the past half year, we have been using Shipkit to enable our continuous delivery story. We published our rationale in https://github.com/mockito/mockito/wiki/Continuous-Delivery-Details and described how we tackled this issue. All versions are still automatically published to Bintray to enable users to opt-in frequent updates. Other users can rely on the Maven Central release, which is published roughly every month (this is not strict, but based on numbers on https://mvnrepository.com/artifact/org.mockito/mockito-core).

All in all, we have not received any more feedback regarding too frequent published versions. We are also happy that we can still receive quick feedback and bisect based on Bintray binaries to find out which version (and thus PR) introduced the regression.

I would like to thank everyone that participated in this discussion and worked with us to tackle this issue. If you have any more specific feedback regarding our release mechanism, please file your feedback at https://github.com/mockito/shipkit

@TimvdLippe TimvdLippe closed this Nov 4, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment