Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automated e2e testing and pushing version upgrades to consumers #85

Open
mockitoguy opened this Issue Apr 23, 2017 · 9 comments

Comments

Projects
None yet
4 participants
@mockitoguy
Copy link
Member

commented Apr 23, 2017

Producer-client lifecycle

  • When working on release tools I constantly build the sample project with new version of tools. It would be great to streamline this so that when we run "build" in the release tools project we have confidence that stuff works.
  • I frequently need to bump versions of tools in the client projects (mockito, example project) by creating PRs and merging them. That's an overhead I'd like to eliminate. In general, I'd like the clients to be on recent version of our tools.

Above are the problems of the challenges of the producer-client lifecycle. Let's figure out a set of plugins an features that will streamline evolution and adoption of new versions.

Suggested implementation

Let's try to identify a scalable solution that can evolve into a general-purspose feature that allows convenient evolution of versions of OSS libraries. Below is the result of our internal discussions with @mstachniuk and @wwilk:

  1. In producer (mockito-release-tools) we can configure a list of “clients” we care about most. We would start with 2: (mockito + mockito-release-tools-example).
  2. For each client we provide a way to locally run e2e test with that client. For example: “./gradlew e2eTest” will run all clients, “./gradlew e2eTests:mockito:e2eTest” will run only e2eTest for Mockito client. e2e test is a) shallow clone b) bump version of producer to locally built c) performRelease -Preleasing.dryRun
  3. Running “build” includes running all “e2eTest” tasks. So if one of the clients is broken the build fails and there is no release on CI.
  4. When CI job is successful and we publish new version of producer we automatically create PR in GitHub for each client. The PR contains the version bump of the producer. For some clients we may choose not to publish PRs, those clients will be used only for e2e testing.
  5. Bonus: we can add a functionality that will make the CI job of the producer automatically merge the PR created for the client. We would use it for some clients, like “example" projects. We don’t want to use it for Mockito, the Mockito team will decide whether they want to merge or not. So the “wait for PR and automatically merge” functionality needs to be configurable per client.

Suggested solution would scale because

  • we acknowledge that there are different clients: those that we only use for testing, friendly projects that we submit PRs with version bumps, finally projects we own and want to force push the change
  • we move the building of the clients (or at least some of that 😉) into their own CI pipeline
  • we don’t bypass GitHub pull request model with force pushing changes without PR (Mockito team would probably not agree to force push version bumps)
  • we don’t bypass client team’s validations when pushing code to their project. If we push version bump directly and it causes client project build failure who is going to fix it? :)
  • we would wrap the feature in a sweet little plugin that is reusable all over the world :DDD
  • we have easy way to build/reproduce locally

Starting point

To get started, let's get a single client local e2e test (example project) hooked up in the release-tools project. For version bumping, we can use Mockito's pattern of keeping the version of tools in gradle.properties file.

Future

In future this could potentially turn into more general capability we would add to release tools. It would be great if clients of release automation could automatically validate if their changes work with selected open source projects on GitHub. For example, I would love if every change of Mockito was tested with some other GitHub projects like guava. This potentially can be extremely useful feature.

@mockitoguy

This comment has been minimized.

Copy link
Member Author

commented Apr 23, 2017

@mstachniuk, how does this one sound? Check out the "future" section, this feature can turn out into something huge!!!

@mstachniuk

This comment has been minimized.

Copy link
Contributor

commented Apr 23, 2017

It sounds great! It's a huge. Let's maybe starts from something simpler.
New successful release of mockito-release-tools automatically bump version of tool in mockito-release-tools-example, make some small change and release it! @szczepiq What do you think?

@szpak

This comment has been minimized.

Copy link
Member

commented Apr 23, 2017

👍

I've implemented automated E2E smoke testing in my gradle-nexus-staging-plugin and I feel much more relaxed releasing new versions. Having those testing automated is crucial to provide reliable Continuous Delivery.

@szpak

This comment has been minimized.

Copy link
Member

commented Apr 23, 2017

It would be great if clients of release automation could automatically validate if their changes work with selected open source projects on GitHub.

@szczepiq Again, we share the vision of the future of CD :)

@mockitoguy

This comment has been minimized.

Copy link
Member Author

commented Apr 23, 2017

@szpak, we share the vision, now it's time to start working together :) Join us and we'll be unstoppable! 4 of us cranking great release automation toolkit!!!

https://github.com/Codearte/gradle-nexus-staging-plugin is a great plugin. We could merge our efforts :)

@mockitoguy

This comment has been minimized.

Copy link
Member Author

commented Apr 23, 2017

Let's maybe starts from something simpler.
New successful release of mockito-release-tools automatically bump version of tool in mockito-release-tools-example, make some small change and release it! @szczepiq What do you think?

Sounds great! We'll have more example projects, for different use cases (for example: minimal configuration, based on defaults VS full configuration; with notable releases VS without notable releases; with bintray VS with just maven central). So it would be good if the impl was generic enough to handle more 'test' projects in the future :D

@mockitoguy

This comment has been minimized.

Copy link
Member Author

commented Apr 23, 2017

@mstachniuk, assigned to you, feel free to unassign if you choose to work on something else.

@mockitoguy

This comment has been minimized.

Copy link
Member Author

commented Apr 28, 2017

I updated the description with a suggested solution.

@mockitoguy mockitoguy changed the title Automated e2e integration tests with example projects Automated validation of clients and version evolution of producers Apr 28, 2017

@mockitoguy mockitoguy changed the title Automated validation of clients and version evolution of producers Streamline producer-client lifecycle Apr 28, 2017

@mockitoguy mockitoguy added the Epic label Apr 28, 2017

@mockitoguy mockitoguy changed the title Streamline producer-client lifecycle Automated e2e testing and pushing version upgrades to consumers May 2, 2017

mstachniuk added a commit that referenced this issue May 6, 2017

mockitoguy added a commit that referenced this issue May 15, 2017

Merge pull request #126 from mockito/ms
#85 Automated e2e testing

mstachniuk added a commit that referenced this issue May 16, 2017

@mstachniuk

This comment has been minimized.

Copy link
Contributor

commented May 25, 2017

Current status:
Check my changes on ms branch feel free to continue if you wish.

For now E2ETestingPlugin makes a few things:

  • clone mockito-release-tools-example to mockito-release-tools-example-pristine
  • clone from *-pristine dir to *-work dir
  • execute ./gradlew publishToMavenLocal testRelease ... in *-work dir

For testing purposes you need to apply E2E plugin in *.gradle file and try run runTestReleaseMockito-release-tools-example task

@mockitoguy mockitoguy assigned epeee and unassigned mstachniuk Jun 18, 2017

@mockitoguy mockitoguy added size:L and removed Epic labels Oct 29, 2017

@mockitoguy mockitoguy removed the help wanted label Mar 18, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.