Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build time A/B testing support #113

Open
mockitoguy opened this Issue May 6, 2017 · 12 comments

Comments

Projects
None yet
2 participants
@mockitoguy
Copy link
Member

mockitoguy commented May 6, 2017

Use case

Run build once, make some change, run build again, then compare build results. Sounds familiar? For certain kinds of changes, doing such A/B testing is super useful. It would be an excellent addition to release tools. After all, we want to really improve the lifecycle of software consumers and producers (easier, safer version bumps!). Example use cases:

  • bumping version of a dependency and validating if the project still works ok (#85). Bumped version can be of some compile dependency but also of a built-time dependency such as the version of a some Gradle plugin we use.
  • making change in build logic and validating if stuff still works.

Example workflow

Example implementation of the build comparison workflow:

  1. Clone a project from GitHub
  2. Copy the clone to another directory so that we have 2 identical clones
  3. Run a set of commands in clone 1, for example "./gradlew build -PsomeToolVersion=1.4"
  4. Run a set of commands in clone 2, for example "./gradlew build -PsomeToolVersion=1.5"
  5. Compare build directories
  6. Analyze comparison results

Suggested design

Suggested starting point design - check out #114 (It's only a suggestion! Pick your own path if you want, just keep the team informed about the design decisions! :)

@mockitoguy

This comment has been minimized.

Copy link
Member Author

mockitoguy commented May 6, 2017

@epeee, is this something you want to take a look? :) It's a very useful feature, not tangled with the internals of the project, and offers interesting challenge for you!

@epeee

This comment has been minimized.

Copy link
Member

epeee commented May 7, 2017

@szczepiq yes, I will take a look at this one. sounds really useful, great idea!

@mockitoguy

This comment has been minimized.

Copy link
Member Author

mockitoguy commented May 7, 2017

@epeee, that's great! This is a big ticket, if you have any design / thoughts before / during implementation, let us know. THANKS!

@mockitoguy

This comment has been minimized.

Copy link
Member Author

mockitoguy commented May 13, 2017

@epeee, do you think you will be able to work on this feature? No pressure - just asking!

@epeee

This comment has been minimized.

Copy link
Member

epeee commented May 13, 2017

@szczepiq yes, I will be able to work on this one (already started working on it). Sadly, I was more busy than usual during the last days.

@mockitoguy

This comment has been minimized.

Copy link
Member Author

mockitoguy commented May 13, 2017

Thank you for confirmation! This makes it easier to plan our roadmap :)

@mockitoguy

This comment has been minimized.

Copy link
Member Author

mockitoguy commented May 23, 2017

@epeee, I totally understand it's hard to find time for OSS at times! Do you mind if we hand this epic over to a different contributor? You can reengage whenever you have availability, just let us know and we will reorganize the team :)

@epeee

This comment has been minimized.

Copy link
Member

epeee commented May 24, 2017

@szczepiq sorry for the delay. What about reassigning it if I'm not able to come up with a solution on Fr (it looks like I will have some time to work on this one tomorrow)?

@mockitoguy

This comment has been minimized.

Copy link
Member Author

mockitoguy commented May 25, 2017

Sounds good!

@epeee

This comment has been minimized.

Copy link
Member

epeee commented May 26, 2017

a quick update: I made quite some progress yesterday on the compareAB task and it looks like I will be able to push a very first version of it today.
cloneGitHubRepository, runA and runB tasks are already available in the abTesting branch.

@epeee

This comment has been minimized.

Copy link
Member

epeee commented May 26, 2017

I just pushed the compareAB task including serialization. Tests are still missing but I will work on them now.
Apart from that the only part which is missing now is the analyze part. It would be great if we could have some kind of discussion/brainstorming about how the analyze result could look like (in order to not go into a wrong direction).

@mockitoguy

This comment has been minimized.

Copy link
Member Author

mockitoguy commented Jun 13, 2017

Hey @epeee, let's deprioritize this ticket for now and revisit when e2e testing is complete after 1.0. I'm finding it hard to focus on this at the moment. In the meantime, I've added 2 more "please contribute" tickets that would be really useful for 1.0 release!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.