Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Monkeypatch Flipper to provide a consistent control group #3434
This is for the exercism/discussions#123 experiment.
Override to consider only the actor value (GitHub username), not the feature name, to provide a consistent control group across features, rather than a different 50% for each feature.
This new test fails without the monkey patch. Considering the boolean output and the random-like nature of Zlib.crc32 output (the hashing algorithm used), 250 iterations should more than cover it. The test completes in 40ms on my machine.
Cool, thanks. I totally didn't realize that @jnunemaker is a Githubber too. He just merged one of my PRs today, so I know he's responsive. Maybe we can just pull him in here.
@jnunemaker We're performing an experimental study on exercism.io with control and experimental groups, and we're using flipper's Percentage of Actors gate to separate the two groups. This would work great out-of-the-box if we used a single flipper feature to manage the entire experiment, but we wanted the ability to turn off exercism features individually, in case of problems in production with some new exercism feature, without disabling all of the experimental features. Since the key/id used in each flipper feature's percentage of actors gate contains the feature name, the group selected for each feature will differ. In order to have a consistent 50% control group across all flipper features, this PR monkeypatches that key/id to instead contain only the actor value.
@kytrinyx and I wanted to brainstorm about how flipper might be able to have a "suite" of features that select a common percentage of actors within that suite. This could take the form of grouping features, but since the Percentage of Actors gate is really the only one that this applies to, it could could take the form of a new gate that is similar to the Percentage of Actors gate—Consistent Percentage of Actors vs Random Percentage of Actors? Or better yet, a configuration option on the Percentage of Actors gate that excludes the feature name from affecting the actors selected? I'd love to hear your thoughts on this.
I would not recommend overriding Flipper's default behavior as that could be confusing for others down the road. I created an example for you of one way to achieve the same result without monkey patching flipper. What do you think of that? Happy to hear your thoughts either way.
This is an interesting idea. I don't have any good ideas off hand for something baked into flipper, but the experimental group thing I added above would be one way to do it currently. For each suite, you could have a group.