-
Notifications
You must be signed in to change notification settings - Fork 90
Finishing an experiment
Josh Hiles edited this page Dec 22, 2023
·
1 revision
As your candidate behavior converges on the controls, you'll start thinking about removing an experiment and using the new behavior.
- If there are any ignore blocks, the candidate behavior is guaranteed to be different. If this is unacceptable, you'll need to remove the ignore blocks and resolve any ongoing mismatches in behavior until the observations match perfectly every time.
- When removing a read-behavior experiment, it's a good idea to keep any write-side duplication between an old and new system in place until well after the new behavior has been in production, in case you need to roll back.
- Publishing results
- Controlling comparison
- Adding context
- Expensive setup
- Keeping it clean
- Ignoring mismatches
- Enabling or disabling experiments
- Ramping up experiments
- Running candidates in parallel (asynchronous)
- Testing
- Handling errors
- Designing an experiment
- Finishing an experiment
Sometimes scientists just gotta do weird stuff. We understand.