Experiments docs and experiment cleanup #771
Conversation
|
Worth noting that we don't pass experiments on to the bootstrap. Will figure that out when we have an experiment in the bootstrap I guess! |
EXPERIMENTS.md
Outdated
|
|
||
| ```bash | ||
| buildkite-agent start --experiment experiment1 --experiment expertiment2 | ||
| ``` |
There was a problem hiding this comment.
Could we add a config file example too? Maybe even above this? Only because I think that's the most common way people configure their agent, and people might not be sure of the array/list syntax.
|
Change made @toolmantim. |
EXPERIMENTS.md
Outdated
| buildkite-agent start --experiment experiment1 --experiment expertiment2 | ||
| ``` | ||
|
|
||
| Or uou can set them in your [agent configuration file](https://buildkite.com/docs/agent/v3/configuration) and they will apply whenever your agent runs: |
There was a problem hiding this comment.
"you" typo here! Probably don't need "and they will apply whenever your agent runs" I reckon.
There was a problem hiding this comment.
Otherwise, looking ace! 🎉
| @@ -0,0 +1,29 @@ | |||
| # Experiments | |||
|
|
|||
| We frequently introduce new experimental features to the agent. You can use the `--experiment` flag to opt-in to them and test them out: | |||
There was a problem hiding this comment.
Can we say something along the lines of "these may be removed at any time, so use with caution". One of the benefits of not documenting experiments is that if we decide to ditch something, and no one will really know about it since it was never documented.
I'm just worried about experiments becoming something "supported" but not really "supported" (if that makes sense). i.e. if someone finds a bug in the msgpack experiment, do we fix it? If it's blocking them from doing something, do we consider it a proper bug? I think it's important to just setup the expectation that this stuff is very experimental, it may not become stable, and you shouldn't rely on anything.
Readings docs also made me wonder what the behaviour is when you activate an experiment that doesn't exist, do we error? Or do we just warn? Might be good to check since if peeps start adding it to their configs, then we remove it as an experiment because it lands in stable, we wouldn't want to fail their agents...
There was a problem hiding this comment.
Yup, good thinking. I think we basically say those features are likely to break or change in the future. We'll respond to bug reports on them, but we make zero guarantees around whether they will exist in the next point release.
There was a problem hiding this comment.
Readings docs also made me wonder what the behaviour is when you activate an experiment that doesn't exist, do we error?
I dunno, I'm fine with the current behaviour. If it's there, it turns on, if not, it doesn't.
There was a problem hiding this comment.
I dunno, I'm fine with the current behaviour. If it's there, it turns on, if not, it doesn't.
Cool cool, just wanted to make sure it didn't blow up with a "experiment-123 does not exist" type thing..
|
@keithpitt I made the changes you requested, let me know what you think. |
This adds an
EXPERIMENTS.mdwhich lists agent experiments available.