[GENERAL SUPPORT]: Impact of adding periodical reference data using attach_trial

[rachelhson]

Question

Hello,

In my application, I connect ax-client to the device, and the output value with the same parameter can change over # of iterations due to its natural degradation. I would like to track the reference parameter and value throughout the BO process. If I periodically add this reference data to the model, using attach_trial, do you think it could affect the BO modeling?

Thank you!

Please provide any relevant code snippet if applicable.

Code of Conduct

• I agree to follow this Ax's Code of Conduct

[bernardbeckerman]

Hi there, It sounds like you have a device that you want to tune the performance of, but performance generally degrades over time as the experiment proceeds, do I have that right?

If so, this is definitely a problem for Ax to treat. Ax typically requires that trials are stationary (i.e., successive measurements of the same arm's metrics should be drawn from the same distribution), or that there is some way to account for any nonstationarity that may exist (e.g., you can measure the performance of an arm relative to some status quo, then subtract the difference to obtain something that is at least close to stationary). Is there a way to account for this nonstationarity? For instance, if the performance degradation follows a predictable curve, you can use that to compensate the nonstationarity; or you can alternate evaluation of the status quo arm with that of test arms.

It's hard to give any more prescriptive advice without knowing more about your system. If you are able to provide more details like what kind of system you are optimizing, why/how performance tends to degrade, or if you're able to measure the performance of multiple arms at once, etc, that may help us make better recommendations.

[rachelhson]

Hello @bernardbeckerman

I understand that it could be a problem for Ax to treat in this case. However, I think the degradation would not be huge, I would say the amount of degradation could be treated as noise level.

In that case, would it be okay to add reference data using attach_trial or would it be better to hide reference data periodically collected from BO modeling?

[Balandat]

In that case, would it be okay to add reference data using attach_trial or would it be better to hide reference data periodically collected from BO modeling?

I think this is largely an empirical question, though my intuition says that including them should be preferable. Could you hold out some data and try both approaches and then evaluate which setup - the one using non-reference data only or the one using both - gives better out-of-sample predictions?

It would probably make sense to track the "age" of the device as an additional parameter that the surrogate can use as a feature to model this degradation. This issue / comment is closely related: #3061 (comment) (if you took this approach you'd definitely want to provide the reference data to the model)

[rachelhson]

This is actually interesting and great idea! Thanks for sharing the idea ! If surrogate model can capture degradation by both different parameters and iterations, that would be useful info and prediction we could have.

Do you think I can add 'trial_index' into meta_data? If then, surrogate model can use as a feature to model the degradation? What would be the right implementation?

[bernardbeckerman]

As Max mentioned, something like the TimeAsFeature transform mentioned here: #3061 (comment) could be helpful. I hope that's helpful!

[bernardbeckerman]

@rachelhson Closing this out since it's been a while, but please let me know if there's anything more we can help with here : )