-
Notifications
You must be signed in to change notification settings - Fork 294
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
warm starting experiment with historical records #1297
Comments
Hi @chanansh, can you help clarify some terminology? When you say "warm-start a model," can you be a little more precise? For example, are you trying to perform hyperparameter optimization on a Machine Learning model? Or are you referring to some other type of model? And when you're talking about past historical observation, what type of data/observation are you talking about precisely? The more you can help clarify your use-case, the more precisely we will be able to help answer your question. Thanks! |
If you're just looking to attach iterations to your experiment, you can try simply using Closing this out for now but please feel free to comment and/or open this back up if you have further questions! |
We want to optimize f(x) by a two step iterations: But I have past historical x, f(x) tuples. |
How are you defining your metric(s)? If you're defining custom metrics as in the developer API tutorial ([link])(https://ax.dev/tutorials/gpei_hartmann_developer.html#8.-Defining-custom-metrics), you could include in your
Hope that helps, and let me know how I can further assist : ) |
mmm. your code snippets does not include where I use past x_i or f(x_i). |
@chanansh, which Ax API are you using? Without knowing this it is not possible to fully answer your question with a code snippet you are looking for. If Service API, you would do this, as @bernardbeckerman pointed out above:
Does this answer your question? |
Perhaps two more words to the motivation/idea of it. Perhaps there is another way to handle this, by overriding or "faking" the trials? Or setting the runner differently? I'm new into ax and don't have the whole picture of what would be the best part to implement something like this |
Hi there, apologies for taking so long on this. Please see the example below, which is based on the Service API tutorial. I've copied the first part from the tutorial sections 1 through 3, then I insert one manual arm between steps 3 and 4, then I perform the optimization loop in step 4. Please let me know if this addresses your questions, and anything else I can help you with.
Once you run that, you can get the experiment dataframe using
which should look something like this: To do this for multiple arms, you can execute the added code above in a loop. Please let me know if that helps! |
Hi, yes that worked! because while fetching the data for the fitting of the GP-Surrogate, it just takes one single point (i've loaded a lot of "historical" points... is that possible? Thanks a lot!!!! |
Let me parse this into two separate questions. Please let me know if I missed anything you're wondering about.
The function Note that all BO experiments generally start with a few Sobol trials to seed the BO model with, so it may be that you need to execute a certain number of trials before the BO model takes effect. You can see this number by printing If you'd like to customize your generation strategy, you can supply some kwargs to the function
Can you share your example, and how you can tell that the manually added arms aren't making it into the GP fit? That would be super useful! |
@bernardbeckerman thank you!
and would like to attach this trials into my experiment using the Service API.
First i attach the historical trials as in your example given above
And they are being added correctly (with the exception that the "generation_model_key" is not being set properly, but it isn't critical. Afterwards I continue with the normal client loop and my
The problem is that while the Botorch model fetches the data, it just takes the first value of the historical data to train and then I get wrong results I've found out, that the problem relies on naming of the arms: all the arms are called "mid". then everything works correctly Thanks!!! |
That's great! Yes arms can be tricky that way - may be worth printing a warning to the user if their arm definition collides with another. Thanks for this detailed response and for raising these issues! Closing this out but please let me know if you have any followup questions! In particular, I'm not sure why the |
@bernardbeckerman Hi, sorry for reopening this old thread (I wanted to avoid excessive clutter), but I have a quick question regarding 'warm starting' an experiment: in my use case I do
and i want to replace the GS in What would be the correct way to go about this? |
what is the best practice to warm start a model with past historical observation?
The text was updated successfully, but these errors were encountered: