Is there a way to set a random seed for the Ax Service API.

[aagarwal1999]

I have a use case in which users want to be able to view the trials they are going to run before they run them and I would like to show the exact values of the trials that are going to be run. Is there a way to set a random seed for the service API for reproducibility.

[lena-kashtelyan]

@aagarwal1999, yep, there is –– you will need to make a custom generation strategy. You can get one by resetting it right after your experiment:

ax_client = AxClient()
ax_client.create_experiment(...)
ax_client.generation_strategy = choose_generation_strategy(
    search_space=ax_client.experiment.search_space,
    random_seed=239,  # Whatever random seed you want.
)

You can import choose_generation_strategy from here:

Ax/ax/service/utils/dispatch.py

Line 18 in 7d34dd3

def choose_generation_strategy(

Then, you will need to set the torch seed before each call to ax_client.get_next_trial, like so: torch.manual_seed(1000). More on the torch seed in this issue: #95.

Let me know if that fully solves the issue.

[aagarwal1999]

I am getting an error stating:

TypeError: choose_generation_strategy() got an unexpected keyword argument 'random_seed'

I suspect this is due to the fact that I am not using the latest version of Ax. However, when I updated to the latest version, I was getting an import error when I try and import AxClient

File "/mnt/nfs/home/ankit/.conda/envs/dstorch-ankit/lib/python3.6/site-packages/ax/models/torch/botorch_defaults.py", line 21, in <module> from botorch.optim.optimize import optimize_acqf ImportError: cannot import name 'optimize_acqf'

[lena-kashtelyan]

@aagarwal1999, you're right that the argument was unexpected because you had a later version; the BoTorch defaults import error, however, is news to me. Did you try to update BoTorch also?

[aagarwal1999]

That fixed it! Thanks

[lena-kashtelyan]

@aagarwal1999, also, I didn't know this when I responded to your question, but setting the random seed this say in the long run is non-ideal as it sets the seed for torch globally, which may impact model performance. I will have a fix for this on master by tomorrow –– will add a random_seed top-level argument to AxClient instantiation, which should make the seed safer to use.

[aagarwal1999]

Hi, has the fix been implemented for the AxClient seed?

[lena-kashtelyan]

Hi, Ankit! Not yet, but a partial fix will be on master soon (possibly today; I will update this issue like I said I would). Partial, because there isn't a way to fully make the Bayesian optimization part of the process reproducible (that is safe; I learned that setting the torch.manual_seed is not safe for the model performance). When we fix the torch seed, it's fixed globally, which can have drastic negative effects on the performance of the BoTorch-backed models, which we use for Bayesian optimization (that's what we are doing through the ad-hoc solution you are using now).

Now, the partial fix will provide a random seed setting to AxClient, which will make the initial quasi-random trials fully reproducible and the subsequent trials somewhat reproducible (but as you generate more trials, they will start differing more and more).

Finally, if you want reproducibility because you want your Ax optimization to use data from previous optimization, you can attach those trials via attach_trial and the data will be used in optimization. However, do make sure that those trials are actually applicable to the new optimization (for example, if you evaluate the trials on some data that has non-stationarity, then attaching data from an old optimization will worsen your new optimization results).

Let me know if that helps / makes sense!

[lena-kashtelyan]

Closing this issue because the random_seed argument for AxClient is now included into the latest stable version –– 0.1.5. Note that it does not make optimization fully reproducible, as described above.