-
Notifications
You must be signed in to change notification settings - Fork 303
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Constrains that include fixed or choice parameters raising an incorrect and confusing "Parameter does not exist in search space" exception #621
Comments
My version of AX is |
@petrcezner, we'll look into this shortly, thank you for reporting! |
Hi, @petrcezner! Thank you very much for reporting this. This is 1) not a bug in a sense that we do intentionally disallow constraints on fixed and choice parameters, but also 2) a bug in that you get a non-informative error that does not explain what is going on and that it happens not on experiment creation, but rather during candidate generation. Schrödinger's bug : ) Re: 1), I believe that the reason why this is not allowed (cc @Balandat to check me on that one and add more detail) is that we expect all constraints to be linear under-the-hood, and it's not possible to represent constraints that include fixed and choice parameters that way. Re: 2), we will address this to make sure a proper error is raised about this earlier instead of the confusing one you ran into. |
@lena-kashtelyan Thank you for your answer! I will check such constraints in the evaluation function. Thank you for your time! |
So there shouldn't be a fundamental issue including fixed parameters, as we can just absorb their value into a new bound. That said, if the problem is simple enough, you can just do this yourself. Instead of As far as choice parameters are concerned, the reason we didn't allow constraints on these was that they were one-hot encoded and thus representing the constraint as a linear constraint is not straightforward. However, we made some changes recently that handle choice parameters in a different way and optimize over their actual values, so it would potentially be possible to incorporate them into parameter constraints (in fact, potentially even nonlinear ones). cc @dme65 , @vishwakftw |
Hello @Balandat , I try to have following constrains: File "MY_PYTHON_PATH\lib\site-packages\ax\service\utils\instantiation.py", line 249, in constraint_from_str
assert right in parameter_names, f"Parameter {right} not in {parameter_names}."
AssertionError: Parameter 288 not in dict_keys(['t1', 't2', 'lw', 'rw', 'dt2']). or File "MY_PYTHON_PATH\lib\site-packages\ax\service\utils\instantiation.py", line 248, in constraint_from_str
assert left in parameter_names, f"Parameter {left} not in {parameter_names}."
AssertionError: Parameter -lw not in dict_keys(['t1', 't2', 'lw', 'rw', 'dt2']). It looks like even thought it is a parameter constraints, the order constraints is checked (because of |
There seems to be some issue with the string parsing here, the Btw, |
Hi, @petrcezner, the expected formatting for constraints is discussed here: https://ax.dev/tutorials/gpei_hartmann_service.html#2.-Set-up-experiment. Under the hood, we construct Ax However, what I'm wondering is why would someone want such a constraint when it can be directly encoded in the bounds of the parameter? If there is a fixed parameter |
I've added new error messages to The fix for this is now on master and will be in the next stable release. |
I was about to post an issue report on this as well, since I need to put constraints on choice variables, especially between them.
I believe that the fix suggested by @Balandat Jul 17 sounds good. |
@LuddeWessen, let me put this into a separate issue! |
The fix is now part of the latest stable version: 0.2.3. |
Hello,
I am trying to have a fixed parameter in parameter_contrains of the experiment. The experiment setup looks like this:
The setup seems to be OK. However, when I try to call
ax_client.get_next_trial(ttl_seconds=30)
I get following error:The problem is that the SearchSpace class accepts only Range and Choice parameters. So I try to change the parameters from fixed to choice in the setup:
and it doesn't work either. The names of the parameters change from
t1
tot1_OH_PARAM_0
, so it again creates the same exception (parameters are not found). Also, the parameter values are in the range [0, 1]. Therefore, the constrain wouldn't be satisfied.I couldn't find any restriction of parameters in the documentation of
ax.core.parameter_constraint.ParameterConstraint
class.May I ask you how to fix it? Is it even possible to have such constraints? If not, is there any workaround except checking the constraints manually in the evaluation function? This issue is probably similar to #574 or #383.
The text was updated successfully, but these errors were encountered: