Issue when starting an AxClient with out-of-design points

[AngelFP]

Hi all,

First of all thank you for the great library, we've been extensively using it for optimizing the performance of particle accelerators and it's been working really well.

We're currently using Ax within a higher level framework that, among other things, enables the user to update the search space. One use case for this is to resume an old optimization with a more narrowed-down parameter range.

To implement this, I've used the fit_out_of_design parameter to enable the model to use data outside of the updated range, as well as this workaround (#768 (comment)) to attach the old out-of-range data to the AxClient. However, I'm facing an issue where, despite using fit_out_of_design=True, the out-of-design data is being filtered out. This is the error I get:

ValueError: There are no feasible observed points.  This likely means that one or more outcome constraints or objective thresholds is set too strictly.

Below is a simple reproducer of this issue:

from ax import Arm
from ax.service.ax_client import AxClient, ObjectiveProperties
from ax.modelbridge.registry import Models
from ax.modelbridge.generation_strategy import (
    GenerationStep,
    GenerationStrategy,
)


ax_client = AxClient(
    generation_strategy=GenerationStrategy(
        steps=[
            GenerationStep(
                model=Models.GPEI,
                num_trials=-1,
                model_kwargs={"fit_out_of_design": True},
            )
        ]
    )
)


ax_client.create_experiment(
    name="test_experiment",
    parameters=[
        {
            "name": "x1",
            "type": "range",
            "bounds": [0.0, 1.0],
        },
        {
            "name": "x2",
            "type": "range",
            "bounds": [0.0, 1.0],
        },
    ],
    objectives={
        "f": ObjectiveProperties(minimize=True),
    },
)


# Add old trials outside of seach space
old_trial_data = [
    [{"x1": -1.0, "x2": 0.1}, {"f": 0.5}],
    [{"x1": -2.0, "x2": 0.2}, {"f": 0.1}],
    [{"x1": 3.0, "x2": 0.2}, {"f": 0.2}],
    [{"x1": 2.0, "x2": 0.4}, {"f": 0.8}],
]
for inputs, outcomes in old_trial_data:
    trial = ax_client.experiment.new_trial()
    trial.add_arm(Arm(parameters=inputs))
    trial.mark_running(no_runner_required=True)
    ax_client.complete_trial(trial_index=trial.index, raw_data=outcomes)


# Try to get next trial.
ax_client.get_next_trial()

I think the issue is in the filter_constraints_and_fixed_features function. In particular, if I comment out these two lines, everything seems to work well

Ax/ax/models/model_utils.py

Line 563 in 800357a

for i, b in enumerate(bounds):

Is this the expected behavior or am I doing something wrong?

[Cesar-Cardoso]

Hi Angel! It is not the expected behavior, and it should be fixed after #2277. Thanks for flagging this issue!

[AngelFP]

Great to hear! Thanks for fixing this so quickly.