Idea for interactive template generation: multi-objective, multi-fidelity, batch optimization, etc.

[sgbaird]

There's an idea I'm playing around with and thinking about implementing to make some of the advanced BO topics more accessible, especially in chemistry and materials science.

With the escience notebook tutorials that I put out (see @sp8rks's YouTube playlist), I started expanding them with progressively complicated names as I was incorporating different features:

• 2.8-ax_service_existing_data.ipynb
• 2.8.0.1-ax_service_existing_data_saasbo.ipynb
• 2.8.0.2-ax_service_existing_data_saasbo_multi_objective.ipynb
• 2.8.0.3-ax_service_existing_data_saasbo_multi_objective_batch.ipynb
• 2.8.0.4-ax_service_existing_data_saasbo_multi_objective_batch_equality.ipynb

A lot of these were based on content from other issues, often using the Service API, and occasionally delving into Developer API or BoTorch components. As I was creating these extra tutorials with a labmate in mind (@ramseyissa), it struck me how modular the functionality was (👏) as well as the sheer number of notebooks that would be required to cover every combination of these.

My idea is to create something somewhat similar to the interactive API snippets:

except allowing for an interactive selection of the desired features.

For example, a user could choose from the following options to create the corresponding template for 2.8.0.4-ax_service_existing_data_saasbo_multi_objective_batch_equality.ipynb:

• multi-objective
• continuous multi-fidelity
• discrete multi-fidelity
• multi-task
• high-dimensional
• batch optimization
• asynchronous optimization (mutually exclusive with batch)
• linear constraints
• equality constraints
• attach existing data (e.g., from CSV file)

Other categories:

• Choice of acquisition function (e.g., minimizing model error Which acquisition function should I use to iteratively improve model accuracy rather than optimize a target value? #930)

And maybe in terms of the user experience, it would look and act like the PyTorch installation page:

pytorch-installation-ux.mp4

In terms of actually implementing the above, I wonder if ipywidgets might come in handy. I'm not very well versed in HTML and Javascript. I'm also having trouble finding the general language to describe this type of interface. An alternative would be a Google Colab notebook that someone can run based on the options they select (which is probably where I would start).

For PyTorch's graphical implementation, see What is the PyTorch installation interactive grid called and how do I make my own?.

The template snippets at the bottom could be generated in advance for all combinations, rather than trying to run something on-the-fly. Additionally, it would be possible to run unit tests for each of the snippet combinations (this could simply be ensuring that it runs without errors).

I wanted to make a post to get some feedback and also have a place where I can refer back to it. What are your thoughts?

EDIT: also gauging interest via Twitter and LinkedIn

[CharlyEmpereurmot]

As a chemist / material scientist discovering Ax just right now and looking into how to have batches for single-objective optimization with ALEBO, I am seduced.

[mpolson64]

This is a neat idea! We are always looking for ways to improve our documentation and tutorials to make learning Ax easier. We already have some plans for improving docs in the pipeline, and I will raise this concept to the team as well. As always, thanks for your dedication to improving Ax for all our users :)

[sgbaird]

I'm fleshing this idea out more. I'm planning to:

• Tease out the PyTorch installation interface code as a MWE that I'll adapt to Ax
• Use Jinja2-based templating to create the scripts

[sgbaird]

I created a repo called Honegumi (骨組み, pronounced "ho neh goo mee") which means skeletal framework in Japanese. I wrote "a gentle introduction to Jinja" Colab tutorial that will make it easier for others to contribute via general feedback, design principles, and features.

Just as a very basic example, the following Jinja template has a toggle for switching between single- and multi-objective, going from the objective_name and minimize kwargs to the objectives kwarg with ObjectiveProperties

from ax.service.ax_client import AxClient
from ax.utils.measurement.synthetic_functions import branin
{% if use_moo %}
from ax.service.utils.instantiation import ObjectiveProperties

obj1_name = "branin"
obj2_name = "neg_branin"

def branin_moo(x1, x2):
    """Multi-objective branin function

    The first objective is the normal branin value and the second
    objective is the negative branin value.
    """
    return {obj1_name: branin(x1, x2), obj2_name: -branin(x1, x2)}
{% endif %}

ax_client = AxClient()
ax_client.create_experiment(
    parameters=[
        {"name": "x1", "type": "range", "bounds": [-5.0, 10.0]},
        {"name": "x2", "type": "range", "bounds": [0.0, 10.0]},
    ],
{% if use_moo %}
    objectives={
        obj1_name: ObjectiveProperties(minimize=True, threshold=None),
        obj2_name: ObjectiveProperties(minimize=True, threshold=None),
    },
{% else %}
    objective_name="branin",
    minimize=True,
{% endif %}
)

for _ in range(15):
    parameters, trial_index = ax_client.get_next_trial()
    results = branin{% if use_moo %}_moo{% endif %}(
        parameters["x1"], parameters["x2"]
        )
    ax_client.complete_trial(trial_index=trial_index, raw_data=results)

{% if use_moo %}
pareto_results = ax_client.get_pareto_optimal_parameters()
{% else %}
best_parameters, metrics = ax_client.get_best_parameters()
{% endif %}

Hoping to define some good design principles and a roadmap. Lmk if you're interested in helping!

[lena-kashtelyan]

@sgbaird, this is awesome, sorry we left this unanswered for a while! Any help you currently need?

[sgbaird]

@lena-kashtelyan, thanks! No worries. Actually, I almost have a full working prototype with just a few options. I'll share that really soon! I'd love to get some help with it.

[sgbaird]

Hi @lena-kashtelyan, what do you think of this kind of functionality in terms of the high-level interface? (nothing specific to Ax in this one - still need to connect the interface to the backend)

https://github.com/sgbaird/honegumi/blob/ax/docs/bootstrap_examples/bootstrap_gpt_help/main.html

(download and open in a browser)

Screencast: https://app.screencast.com/ywGS5w2FsLygO

[lena-kashtelyan]

This is very cool! Where are you thinking of hosting this, @sgbaird? Is the thought to put these on the Ax website or somewhere else?

[sgbaird]

@lena-kashtelyan, I've been wondering about that. For now, I plan to have a small Readthedocs page based on https://github.com/sgbaird/honegumi. While I'm focusing on Ax implementations, I also wanted to set it up so that solutions for other platforms could be added.

[sgbaird]

Hi @lena-kashtelyan, I'm actively thinking about that and open to feedback. While I'm prototyping, I plan to expose it via a readthedocs generated from https://github.com/sgbaird/honegumi. I'm designing the tool in a way that can be extended to other platforms; however, the Ax functionality has its own namespace and therefore can be installed independently from other platforms. Right now, my focus is on Ax.

I have almost all of the functionality ready:

• https://github.com/sgbaird/honegumi/blob/ax/scripts/generate_scripts.py
• https://github.com/sgbaird/honegumi/tree/ax/docs/generated_scripts/ax
• https://github.com/sgbaird/honegumi/tree/ax/tests/generated_scripts/ax
• https://github.com/sgbaird/honegumi/actions/runs/5766340905
• https://github.com/sgbaird/honegumi/tree/ax/docs/sandbox/jinja2

The last major piece of plumbing I need is programmatically running pytest and gathering the results within Python to pass it to the HTML file.

[lena-kashtelyan]

Wow, very cool! @mpolson64 and @esantorella are thinking through a potential redesign of our website infra/deployment, and this seems like it could be an excellent fit to play into that project. I'll assign this issue to @mpolson64 to think that through : )

[sgbaird]

Hi @lena-kashtelyan, @mpolson64, and @esantorella, I have a working example on the index page of https://honegumi.readthedocs.io/. I'm keen to get your thoughts! See the short screencast below:

honegumi-demonstration.mp4

cc @Balandat @saitcakmak @bernardbeckerman @Ryan-Rhys

[lena-kashtelyan]

Wow this is so cool!! cc @mpolson64 who is thinking through our potential website reset –– would be great to feature this!

[sgbaird]

Thank you! I felt really good getting to this working example. I would love some help on continuing to expand/develop this and make sure I don't make some early decisions that make it really hard to maintain/scale later on, add features that aren't as useful, etc.

[sgbaird]

Lots of progress made since then! https://honegumi.readthedocs.io/

• Now there are 10 visible options
• We added tooltips next to options
• Tutorials and conceptual docs added (more to come)