Refactor Input Preproccesing and Mixed Optimization by jduerholt · Pull Request #626 · experimental-design/bofire

jduerholt · 2025-08-15T10:44:25Z

Motivation

This PR contains a major refactoring of BoFire. Here is a short summary:

botorch introduced the optimize_acqf_mixed_alternating functionality, which thanks to @TobyBoyne can now also deal with categoricals. I added a few other features there so that it is now ready for use within BoFire. optimize_acqf_mixed_alternating performs a block gradient descent optimization of the acqf and is an alternative to the GA on large mixed domains.
optimize_acqf_mixed_alternating expects the categoricals always in an ordinal encoding. So far, BoFire was transforming the categoricals upfront to a vector encoding (like one-hot) before entering the actual surrogate. This will now change, categoricals will be always ordinal encoded, and different vector encodings can be realized by using a botorch input transform namely the NumericToCategoricalEncoding (https://github.com/pytorch/botorch/blob/b1097c6c475f29f694532c3282393ce8a67a9d6c/botorch/models/transforms/input.py#L1628C7-L1628C35). For this reason, the meaning of the input_preproccesing_specs will change, it is now not a transformation that is applied before the data enters the actual botorch model, instead it is a transformation which is applied within the model.
This change will make setting up the acqf optimization much cleaner as we do not have to account on this level for the different encodings.
I am currently rewriting the AcquisitionOptimizer. @LukasHebing: why is the domain not an attribute of the acquisition optimizer? Would make it cleaner, or? I do not remember, why we decided to do it in the way we did it :D

Have you read the Contributing Guidelines on pull requests?

Yes

Test Plan

Unit tests.

jduerholt · 2025-09-17T14:34:01Z

Hey guys, this PR escalated a bit but it is ready for review now. The failing tests are due to NumericalToCategoricalEncoding input transform being not yet part of the latest botorch release. For this reason it only runs through in testing_against_latest_botorch. But I pinged the botorch guys regarding filing a new release for botorch.

So, happy reviewing, in case of questions reach out to me!

jduerholt · 2025-09-18T15:33:10Z

@bertiqwerty @TobyBoyne @LukasHebing

Anybody volunteering to review, or should I provide more info etc?

Best,

Johannes

TobyBoyne · 2025-09-18T15:34:01Z

I will have a look over the weekend :)

LukasHebing · 2025-09-18T15:35:55Z

@bertiqwerty @TobyBoyne @LukasHebing

Anybody volunteering to review, or should I provide more info etc?

Best,

Johannes

I think, I understood the motivation and can see where this is going. I can take a deeper look in next week, because these are a lot of changes...

bofire/data_models/surrogates/mixed_single_task_gp.py

bofire/strategies/utils.py

LukasHebing · 2025-09-30T10:50:56Z

@LukasHebing: the problem is that this input transform is not yet in the latest botorch release (it is in main and will be part of the next release), so one would need to install botorch from source for this. I can just copy the source code in here, should I? And as soon as it is in botorch release, we remove it again from bofire, maybe this is the smartest way of dealing with it ...

Ah, now I understand the problem. Is it possible to use the github branch as a source, instead of the pypi release? I know that is possible with poetry or uv.

Otherwise, we need to wait for the release anyway before we merge this PR, or?

I'll try to add botorch manually and check that.

jduerholt · 2025-10-01T12:40:43Z

@LukasHebing, I copied the class NumericToCategoricalEncoding from main botorch to bofire, until they have it in the next release, then we can remove it again (I also do not copy the tests over). But this gives us option to move forward, as I want to implement other things based on this PR ;)

jduerholt · 2025-10-01T14:12:38Z

Main tests are now passing, so you could clone it locally, I will have a look on the rest.

LukasHebing · 2025-10-02T04:16:55Z

@LukasHebing, I copied the class NumericToCategoricalEncoding from main botorch to bofire, until they have it in the next release, then we can remove it again (I also do not copy the tests over). But this gives us option to move forward, as I want to implement other things based on this PR ;)

That is a great solution. Thanks!

jduerholt · 2025-10-02T13:00:05Z

The test regarding latest botorch is failing due to a botorch bug in current main (meta-pytorch/botorch#3033). I try to fix this, but this just an edge case and should not affect the review process. Best, Johannes

LukasHebing

Sorry for the delay: I could now follow all the categoric variable handling which is done under the hood.

LukasHebing · 2025-09-18T04:30:09Z

bofire/data_models/priors/constraint.py

Doesn't have botorch also a great library for priors, which we use anyway later?
Is it maybe possible to just store a reference to the botorch priors / constraints with parameters, instead of building this elaborated structure? That would be more flexible, otherwise you need to add the infrastructure for all the new priors here as well.

Hmm, we are using the botorch priors, what I added was the option to use more of them also within our data models.

LukasHebing · 2025-09-23T07:38:55Z

bofire/data_models/surrogates/botorch.py

            if (
-                v.get(key, CategoricalEncodingEnum.ONE_HOT)
-                != CategoricalEncodingEnum.ONE_HOT
+                v.get(key, CategoricalEncodingEnum.ORDINAL)


Sorry, I also struggle to understand the scope of input_preprocessing_specs:

When ORDINAL is the only possible option. Why do we have other options.

Where are the user-specified encodings (descriptor, etc.)?

LukasHebing · 2025-09-29T12:52:49Z

bofire/strategies/utils.py

-    """Default input preprocessing specs for the GA optimizer: If none given, will use OneHot encoding for all categorical inputs"""
-    input_preprocessing_specs = {}
-    for input_ in domain.inputs.get():
-        if isinstance(input_, CategoricalDescriptorInput):


I guess, the CategoricalDescriptorInput will be handled in the botorch transformation?

TobyBoyne

Finally had the chance to look through everything again, thank you for waiting :)

Everything looks good, only remaining unresolved comments are pretty minor. Approved!

tests/bofire/strategies/test_acqf_optimization.py

jduerholt · 2025-10-06T15:24:38Z

I found a critical bug and fixed it, but will see over the next days if this is also affected anywhere else ;)

bertiqwerty

Sorry, I am a little late to the party. Thanks Johannes. Thanks to the other reviewers. Looks good from my side. I just left some minor comments.

bofire/utils/torch_tools.py

bofire/surrogates/random_forest.py

bofire/strategies/predictives/botorch.py

bofire/strategies/predictives/acqf_optimization.py

bofire/utils/torch_tools.py

tests/bofire/data_models/specs/surrogates.py

jduerholt · 2025-10-08T11:34:29Z

Ok, everything is now settled, tests are failing due to a new pydantic version which was released yesterday. Here is the fix: #638

But we have to wait until we merge this for a new botorch version, as there is one bug in the current release version which will lead to trouble. It is fixed now in botorch, but they have to file a release, I pinged Max regarding this. Let us see what they say.

jduerholt added 18 commits August 15, 2025 11:41

remove categorical methods etc.

c109cf8

remove input_preprocessing_specs

16b1ec0

refactor surrogates

2f1de33

refactor mixedgp

48d06c0

remove unused ard_num_dims keyword

48f1dbc

data models tests working again

5e0b15c

reimplement CategoricalGP

b70a6d3

more tests

b57f769

more tests

389cbde

add tests on prior constraints

54238c9

add prior tests to mapper

cf706d6

add more tests

56750be

add more tests

1620210

fix tests

753b125

more on tests

ab90483

fix test

dc0d8d3

fix another test

9303629

please pyright

ac684d1

jduerholt marked this pull request as ready for review September 17, 2025 12:51

jduerholt requested review from LukasHebing, bertiqwerty and e-dorigatti September 17, 2025 14:31

jduerholt requested a review from TobyBoyne September 17, 2025 14:34

jduerholt mentioned this pull request Sep 17, 2025

Ordinal encoding for surrogates for mixed alternating optimization #603

Closed

e-dorigatti reviewed Sep 21, 2025

View reviewed changes

bofire/data_models/surrogates/mixed_single_task_gp.py Show resolved Hide resolved

e-dorigatti reviewed Sep 21, 2025

View reviewed changes

bofire/strategies/utils.py Outdated Show resolved Hide resolved

make encoder part of bofire temporary

8961671

fix failing test in optimization_only

9905e61

Merge branch 'main' into refactor/mixed_optimization

16b8cc7

LukasHebing approved these changes Oct 2, 2025

View reviewed changes

TobyBoyne approved these changes Oct 6, 2025

View reviewed changes

tests/bofire/strategies/test_acqf_optimization.py Show resolved Hide resolved

tests/bofire/strategies/test_acqf_optimization.py Outdated Show resolved Hide resolved

jduerholt added 3 commits October 6, 2025 16:20

fix notebooks

e77732f

fix severe bug

2dae914

update notebook

e77e678

bertiqwerty approved these changes Oct 7, 2025

View reviewed changes

jduerholt added 5 commits October 7, 2025 11:24

fix kernel mapping

dfd787a

remove uncommented code

1740713

add enum

393d4cf

finalize comments from Behrang

9b49d08

refine test

93d0c82

jduerholt added 8 commits October 10, 2025 14:09

increase max attempts

7ffa287

changes for botorch 0.16.0

56b1a0b

please pyright

9a865d1

Merge branch 'main' into refactor/mixed_optimization

6d45bf5

update batch limit in tests

c500bad

please pyright

d21912f

pin pyomo version

64225e8

fix toml

cc09458

jduerholt merged commit 5126e09 into main Oct 24, 2025
10 of 12 checks passed

Conversation

jduerholt commented Aug 15, 2025

Motivation

Have you read the Contributing Guidelines on pull requests?

Test Plan

Uh oh!

jduerholt commented Sep 17, 2025

Uh oh!

jduerholt commented Sep 18, 2025

Uh oh!

TobyBoyne commented Sep 18, 2025

Uh oh!

LukasHebing commented Sep 18, 2025

Uh oh!

Uh oh!

Uh oh!

LukasHebing commented Sep 30, 2025

Uh oh!

jduerholt commented Oct 1, 2025

Uh oh!

jduerholt commented Oct 1, 2025

Uh oh!

LukasHebing commented Oct 2, 2025

Uh oh!

jduerholt commented Oct 2, 2025

Uh oh!

LukasHebing left a comment

Choose a reason for hiding this comment

Uh oh!

LukasHebing Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

jduerholt Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

LukasHebing Sep 23, 2025

Choose a reason for hiding this comment

Uh oh!

LukasHebing Sep 29, 2025

Choose a reason for hiding this comment

Uh oh!

jduerholt Oct 6, 2025

Choose a reason for hiding this comment

Uh oh!

TobyBoyne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jduerholt commented Oct 6, 2025

Uh oh!

bertiqwerty left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jduerholt commented Oct 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants