Add LFBO Pattern Search

[ethche]

Includes the LFBO Pattern Search autotuner, which modifies PatternSearch to search through configs using a learned acquisition function (RandomForestClassifier) according to the likelihood-free bayesian optimization framework [1].

• Similar to PatternSearch, we generate neighbors from search copies. But instead we generate random neighbors instead of exhaustive set.
• Filters a fraction of them to evaluate using a fitted RandomForestClassifier. This classifier is trained to learn which configs are the best x% of configs. As a result, the classifier learns which configs are likely to improve upon the best config seen so far.

This improves over PatternSearch in kernel latency and autotuning wall-clock time on B200 for a set of benchmark kernels. DifferentialEvolution can improve further upon this, but takes substantially longer. DESurrogate in #1096, has comparable performance. We also compare to UCBPatternSearch, a previous proposal which uses a Gaussian Process + UCB acquisition function.

Kernel latency:

Autotuning Wall-clock Speedup:

Autotuning Convergence Time:

Some example convergence plots:

[1] J. Song, et al. A General Recipe for Likelihood-free Bayesian Optimization

[jansel]

I don't understand what this is? What is it encoding?

[ethche]

This is just the dimension of the encoding

[ethche]

Changed the name to be more clear

[jansel]

Can we reuse encode_scalar here? If not, I'd like to combine this with encode_scalar since they are solving the same problem.

[ethche]

@jansel This is pretty much identical to encode_scalar for integer and poweroftwo. But the previous encode_scalar did not have any functionality for ListOf or PermutationFragments. How should we handle those two?

[ethche]

I'm happy to rename everything to encode_scalar, but given ListOf and PermutationFragments, I think we should allow encode to output a list of floats

[jansel]

This seems kind of error prone, can we have a cleaner way to do it?

[ethche]

Is this better?

[ethche]

Just for some background, I performed some new experiments and found that a new method, called LFBO Pattern Search, performs just as well as UCBPatternSearch, but doesn't need any new imports.

It's based on a RandomForestClassifier surrogate. It is trained on classifying which configs are among the top x% in performance. As such, it can be trained without having to filter out configs that time-out or have poor accuracy. I've structured the file to be similar to de-surrogate.