Add DE-Surrogate hybrid autotuner algorithm + early stopping option for DE and DE-Surrogate

[FranciscoThiesen]

This introduces DESurrogateHybrid, a novel hybrid optimization algorithm that combines Differential Evolution's robust exploration with Random Forest surrogate model's sample efficiency for GPU kernel autotuning.

Key features:

• Generates 3× more candidates than standard DE but only evaluates the most promising ones as predicted by the Random Forest surrogate
• Achieves 6.53% average performance improvement over standard DE
• 1.20× faster wall-clock time despite evaluating more configurations
• Learns kernel-specific optimization patterns automatically

Implementation:

• Works directly with Helion's discrete parameter spaces
• Uses ConfigEncoder to convert configurations to numerical vectors
• Refits surrogate model every 5 generations for continuous learning
• Configurable parameters: population_size, candidate_ratio, surrogate_threshold

Testing on 3 diverse kernels (MatMul, GELU, FusedReLU) shows:

• MatMul (compute-bound): -15.0% improvement, 1.39× faster convergence
• GELU (bandwidth-bound): -5.4% improvement
• FusedReLU (memory-bound): +0.8% (competitive, within margin)

[jansel]

Do you have more results you could share? Which kernels have you tested this on? What is the impact on tuning time and resulting performance?

[FranciscoThiesen]

@jansel sure.

I've plotted the convergence of this versus regular DE for 3 different kernels:

[FranciscoThiesen]

@jansel A better write-up:

Detailed Benchmark Results

Hardware Configuration

• GPU: NVIDIA H200 (8× GPUs, 141GB memory each)
• Compute Capability: 9.0 (sm_90a)
• Driver: 570.133.20
• CUDA/Triton: Latest via Helion

Kernels Tested

I evaluated DE-Surrogate on 3 diverse kernels spanning different computational characteristics:

1. MatMul-1024: Matrix multiplication (1024×1024 matrices)
   - Type: Compute-bound
   - Characteristics: Heavy arithmetic, benefits from good block/warp config
2. GELU-1M: GELU activation function (1M elements)
   - Type: Bandwidth-bound
   - Characteristics: Memory throughput critical, simpler compute
3. FusedReLUAdd-1M: Fused ReLU+Add (1M elements)
   - Type: Memory-bound mixed
   - Characteristics: Multiple memory operations, fusion optimization

All algorithms ran with ~1600 evaluations per kernel for fair comparison.

Performance Results

Kernel	Standard DE	DE-Surrogate	Improvement
MatMul-1024	0.0114 ms	0.0097 ms	-15.0% ⚡
GELU-1M	0.0071 ms	0.0067 ms	-5.4% ⚡
FusedReLUAdd-1M	0.0076 ms	0.0077 ms	+0.8%

Average: 6.53% better performance

• 2 out of 3 wins, competitive on the third
• Biggest gains on complex, compute-bound kernels where surrogate learning is most valuable
• Essentially tied on simpler kernels (within measurement noise)

Tuning Time Results

Kernel	Standard DE	DE-Surrogate	Speedup
MatMul-1024	243.9s	330.5s	0.74×
GELU-1M	471.6s	333.6s	1.41× ⚡
FusedReLUAdd-1M	497.0s	345.5s	1.44× ⚡
TOTAL	1212.5s (20.2 min)	1009.6s (16.8 min)	1.20× ⚡

1.20× faster in total wall-clock time despite evaluating the same number of configs.

Key Insights

1. Surrogate learning pays off: On complex kernels (MatMul), the Random Forest learns which parameter combinations matter, finding 15%
   better configs even though tuning takes longer on that specific kernel.
2. Avoids pathological configs: Learns to avoid configs that timeout or fail compilation (e.g., block_size > 65536), leading to overall
   speedup.
3. Robustness: Never performs catastrophically worse than standard DE—worst case is +0.8% slower on FusedReLU, which is within noise
   margin.
4. Sample efficiency: Generates 3× more candidate configs per generation but only evaluates the top 1/3 as predicted by the surrogate,
   efficiently exploring larger search spaces.

How It Works

Standard DE generates N candidates → evaluates ALL N.

DE-Surrogate generates 3×N candidates → Random Forest predicts performance → evaluates only top N.

This allows exploring more of the search space while learning kernel-specific patterns like "block_size=128 with num_warps=8 is fast
for matmul" without wasting evaluations.

[jansel]

How does it compare to PatternSearch?

[jansel]

#1095 might be helpful for evaluation.

[FranciscoThiesen]

@jansel this is what I've got (I've trimmed the X-axis, since it converged somewhat fast).

Detailed Results by Kernel

MatMul-1024 (Compute-Bound)

Algorithm	Time (s)	Best (ms)	Evaluations
DifferentialEvolution	388.6	0.01802	1600
DE-Surrogate	316.1	0.01747 (-3.1%)	1640
PatternSearch	275.4	0.01802	1804

Winner: DE-Surrogate - Best performance (0.01747ms) and faster than standard DE

GELU-1M (Bandwidth-Bound)

Algorithm	Time (s)	Best (ms)	Evaluations
DifferentialEvolution	548.9	0.00656	1600
DE-Surrogate	410.2	0.00653 (-0.5%)	1640
PatternSearch	168.5	0.00691 (+5.8%)	624

Winner: DE-Surrogate - Best performance and 25% faster than standard DE

FusedReLUAdd-1M (Memory-Bound)

Algorithm	Time (s)	Best (ms)	Evaluations
DifferentialEvolution	553.0	0.00643	1600
DE-Surrogate	439.0	0.00643	1640
PatternSearch	162.1	0.00640	788

Winner: 3-way tie - All algorithms found essentially the same optimum (0.0064ms)

[FranciscoThiesen]

So to summarize, DE-Surrogate is consistently faster and better than DE.

PatternSearch can be faster and offer a better balance when dealing with simpler kernels for now, but I can definitely add some early-stopping criteria to DE as well if you think that is valuable. @jansel

[jansel]

Looks like tests/lints are failing.

[jansel]

Lints/tests still failing, other than that looks good.

[jansel]

Lints still failing?

You can run:

./lint.sh install
./lint.sh

[FranciscoThiesen]

Lints still failing?

You can run:

./lint.sh install
./lint.sh

Fixed it, sorry for the back and forth

[FranciscoThiesen]

All green, ready to merge!