Skip to content

Cross Dataset Comparison

Jump to bottom
Dave Graham edited this page Mar 17, 2026 · 11 revisions

Cross-Dataset Comparison — Deep Dive

Autonomous agent runs on the same hardware (Apple M5 Max, 64 GB) reveal that optimal model configuration is dataset-dependent. The agent independently discovers different architectures, learning rates, and regularization strategies for each dataset.

Cross-Dataset Comparison

Results Summary

Climbmix (web crawl) FineWeb-Edu (educational)
Best val_bpb 1.335 1.295
Baseline 1.353 1.409
Improvement -0.018 (1.3%) -0.114 (8.1%)
Experiments 81 101
Keep rate 11.1% 18.8%
Crash rate 3.7% 2.0%

Architecture Divergence

The most striking finding: the same agent on the same hardware converges to completely different model architectures depending on the training data.

Parameter Default Climbmix Optimal FineWeb-Edu Optimal Direction
ASPECT_RATIO 64 64 (unchanged) 32 (halved) ←
Peak Memory 26.1 GB 26.1 GB 15.5 GB ←
tok/sec ~30K ~31K ~62K →
Training Steps ~288 ~289 ~572 →

FineWeb-Edu benefits from a smaller model that runs 2× more gradient steps in the same 5-minute budget. Climbmix's diverse web crawl text needs the full model width.

Why the Architecture Matters

The autoresearch training loop is time-budgeted (5 minutes). This creates a fundamental tradeoff:

  • Wider model (AR=64): More parameters, richer representations, but fewer gradient steps per budget
  • Narrower model (AR=32): Fewer parameters, but 2× more steps — the model sees more gradient updates

Climbmix data (raw web crawl) has high token diversity and noisy patterns that benefit from wider representations. FineWeb-Edu (educationally curated) has more consistent structure — a narrower model can learn the patterns efficiently if given enough gradient steps.

Hyperparameter Divergence

Every tunable hyperparameter diverged between datasets, with some moving in opposite directions:

Parameter Default Climbmix FineWeb-Edu Divergence
MATRIX_LR 0.04 0.06 (1.50×) 0.053 (1.33×) Both ↑, different magnitude
SCALAR_LR 0.5 0.5 (1.00×) 0.3 (0.60×) Climbmix keeps default, FineWeb ↓
WEIGHT_DECAY 0.2 0.02 (0.10×) 0.12 (0.60×) Both ↓, 6× difference
WARMDOWN_RATIO 0.5 0.4 (0.80×) 0.47 (0.94×) Both ↓, different degree
UNEMBEDDING_LR 0.004 0.0041 (1.03×) 0.0028 (0.70×) Opposite directions
EMBEDDING_LR 0.6 0.585 (0.98×) 0.59 (0.98×) Both slightly ↓

Key Observations

WEIGHT_DECAY shows a 6× difference (0.02 vs 0.12). Climbmix aggressively reduced regularization (to 10% of default), while FineWeb-Edu only moderately reduced it (to 60% of default). This suggests educational text has more consistent patterns that benefit from regularization preventing overfitting, while noisy web crawl data needs the model to have maximum freedom to fit diverse distributions.

UNEMBEDDING_LR moves in opposite directions — up to 0.0041 on climbmix, down to 0.0028 on FineWeb-Edu. The unembedding layer maps hidden states to token predictions. Climbmix's diverse vocabulary benefits from a slightly faster-learning output layer, while FineWeb-Edu's more structured text prefers a more conservative one. Notably, the FineWeb-Edu UNEMBEDDING_LR change was responsible for the dramatic late breakthrough (exp89: -0.047 val_bpb from a single 0.0001 change).

SCALAR_LR stayed at default for climbmix but dropped 40% for FineWeb-Edu. The scalar learning rate controls LayerNorm parameters. Educational text's more regular structure may need gentler normalization updates.

Improvement Dynamics

Metric Climbmix FineWeb-Edu
Biggest single gain MATRIX_LR 0.04→0.06 (-0.013) AR 40→32 (-0.036)
Late breakthrough? No (diminishing returns) Yes — exp89 broke a 20-experiment plateau
Key lever Learning rate tuning Architecture change + late LR discovery
Optimization character Smooth, incremental Two sharp phase transitions

Climbmix: Smooth Optimization

The climbmix run followed a textbook optimization curve — each kept experiment improved by a smaller amount than the previous one. The agent efficiently narrowed in on optimal learning rates and regularization without any architectural changes. After ~50 experiments, improvements became sub-0.001 and the agent was clearly at a local optimum.

FineWeb-Edu: Phase Transitions

The FineWeb-Edu run had a dramatically different character with two sharp phase transitions:

  1. Phase transition #1 (exp34): ASPECT_RATIO 40→32 dropped val_bpb by 0.036 — the largest single improvement in any run. This wasn't gradual; the agent tried AR=48 (-0.015), then AR=40 (-0.001), then AR=32 (-0.036). The jump from AR=40 to AR=32 was disproportionately large.

  2. Phase transition #2 (exp89): After 20 experiments stuck at ~1.342, a tiny UNEMBEDDING_LR change (0.003→0.0029) caused a 0.047 improvement — breaking through the plateau to 1.295. The next experiment (0.0029→0.0028) only gained 0.0002, confirming this was a sharp nonlinearity, not a smooth gradient.

These phase transitions suggest the FineWeb-Edu loss landscape has discrete basins separated by narrow ridges in hyperparameter space.

Best Configurations Side-by-Side

Climbmix (Mar 16, 81 experiments)

ASPECT_RATIO = 64          # Default
MATRIX_LR = 0.06           # ↑ 1.5×
EMBEDDING_LR = 0.585       # ↓ slightly
UNEMBEDDING_LR = 0.0041    # ↑ slightly
SCALAR_LR = 0.5            # Default
WEIGHT_DECAY = 0.02        # ↓ 10×
WARMDOWN_RATIO = 0.4       # ↓ 0.8×
# Peak memory: 26.1 GB | Steps: 289 | MFU: 28.0%

FineWeb-Edu (Mar 17, 101 experiments)

ASPECT_RATIO = 32          # ↓ halved
MATRIX_LR = 0.053          # ↑ 1.3×
EMBEDDING_LR = 0.59        # ↓ slightly
UNEMBEDDING_LR = 0.0028    # ↓ 0.7×
SCALAR_LR = 0.3            # ↓ 0.6×
WEIGHT_DECAY = 0.12        # ↓ 0.6×
WARMDOWN_RATIO = 0.47      # ↓ slightly
# Peak memory: 15.5 GB | Steps: 572 | MFU: 18.9%

Implications

  1. Hyperparameters found on one dataset do not transfer. Weight decay, learning rates, and even architecture differ significantly. Running the climbmix-optimal config on FineWeb-Edu (or vice versa) would leave substantial performance on the table.

  2. Architecture search is critical. The FineWeb-Edu run's biggest gain came from an architectural change that the climbmix run never needed. A fixed-architecture search would have missed 46% of the total improvement.

  3. Long runs find surprises. The FineWeb-Edu breakthrough at exp89 would have been missed in a 75-experiment run. Some hyperparameter landscapes have non-obvious features that only emerge from exhaustive search.

  4. The agent adapts its strategy. On climbmix, the agent focused on smooth LR optimization. On FineWeb-Edu, it explored architecture changes and kept probing parameters it had previously discarded. The same LLM agent develops different search strategies for different loss landscapes.

See individual run pages for full experiment logs: Climbmix (Mar 16) | FineWeb-Edu (Mar 17)

Clone this wiki locally