Skip to content

Record: 30ep Cosine TTT on LeakyReLU² stack (3-seed mean val_bpb=1.0781)#672

Open
andrewbaggio1 wants to merge 1 commit intoopenai:mainfrom
andrewbaggio1:submission/record-cosine-ttt-30ep-8xH100
Open

Record: 30ep Cosine TTT on LeakyReLU² stack (3-seed mean val_bpb=1.0781)#672
andrewbaggio1 wants to merge 1 commit intoopenai:mainfrom
andrewbaggio1:submission/record-cosine-ttt-30ep-8xH100

Conversation

@andrewbaggio1
Copy link

@andrewbaggio1 andrewbaggio1 commented Mar 25, 2026

Summary

3-seed mean val_bpb: 1.0781 (std=0.0041) | 15.62 MB artifact | 8xH100 SXM

Single change from PR #518: TTT_EPOCHS=30. All architecture identical.

Results (8xH100 SXM)

Seed Sliding BPB (s64) Artifact
1337 1.0743 15.62 MB
42 1.0774 15.62 MB
7 1.0825 15.62 MB
Mean ± Std 1.0781 ± 0.0041

vs. Verified SOTA

Submission Mean BPB
Ours 1.0781
PR #549 (verified SOTA) 1.1194
Improvement -0.041

Timing

  • Training: 600s (10 min cap)
  • TTT (30 epochs cosine): 494s
  • Sliding eval (stride=64): 96s
  • Total eval: 590s (under 10 min)

Architecture

PR #518's full stack: 11L LeakyReLU(0.5)², d=512, 4 KV GQA, MLP 3x, BigramHash(2048), SmearGate, XSA4, Partial RoPE, LN Scale, EMA, SWA, Late QAT, OrthoInit, VE128. Int6+zstd-22.

Run command

SEED=1337 torchrun --standalone --nproc_per_node=8 train_gpt.py

Credits

PR #518, PR #481 (mrdavtan), PR #442 (sjp611), PR #398 (felipe-parodi)

Test plan

  • train_gpt.py compiles
  • 3 seeds verified, all artifacts < 16 MB
  • Mean beats verified SOTA by 0.041 BPB
  • Training < 10 min, eval < 10 min on 8xH100
  • PR only adds files to one new folder

🤖 Generated with Claude Code

@andrewbaggio1 @claude
Record: 30ep Cosine TTT on LeakyReLU² stack (3-seed mean val_bpb=1.0781)
3f9fa54 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dhruvjatkar pushed a commit to dhruvjatkar/parameter-golf that referenced this pull request Mar 25, 2026
@claude
Add PR openai#672 (Cosine TTT, TTT_EPOCHS=30, 1.0781 BPB) as new base…
2b44b5e 
…line reference

Fetched train_gpt.py verbatim from upstream openai/parameter-golf PR openai#672
which achieves 1.0781 BPB (3-seed mean, std=0.0041) using TTT_EPOCHS=30
with cosine TTT schedule. This replaces 1.1194 as the baseline to beat.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@dhruvjatkar dhruvjatkar mentioned this pull request Mar 25, 2026
Closed
3 tasks
dhruvjatkar pushed a commit to dhruvjatkar/parameter-golf that referenced this pull request Mar 25, 2026
@claude
Update AGENTS.md: new baseline target, single-agent protocol
b7bf52e 
- Target to beat: 1.0781 BPB (PR openai#672, TTT_EPOCHS=30 Cosine TTT)
- Add single-agent protocol section
- Mark crontab auto-submitter as non-functional
- Add operational lessons from March 2026
- Update preferred source script to PR672 baseline

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@dhruvjatkar dhruvjatkar mentioned this pull request Mar 25, 2026
Closed
dhruvjatkar pushed a commit to dhruvjatkar/parameter-golf that referenced this pull request Mar 25, 2026
@claude
Update CLAUDE.md: target 1.0781 BPB, single-agent protocol, operation…
ab32200 
…al lessons

- New target: 1.0781 BPB (PR openai#672, TTT_EPOCHS=30 Cosine TTT)
- Merged SOTA kept as 1.1194 for context
- Add single-agent protocol (one agent on cluster at a time)
- Add operational lessons from March 2026
- Mark crontab auto-submitter as non-functional
- Update milestones relative to 1.0781
- Update preferred source script to PR672 baseline

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@dhruvjatkar dhruvjatkar mentioned this pull request Mar 25, 2026
Closed
dhruvjatkar pushed a commit to dhruvjatkar/parameter-golf that referenced this pull request Mar 25, 2026
@claude
Update plan.md: reset baseline to 1.0781 BPB and reprioritize directions
87c218b 
PR openai#672 maxes TTT at 30 epochs (590s/600s eval budget), so all future
improvements must be orthogonal to TTT. This update:
- Sets 1.0781 BPB (PR openai#672) as the new target to beat
- Reorders Top 8 directions: XSA-all confirmed at #1, Full GPTQ #2,
  SwiGLU #3, Muon-VS #4, aggressive quant #5, MASA openai#6,
  depth recurrence openai#7 with int6 risk warning, AdEMAMix openai#8
- Deprioritizes TTT-related directions already exploited by PR openai#672
- Collapses ~1000 lines of stale Round 0-3.9 session logs into a
  concise historical summary
- Removes resolved blockers (flash_attn, SSH hangs, local runtime)
- Adds fresh Round 1 section with 5 submitted experiments

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@dhruvjatkar dhruvjatkar mentioned this pull request Mar 25, 2026
Closed
5 tasks
dhruvjatkar pushed a commit to dhruvjatkar/parameter-golf that referenced this pull request Mar 25, 2026
@claude
Update CLAUDE.md: target 1.0781 BPB, single-agent protocol, operation…
b78d909 
…al lessons

- New target: 1.0781 BPB (PR openai#672, TTT_EPOCHS=30 Cosine TTT)
- Merged SOTA kept as 1.1194 for context
- Add single-agent protocol (one agent on cluster at a time)
- Add operational lessons from March 2026
- Mark crontab auto-submitter as non-functional
- Update milestones relative to 1.0781
- Update preferred source script to PR672 baseline

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dhruvjatkar pushed a commit to dhruvjatkar/parameter-golf that referenced this pull request Mar 25, 2026
@claude
Update AGENTS.md: new baseline target, single-agent protocol
013e730 
- Target to beat: 1.0781 BPB (PR openai#672, TTT_EPOCHS=30 Cosine TTT)
- Add single-agent protocol section
- Mark crontab auto-submitter as non-functional
- Add operational lessons from March 2026
- Update preferred source script to PR672 baseline

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dhruvjatkar pushed a commit to dhruvjatkar/parameter-golf that referenced this pull request Mar 25, 2026
@claude
Update plan.md: reset baseline to 1.0781 BPB and reprioritize directions
bfc3cfd 
PR openai#672 maxes TTT at 30 epochs (590s/600s eval budget), so all future
improvements must be orthogonal to TTT. This update:
- Sets 1.0781 BPB (PR openai#672) as the new target to beat
- Reorders Top 8 directions: XSA-all confirmed at #1, Full GPTQ #2,
  SwiGLU #3, Muon-VS #4, aggressive quant #5, MASA openai#6,
  depth recurrence openai#7 with int6 risk warning, AdEMAMix openai#8
- Deprioritizes TTT-related directions already exploited by PR openai#672
- Collapses ~1000 lines of stale Round 0-3.9 session logs into a
  concise historical summary
- Removes resolved blockers (flash_attn, SSH hangs, local runtime)
- Adds fresh Round 1 section with 5 submitted experiments

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dhruvjatkar pushed a commit to dhruvjatkar/parameter-golf that referenced this pull request Mar 25, 2026
@claude
Add PR openai#672 (Cosine TTT, TTT_EPOCHS=30, 1.0781 BPB) as new base…
5c7db57 
…line reference

Fetched train_gpt.py verbatim from upstream openai/parameter-golf PR openai#672
which achieves 1.0781 BPB (3-seed mean, std=0.0041) using TTT_EPOCHS=30
with cosine TTT schedule. This replaces 1.1194 as the baseline to beat.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@andrewbaggio1 andrewbaggio1 mentioned this pull request Mar 25, 2026
Closed
4 tasks
xexyz added a commit to xexyz/parameter-golf that referenced this pull request Mar 25, 2026
@xexyz
Add PR openai#414 + 30-epoch cosine TTT submission (1.0988 BPB)
d71f5da 
30-epoch cosine pre-eval Test-Time Training on PR openai#414 consensus stack.
Adapts quantized model on validation data before sliding-window eval.

- Pre-TTT post-quant: 1.1594 BPB
- Post-TTT sliding (stride=64): 1.0988 BPB
- Total artifact: 15,900,191 bytes (under 16MB)
- 5434 training steps + 30ep TTT + sliding eval on 8xH100

Built on PR openai#414 by @signalrush. TTT recipe from PR openai#518/@sofiabod, PR openai#672/@andrewbaggio1.
@xexyz xexyz mentioned this pull request Mar 25, 2026
Open
Bharath-970 added a commit to Bharath-970/parameter-golf that referenced this pull request Mar 25, 2026
@Bharath-970
Replace LoRA TTT with 30ep cosine full-model TTT in 16L XSA-all submi…
4fb6969 
…ssion

Swap score-first LoRA TTT for the simpler and more effective cosine TTT
approach from PR openai#672 (1.0781 BPB): fine-tune all model weights on val
data for 30 epochs with cosine LR decay and per-layer LR groups (3x
MLP-out, 0.5x MLP-in), followed by sliding-window stride=64 eval.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@andrewbaggio1