Experiment: See if WSD models are less amenable to SFT

[dlwh]

Description

Tootsie 8b (as of #916 ) seems to not be as good as llama 3 or olmo in terms of SFT-ability. We don't understand why. (We're getting closer, but still not there.)

We'd like to see if WSD is the issue, and whether we can reproduce these problems at the 1.4B scale.

We'll train a few 1.4B for a trillion tokens or so and see:

• High LR WSD (similar to DCLM and tootsie)
• Cosine (~olmo)
• High LR Cosine

Then we'll see norms and stuff and whether there's a difference in SFTability.

Hypothesis or Goal

Links

(Delete any that aren't applicable)

• WandB Report: (link)
• Data Browser: (link)
• Experiment JSON: (link)
• (etc.)

Results

(What did you find, including relevant evaluation metrics, etc.)

[Helw150]

The pretraining runs for the 1.4B models is done which confirms the initial suspicion that WSD leads to higher norms (even for decayed weights, but especially for non-decayed weights) than Cosine schedule.

Both runs have roughly equivalent training loss by the end (this is all single epoch so train-loss is a reasonable eval metric).

Some indicators that the higher norm might lead to worse eval scores, but the SFT runs are going now so that'll give us more signal!

[Helw150]

Train loss during SFT does not seem to be significantly across these runs. Kicking off AlpacaEval

[Helw150]

Writeup here: https://wandb.ai/marin-community/marin/reports/How-does-Learning-Rate-Schedule-Impact-SFT---VmlldzoxMjgyNDkyOQ

[dlwh]

This experiment has been archived to reduce clutter and preserve velocity (see PR #1999). It is last available in the archive/20251114 tag at https://github.com/marin-community/marin/tree/archive/20251114/experiments/exp950_sft_amenability.py. Please open an issue if you need help unarchiving it.