Downstream results: extra seeds, single-epoch check, appendix tables - #9
Merged
sanderland merged 1 commit intoAug 2, 2026
Conversation
…ndix tables Ports what was left on the results branch onto this layout. The round-3 numbers are already here; these are the additions. Extra seeds. bnd_wpd at 8 shards now has seeds 0-5 rather than 0-2. The mean moves from 0.879999 to 0.879932, a shift of 0.10 standard deviations, and the standard deviation grows from 0.00049 to 0.00067, still an order of magnitude below the 0.0053 difference against plain. Three seeds is enough, and the appendix table says so as the justification for reporting the other arms at three. Single-epoch check. At 8 shards every arm makes 3.4 to 3.6 passes over the same 2 GB, which suppresses the mechanism better compression is supposed to exploit, so plain and bnd_wpd were rerun at 32 shards in their own NANOCHAT_BASE and OUT. Both improve by about 0.036 with four times the unique text, and the difference between them is unchanged: +0.00532 to +0.00497. results_n32.tsv. collect_results.py: the tag regex did not allow a variant suffix, so every plain_bpe_d12_s0_n32 style run parsed as its own single-run arm instead of grouping. The suffix is now optional and recorded in a `variant` column. make_tex_tables.py writes downstream_appendix.tex alongside the main tables, both generated from the TSVs rather than typed. Also carries the section prose and the limitations paragraph, which had no home left after the old acl_latex.tex was dropped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sanderland
merged commit Aug 2, 2026
162a019
into
sanderland:claude/fineweb-space-neighbors-k10ufw
1 check passed
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the remaining downstream work onto the current layout. The round-3 numbers were already here; these are the additions.
Do more seeds change it? No.
bnd_wpdat 8 shards, seeds 0-5 instead of 0-2:The mean moves 0.10 standard deviations. The sd grows, so three seeds was mildly optimistic about spread, but 0.00067 is still an order of magnitude below the 0.0053 difference against
plain. The appendix table states this as the justification for reporting the other arms at three seeds.Does it survive single-epoch training? Yes.
At 8 shards every arm makes 3.4 to 3.6 passes over the same 2 GB, which suppresses the mechanism better compression is meant to exploit.
plainandbnd_wpdwere rerun at 32 shards, in their ownNANOCHAT_BASEandOUTso the 8-shard setup was not altered (the loader uses every shard except the last, so adding shards to the shared directory would silently change what an 8-shard run trains on).Both arms gain about 0.036 from four times the unique text, and the gap is unchanged, a 6% reduction well inside the seed spread. The single-epoch comparison is tighter (t = 27.3 against 16.7), since seed variance shrinks when nothing is repeated.
One bug
collect_results.py's tag regex did not allow a variant suffix, so everyplain_bpe_d12_s0_n32style run parsed as its own single-run arm rather than grouping intoplainandbnd_wpd. Caught because the summary printed six one-run arms instead of two. The suffix is optional now and recorded in avariantcolumn.Also
make_tex_tables.pywritesdownstream_appendix.texbeside the main tables. Both appendix tables are generated from the TSVs; no number is typed.downstream_section.texanddownstream_limitations.tex, which had no home left after the oldacl_latex.texwas dropped. Paths adjusted to this layout.🤖 Generated with Claude Code