Proposal: a Co-Lab on why temperature 0 does not give the same answer twice

[Shakhtar-Sankur]

Hi! I'd like to contribute a Co-Lab, and wanted to check the fit before opening a PR.

The Collective Communication chapter in Vol. II says that reduction order "introduces a second, numerical contract": floating-point addition is not associative, so different orders give different low-order bits. I couldn't find a lab that follows this into inference, where it causes a problem every LLM server has. The same prompt at temperature 0 can give different answers, because the server batches each request with different neighbours and fast kernels choose their reduction order by batch size. Thinking Machines Lab sent one prompt 1,000 times to Qwen3-235B at temperature 0 and got 80 different completions. With batch-invariant kernels, all 1,000 were identical (He et al., "Defeating Nondeterminism in LLM Inference", September 2025).

I built a prototype called "The Same-Bits Contract". It follows the Predict, Discover, Explain pattern, with a prediction lock in each part:

A. One sum, five orders. The same 4,096 float32 numbers, summed in five different orders, give 4 different results.
B. The batch picks the kernel. One request's 128 outputs, served alone and then in a batch of 32, through a split-K matmul whose split depends on batch size. In float16, 113 of the 128 outputs change.
C. From bits to words. A small seeded language model decodes greedily for 64 users. 7 of them get different text only because of batching, and students see the token where each one diverges.
D. Batch-invariant kernels. A fixed split brings that to 0 of 64, and students count the extra reduction work it costs.
E. Integer arithmetic. int8 weights and activations with exact int32 sums also give 0 of 64, while keeping the adaptive split. The cost is accuracy: 94.5% token agreement with the float model.

The synthesis asks which approach to choose for an RL trainer that rescores its sampler's tokens, and for an audit that has to reproduce a result on different hardware.

All the numbers are measured rather than modelled. numpy's float16 and float32 are real IEEE 754 formats, rounded after every operation, and each kernel spells out its reduction order. It only needs numpy, so it should run under Pyodide, and 8 tests check every number the lab shows.

Prototype (marimo): https://github.com/Shakhtar-Sankur/samebit/tree/main/labs/same_bits

Two questions before I port it to the repo's structure (mlsysim experiments, DesignLedger, evidence capture, WASM bootstrap, labs/tests):

1. Would it fit better as a new part of Vol. II Lab 10 (Inference), as a part of Vol. I Lab 13 (Model Serving), or as a standalone lab?
2. Should the experiment functions go in mlsysim/labs/experiments/ like the other labs?

I'm happy to adapt it to your conventions. Thank you for the book!

[Shashank-Tripathi-07]

Hey @Shakhtar-Sankur !

Yes, the feature looks great and the idea sounds cool to me. having a colab for this sounds cool, @profvjreddi can guide you better on this and how exactly it should be executed 👍🏻

[Shakhtar-Sankur]

Thank you! To make it easier to decide how it should be executed, I've built a full draft in the Co-Lab format: https://github.com/Shakhtar-Sankur/cs249r_book/tree/same-bits-lab

The Same-Bits Contract (5 parts + synthesis, about 45 minutes):

• A · One sum, five orders: the same 4,096 float32 numbers give 4 different sums.
• B · The batch picks the kernel: one row, alone vs batched: 108–113 of 128 outputs change bits.
• C · From bits to words: up to 7 of 64 users get different temperature-0 text.
• D · Batch-invariant kernels: a fixed split makes it 0 of 64, and students measure the extra partial-sum work.
• E · Integer arithmetic: int8 with exact int32 sums also gives 0 of 64 at any batch, at 94.5% token agreement with float.
• Synthesis: students choose a determinism contract for their track's stakeholder: the RL trainer (Cloud), incident replay (RoboTaxi), reproducing a reported suggestion (iPhone), or firmware validation (Oura Ring).

It follows the lab protocol: structured predictions before results, capture_evidence/audit_evidence, a report, and a Design Ledger save. All the math is in a new mlsysim.labs.experiments.same_bits_experiments module, with tests.

The test suite fixes 17 labs per volume, so I put it in labs/proposed/ rather than taking a chapter slot. conftest.py adds proposed labs to ALL_LABS, so every per-lab check runs on it, and the catalog, track arcs and site are untouched.

@profvjreddi, where would you like it to sit: as a follow-on to Vol II Lab 10 (Inference at Scale), or in Vol I Lab 12 (Benchmarking)? I'm happy to reshape it to fit either, and to open the PR once you've decided.

[Shakhtar-Sankur]

Thank you, @Shashank-Tripathi-07! @profvjreddi, I’d be glad to build it to your guidance. A working prototype with 8 tests is here: https://github.com/Shakhtar-Sankur/samebit/tree/main/labs/same_bits. Should it follow the Co-Labs format exactly, and where in the book would it fit best?