feat(locomo): sharded full-run kit + merged summarize (#184) - #185
Merged
Conversation
fazpu
enabled auto-merge (squash)
July 30, 2026 12:11
Coverage report
This PR does not seem to contain any modification to coverable code. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #184.
What
Per-sample deployment isolation makes LoCoMo conversations embarrassingly parallel — N hosts can run disjoint sample subsets under the same protocol pin. The first full publication run took ~20h sequentially (~2h/conversation); sharded across hosts it completes in the wall-clock of the slowest conversation.
Merged summarize (harness).
summarizeaccepts--runmultiple times. Single--run: output byte-identical (a wrap-serializer omits the new fields for unmerged summaries; older stored summaries still parse). Multiple runs: validates identicalprotocol_name/protocol_fingerprint/tier/dataset_sha256/manifest_sha256/item_ids_sha256and pairwise-disjoint recorded samples (configsample_idslist the full manifest on every shard by design), then materializes a temporary combined run dir and scores it with the unchanged single-run code path — recompute-from-items by construction, never summary-summing. Addsmerged_run_count+missing_sample_ids(manifest samples with no records).Sharding ops kit (
benchmarks/locomo/sharding/): balanced shard planner (largest-first packing, stdlib, tested), per-host driver encoding the operational lessons from the first full run (worker scalingextract=3/normalize=6/embed-claim=2on every compose up, true-drain waits default 6h — no 80-min caps, projections after ingest, forensic pg dump before each wipe, dead-letter abort, resumable per-sample status), collector + merge script, operator README with env-injected secrets. Optional — any N ssh-able hosts work.No protocol identity, prompts, budgets, or fingerprint content touched; answer/judge runner logic untouched.
Reviews
Grok-4.5 adversarial review: APPROVE WITH NOTES — verified single-run byte-identity (CLI stdout and model_dump), record-based disjointness (specifically proved config sample_ids are not used), merge==reference-single-run score invariant, true-drain semantics incl. failed/dead-letter handling, scale flags on every up, no secrets. Notes (non-blocking): identity-field test covers 1/6 fields explicitly, make_shards is publication-centric (tier-manifest filtering as follow-up), forensic-dump retry gap on resume, bash>=4 requirement.
Claude (final review): verified serializer omission, temp-dir merge construction, ran the suite (68 passed) + ruff format --check.
I have read and agree to the RememberStack Contributor License Agreement v1.0.