feat(julia-ci): record the timings this workflow already reads, and stop the fallback reading as transient - #20
Merged
Conversation
…top the fallback reading as transient `julia-ci.yml` loaded the `ci-timings` orphan branch and planned LPT bin-packing from it, but nothing in the file ever WROTE that branch. MEASURED 2026-07-30: none of the three consumer callers has a `ci-timings` branch, while ParaLinearAlgebra — which carries the producer inline — has one refreshed the same day. AbstractQAtlas runs `shards: 4`, so its planner has been round-robin on every run since adoption with no path to becoming anything else. The producer half is ported from ParaLinearAlgebra's inline CI: the test step emits under `CI_TIMINGS_DIR` when `CI_EMIT_TIMINGS=1`, each shard uploads its TSV, and `record-timings` merges them into the branch (latest value wins). Both env vars are repo-agnostic, matching `CI_CASES` — ParaLinearAlgebra's own names are `PLA_EMIT`/`PLA_CIOUT_DIR`, which a reusable cannot know, so the contract header now names the neutral pair as part of the sharding contract. Two deliberate choices about failure: `record-timings` is `continue-on-error`. A bookkeeping job that reddens a run whose every test passed trains people to ignore red — ParaLinearAlgebra has had exactly that, a `failure` verdict on main where all eight shards and the aggregate were green. 0 collected rows is a WARNING, not an error, and the branch is left untouched. A suite that has not implemented the emit is a valid state and must not be punished; pushing an empty `timings.tsv` would be worse, because `plan` would then load zero rows and look like it worked. What replaces that silence is a warning in `plan`, which runs on every job and is required: when `shards > 1` and no timing data loaded, it now says the round-robin fallback is PERMANENT unless the suite emits, instead of printing a line that reads identically on run 1 and run 500. That was the actual defect — the fallback was honest about what it was doing and silent about the fact that it would never stop. Purely additive: 84 insertions, 0 deletions. `shards: 1` callers are unaffected (the producer is gated on `shards > 1` and push:main). Closes #19
sotashimozono
added a commit
to QAtlasHub/TestShards.jl
that referenced
this pull request
Jul 30, 2026
`v1` is a moving tag, not a rotting pin — but it is a second pointer pushed by hand, and it is one commit behind `main` today, which is the failure it invites: a fix merges and reaches nobody. The hub's own `uses:` examples already say `@main`, and all 166 references to the lab-sotashimozono hub are `@main`; this brings both owners onto one convention. Blast radius is exactly QAtlasHub/.github#20 (julia-ci.yml, +84/-0, purely additive, gated on shards>1 and push:main) — nothing else differs between v1 and main. Files: AutoMerge.yml, AutoRegister.yml, CleanupPreview.yml, CompatHelper.yml, DocsPreview.yml, Documentation.yml, FormatCheck.yml, PRLabeler.yml, TagBot.yml, VersionCheck.yml, release-drafter.yml Refs lab-sotashimozono/.github#15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
julia-ci.ymlreads theci-timingsorphan branch but contains no job that writes it, so thetimings-informed LPT planning it advertises can never activate for a caller.
The header says:
and the
planjob implements the read (lines 60–67). Nothing in the file producestimings.tsv.The
testjob does not upload per-shard timings and there is norecord-timingsjob.Measured, 2026-07-30
shardsci-timingsbranchQAtlasHub/AbstractQAtlas.jlQAtlasHub/DataVault.jlQAtlasHub/ParamIO.jllab-sotashimozono/ParaLinearAlgebra.jl(inline CI, hasrecord-timings)2026-07-30T03:33:30Zshards: 1needs no timings, so DataVault and ParamIO are unaffected. AbstractQAtlas is not: itfans out four ways and its planner has been round-robin on every run since adoption, with no path to
ever becoming anything else. The failure is silent — round-robin is a legitimate documented fallback,
so the log line "no ci-timings branch yet — round-robin fallback" looks like a first-run state
forever rather than a permanent one.
Fix
Port the producer half from
ParaLinearAlgebra.jl's inlineCI.yml(jobstestupload +record-timings, lines 110–160):testuploadstest/.ci-out/timings-*.tsvas an artifact per shard;record-timingsjob downloads them, merges into the existingtimings.tsv(latest value wins),and force-pushes the
ci-timingsorphan branch. Needscontents: write.Gate it on
inputs.shards > 1so unsharded callers pay nothing, and ongithub.ref == 'refs/heads/main'so PR runs do not race each other for the branch.
Why this blocks a larger migration
lab-sotashimozono/.githubhas nojulia-ci, and 45 repos across the three owners still carry aninline
CI.yml(6016 lines total) that this reusable is meant to absorb — seelab-sotashimozono/.github#15. ParaLinearAlgebra's inline version does record timings, so adopting
the reusable as it stands would be a regression for it and for every other sharded repo. Fixing the
producer here first, then porting, keeps the known defect from being copied into a second hub.
Not a gap
The
compatjob (minimum LTS + latest) has no counterpart here, but it does not need one:AbstractQAtlas.jlalready expresses it as a second call withjulia-version: "1.10", which is thebetter factoring. Only the timings producer is missing.
What this PR does
teststepCI_EMIT_TIMINGS/CI_TIMINGS_DIR(push:main andshards > 1only) and uploadstimings-*.tsvper shardrecord-timings(new)ci-timingsorphan branch, latest value winsplanshards > 1and no timing data loaded, warns that round-robin is permanent, not first-runCI_EMIT_TIMINGS/CI_TIMINGS_DIRand thecontents: writethe caller must grantEnv names are repo-agnostic on purpose. ParaLinearAlgebra's inline emit is gated on
PLA_EMITandwrites to
PLA_CIOUT_DIR— names a reusable cannot set — so the neutral pair becomes part of thedocumented contract alongside
CI_CASES.Failure behaviour, chosen deliberately
record-timingsiscontinue-on-error. ParaLinearAlgebra's inline version has already produced afailureverdict onmainwhere all eight shards and the aggregate passed and only the timingspush failed. A bookkeeping job that reddens a green suite teaches people to ignore red.
state. Pushing an empty
timings.tsvwould be strictly worse:planwould load zero rows andreport success.
plan, which is required and runs every time. It nowdistinguishes "no data yet" from "no data, ever" — the original line read the same on run 1 and run
500, which is why this went unnoticed across every caller.
Purely additive (84 insertions, 0 deletions).
shards: 1callers see no change.Adoption note
lab-sotashimozono/.githubhas nojulia-ci, and 45 repos across the three owners still carry aninline
CI.yml(6016 lines) this is meant to absorb — lab-sotashimozono/.github#15. Fixing theproducer here first keeps the defect from being copied into a second hub when that port happens.
Closes #19