Skip to content

feat(julia-ci): record the timings this workflow already reads, and stop the fallback reading as transient - #20

Merged
sotashimozono merged 1 commit into
mainfrom
fix/19-julia-ci-timings-producer
Jul 30, 2026
Merged

feat(julia-ci): record the timings this workflow already reads, and stop the fallback reading as transient#20
sotashimozono merged 1 commit into
mainfrom
fix/19-julia-ci-timings-producer

Conversation

@sotashimozono

Copy link
Copy Markdown
Member

julia-ci.yml reads the ci-timings orphan branch but contains no job that writes it, so the
timings-informed LPT planning it advertises can never activate for a caller.

The header says:

timings-informed LPT planning kicks in automatically when a ci-timings orphan branch with
timings.tsv exists (else deterministic round-robin).

and the plan job implements the read (lines 60–67). Nothing in the file produces timings.tsv.
The test job does not upload per-shard timings and there is no record-timings job.

Measured, 2026-07-30

caller shards ci-timings branch
QAtlasHub/AbstractQAtlas.jl 4 absent
QAtlasHub/DataVault.jl 1 absent
QAtlasHub/ParamIO.jl 1 absent
lab-sotashimozono/ParaLinearAlgebra.jl (inline CI, has record-timings) 8 present, updated 2026-07-30T03:33:30Z

shards: 1 needs no timings, so DataVault and ParamIO are unaffected. AbstractQAtlas is not: it
fans out four ways and its planner has been round-robin on every run since adoption, with no path to
ever becoming anything else. The failure is silent — round-robin is a legitimate documented fallback,
so the log line "no ci-timings branch yet — round-robin fallback" looks like a first-run state
forever rather than a permanent one.

Fix

Port the producer half from ParaLinearAlgebra.jl's inline CI.yml (jobs test upload +
record-timings, lines 110–160):

  • test uploads test/.ci-out/timings-*.tsv as an artifact per shard;
  • a record-timings job downloads them, merges into the existing timings.tsv (latest value wins),
    and force-pushes the ci-timings orphan branch. Needs contents: write.

Gate it on inputs.shards > 1 so unsharded callers pay nothing, and on github.ref == 'refs/heads/main'
so PR runs do not race each other for the branch.

Why this blocks a larger migration

lab-sotashimozono/.github has no julia-ci, and 45 repos across the three owners still carry an
inline CI.yml (6016 lines total) that this reusable is meant to absorb — see
lab-sotashimozono/.github#15. ParaLinearAlgebra's inline version does record timings, so adopting
the reusable as it stands would be a regression for it and for every other sharded repo. Fixing the
producer here first, then porting, keeps the known defect from being copied into a second hub.

Not a gap

The compat job (minimum LTS + latest) has no counterpart here, but it does not need one:
AbstractQAtlas.jl already expresses it as a second call with julia-version: "1.10", which is the
better factoring. Only the timings producer is missing.


What this PR does

test step sets CI_EMIT_TIMINGS / CI_TIMINGS_DIR (push:main and shards > 1 only) and uploads timings-*.tsv per shard
record-timings (new) merges the artifacts into the ci-timings orphan branch, latest value wins
plan when shards > 1 and no timing data loaded, warns that round-robin is permanent, not first-run
contract header names CI_EMIT_TIMINGS / CI_TIMINGS_DIR and the contents: write the caller must grant

Env names are repo-agnostic on purpose. ParaLinearAlgebra's inline emit is gated on PLA_EMIT and
writes to PLA_CIOUT_DIR — names a reusable cannot set — so the neutral pair becomes part of the
documented contract alongside CI_CASES.

Failure behaviour, chosen deliberately

  • record-timings is continue-on-error. ParaLinearAlgebra's inline version has already produced a
    failure verdict on main where all eight shards and the aggregate passed and only the timings
    push failed. A bookkeeping job that reddens a green suite teaches people to ignore red.
  • 0 rows warns and leaves the branch alone. A suite that has not implemented the emit is a valid
    state. Pushing an empty timings.tsv would be strictly worse: plan would load zero rows and
    report success.
  • The signal that replaces the silence lives in plan, which is required and runs every time. It now
    distinguishes "no data yet" from "no data, ever" — the original line read the same on run 1 and run
    500, which is why this went unnoticed across every caller.

Purely additive (84 insertions, 0 deletions). shards: 1 callers see no change.

Adoption note

lab-sotashimozono/.github has no julia-ci, and 45 repos across the three owners still carry an
inline CI.yml (6016 lines) this is meant to absorb — lab-sotashimozono/.github#15. Fixing the
producer here first keeps the defect from being copied into a second hub when that port happens.

Closes #19

…top the fallback reading as transient

`julia-ci.yml` loaded the `ci-timings` orphan branch and planned LPT bin-packing from it, but nothing
in the file ever WROTE that branch. MEASURED 2026-07-30: none of the three consumer callers has a
`ci-timings` branch, while ParaLinearAlgebra — which carries the producer inline — has one refreshed
the same day. AbstractQAtlas runs `shards: 4`, so its planner has been round-robin on every run since
adoption with no path to becoming anything else.

The producer half is ported from ParaLinearAlgebra's inline CI: the test step emits under
`CI_TIMINGS_DIR` when `CI_EMIT_TIMINGS=1`, each shard uploads its TSV, and `record-timings` merges
them into the branch (latest value wins). Both env vars are repo-agnostic, matching `CI_CASES` —
ParaLinearAlgebra's own names are `PLA_EMIT`/`PLA_CIOUT_DIR`, which a reusable cannot know, so the
contract header now names the neutral pair as part of the sharding contract.

Two deliberate choices about failure:

`record-timings` is `continue-on-error`. A bookkeeping job that reddens a run whose every test passed
trains people to ignore red — ParaLinearAlgebra has had exactly that, a `failure` verdict on main
where all eight shards and the aggregate were green.

0 collected rows is a WARNING, not an error, and the branch is left untouched. A suite that has not
implemented the emit is a valid state and must not be punished; pushing an empty `timings.tsv` would
be worse, because `plan` would then load zero rows and look like it worked.

What replaces that silence is a warning in `plan`, which runs on every job and is required: when
`shards > 1` and no timing data loaded, it now says the round-robin fallback is PERMANENT unless the
suite emits, instead of printing a line that reads identically on run 1 and run 500. That was the
actual defect — the fallback was honest about what it was doing and silent about the fact that it
would never stop.

Purely additive: 84 insertions, 0 deletions. `shards: 1` callers are unaffected (the producer is
gated on `shards > 1` and push:main).

Closes #19
@sotashimozono
sotashimozono merged commit 56da8a0 into main Jul 30, 2026
1 check passed
sotashimozono added a commit to QAtlasHub/TestShards.jl that referenced this pull request Jul 30, 2026
`v1` is a moving tag, not a rotting pin — but it is a second pointer pushed by hand, and it
is one commit behind `main` today, which is the failure it invites: a fix merges and reaches
nobody. The hub's own `uses:` examples already say `@main`, and all 166 references to the
lab-sotashimozono hub are `@main`; this brings both owners onto one convention.

Blast radius is exactly QAtlasHub/.github#20 (julia-ci.yml, +84/-0, purely additive, gated on
shards>1 and push:main) — nothing else differs between v1 and main.

Files: AutoMerge.yml, AutoRegister.yml, CleanupPreview.yml, CompatHelper.yml, DocsPreview.yml, Documentation.yml, FormatCheck.yml, PRLabeler.yml, TagBot.yml, VersionCheck.yml, release-drafter.yml

Refs lab-sotashimozono/.github#15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

julia-ci.yml reads the ci-timings branch but nothing writes it — LPT planning has never activated

1 participant