Skip to content

feat(profiling): ingest --axis compile — warm-pin drift detection - #220

Merged
Jammy2211 merged 5 commits into
mainfrom
feature/compile-ingest-pins
Aug 10, 2026
Merged

feat(profiling): ingest --axis compile — warm-pin drift detection#220
Jammy2211 merged 5 commits into
mainfrom
feature/compile-ingest-pins

Conversation

@Jammy2211

Copy link
Copy Markdown
Contributor

Stacked on #219 — base is feature/compile-axis-campaign-coverage, not main,
because this needs that PR's load_compile_corpus / compile_tier_of. Merge #219
first and this retargets to main cleanly. Review the [two-commit diff]
(feature/compile-axis-campaign-coverage...feature/compile-ingest-pins).

Completes the Brain side of phase 2 (autolens_profiling#103). Pairs with
autolens_profiling#104, which owns the pins this reads.

Why

This is the surveillance the whole arc exists for. The persistent compilation cache
(117.0 s → 2.3 s CPU; 5517.8 s → 937.1 s A100) and --xla_gpu_autotune_level=0 (17×
on the pathological cold probe) are settings. A config drift or an XLA_FLAGS
clobber — the PyAutoNerves#127 failure that went undetected for two months — puts the
70-minute worst case straight back with nothing failing.

What it does

Reads the workspace's jax_compile/pins.json and reports warm rows that are
unpinned or have drifted. Every comparison happens strictly inside one
(hardware, hostname, jax_version, mixed_precision, cache_state). A jax_version
bump recompiles once by design, so it surfaces as a new unpinned key, never as
drift.

Against the real corpus: 25 pins, 0 drifted, 0 unpinned.

Two corrections that came from running it, not from designing it

1. Rows predating their pin are not drift. The first run flagged four — all
July-16 measurements that the July-28 pin had been chosen over. It was reporting
the improvement that set the pin as though it were a regression. Drift now requires a
row newer than its pin.

2. That exposed the deeper flaw, fixed in autolens_profiling#104: pins must be
sticky. With "most recent warm row wins", re-deriving pins after a cache
regression would have moved the pin onto the regressed value and every later check
would have reported all-clear forever. An existing pin now never moves without an
explicit --repin.

I would not have found either from the design alone — both only showed up against
real data.

Thresholds

Drift needs both gates: ≥ 2.0× the pin and1.0 s absolute. The ratio
alone screams about sub-second cells where 100 ms of jitter is 3×; the floor alone
misses a cheap cell degrading by an order of magnitude. Both are generous because
host load alone has produced errors in this corpus, and an alarm that cries
wolf gets ignored.

Tests

11 more (27 total in the file), including:

  • a synthetic warm-reverting-to-cold row proving the alarm does fire;
  • each comparability field proving it does not fire across the key;
  • a jax_version bump classified as a new key rather than drift;
  • both threshold gates asserted independently;
  • a guard that the Brain's mirrored key definition matches the workspace's pins.py
    — mirrored rather than imported, since importing the workspace would drag the JAX
    stack into the Brain.

Full suite: 316 passed. The 2 test_skill_install.py failures reproduce on clean
main in this environment and are untouched.

triage --axis compile still exits 5 — classification is phase 3.


Generated by Claude Code

claude added 4 commits August 10, 2026 13:12
The Profiling Agent could not see the compile-time corpus autolens_profiling
is already producing. AGENTS.md listed "JAX compilation-time profiling of
likelihood functions" under Future modes, _profiling.py had no occurrence of
"compile", and all three modes read only results/runtime/ -- while
scripts/misc/jax_compile/ held 93 committed probe records that nothing
cross-referenced against the science grid.

campaign --axis compile answers how much of the grid has compile data on a
tier. Records are placed by their own (dataset_class, model_type, instrument)
rather than by path, since results are filed under <hardware>/<model_type> and
the path drops the class and instrument entirely.

Tier mapping is deliberately NOT TIER_CONFIGS. That map keys off sweep config
names which fold precision into the name (local_cpu_fp64 / local_cpu_mp),
whereas a compile record carries a raw hardware string plus a separate
mixed_precision bool; reusing it would mis-bucket every row. "other" is a real
answer rather than a fallback -- the corpus holds RTX-2060 rows belonging to
neither tier.

Off-grid records (knn, delaunay_matern, the datacube_img* multi-band classes)
and non-tier hardware get their own buckets: real measurements that are
neither grid coverage nor noise. Malformed records are reported with file and
index rather than skipped, surfacing the 4 that carry null hardware, class and
instrument.

The mode reports coverage only and never compares two timings. Compile timings
are host-load-sensitive -- jax_compile/README.md records the first
measurements being wrong by up to 7x (851s vs 117s for the same compile)
because XLA compiles on the host cores -- so rows are comparable only within
(hardware, jax_version, mixed_precision, cache state). Comparison waits on the
pins in phase 2.

ingest and triage reject --axis compile with exit 5 rather than ignoring it,
so a compile flag can never silently return a runtime answer.

The transform axis is read from probe.py's TRANSFORMS literal via the same ast
route load_grid uses for CELLS, so the Brain cannot drift from the instrument.

Adds tests/test_profiling_conductor.py -- profiling was the only conductor
without a test file, so the "runtime axis unchanged" requirement had nothing
to assert against. Hermetic: synthetic workspace fixtures, no real checkout.
Runtime-axis output verified byte-identical against the real workspace.

Phase 1 of 3; PyAutoMind draft/feature/profiling/.

Closes #218

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
…atch

The first glob never matched (jax_compile/ has no hpc/ dir) so the fallback
always won, but had it matched it would have printed hpc/batch_gpu/<name>
for a file that lives elsewhere. Use the same path the runtime campaign does.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
jax_compile/ hosts export_probe.py and trace_profile.py, which append their
own schema into the SAME results/<hardware>/ tree probe.py writes to. Their
records have no hardware/dataset_class/instrument because they are a different
record kind, not because they are corrupt -- so the 4 the mode was reporting
as malformed would have sent someone to fix two files that work correctly.

Split the two: missing the whole identity triple is a sibling instrument
(reported per file, under its own bucket); missing only some key fields is
genuine corruption and stays malformed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
Completes the Brain side of phase 2, and is the surveillance the arc exists
for: the persistent cache and --xla_gpu_autotune_level=0 are SETTINGS, so a
config drift or an XLA_FLAGS clobber (the PyAutoNerves#127 failure that went
undetected for two months) puts the worst case back with nothing failing.

Reads the workspace's jax_compile/pins.json and reports warm rows that are
unpinned or have drifted. Every comparison happens strictly inside one
(hardware, hostname, jax_version, mixed_precision, cache_state); cross-key
pairs are never a regression. A jax_version bump recompiles ONCE BY DESIGN,
so it surfaces as a new unpinned key rather than as drift.

Two corrections found by running it against the real corpus rather than
trusting the design:

1. Rows PREDATING their pin are not drift. The first run flagged four, all of
   them July-16 measurements the July-28 pin had been chosen over -- i.e. it
   reported the improvement that set the pin as though it were a regression.
   Drift now requires a row newer than its pin.

2. That exposed the deeper flaw, fixed in the workspace: pins must be sticky.
   With "most recent wins", re-deriving pins after a cache regression would
   have quietly baked the regression in and the surveillance would report
   all-clear forever.

Thresholds are deliberately generous and require BOTH gates -- >= 2.0x the pin
AND >= 1.0s absolute. The ratio alone screams about sub-second cells where
100ms of jitter is 3x; the floor alone misses a cheap cell degrading by an
order of magnitude. Host load alone has produced 7x errors in this corpus, and
an alarm that cries wolf gets ignored.

Against the real corpus: 25 pins, 0 drifted, 0 unpinned.

11 more tests, including a synthetic warm-reverting-to-cold row proving the
alarm fires, each comparability field proving it does not fire across the key,
and a guard that the Brain's mirrored key definition matches the workspace's
pins.py (mirrored rather than imported, since importing would drag the JAX
stack into the Brain).

Stacked on feature/compile-axis-campaign-coverage (needs its
load_compile_corpus/compile_tier_of).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
# Conflicts:
#	agents/conductors/profiling/AGENTS.md
#	agents/conductors/profiling/_profiling.py
#	tests/test_profiling_conductor.py
@Jammy2211
Jammy2211 changed the base branch from feature/compile-axis-campaign-coverage to main August 10, 2026 15:12
@Jammy2211
Jammy2211 merged commit 74d7b1b into main Aug 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants