feat(profiling): add a compile-time axis to campaign coverage - #219
Merged
Conversation
The Profiling Agent could not see the compile-time corpus autolens_profiling is already producing. AGENTS.md listed "JAX compilation-time profiling of likelihood functions" under Future modes, _profiling.py had no occurrence of "compile", and all three modes read only results/runtime/ -- while scripts/misc/jax_compile/ held 93 committed probe records that nothing cross-referenced against the science grid. campaign --axis compile answers how much of the grid has compile data on a tier. Records are placed by their own (dataset_class, model_type, instrument) rather than by path, since results are filed under <hardware>/<model_type> and the path drops the class and instrument entirely. Tier mapping is deliberately NOT TIER_CONFIGS. That map keys off sweep config names which fold precision into the name (local_cpu_fp64 / local_cpu_mp), whereas a compile record carries a raw hardware string plus a separate mixed_precision bool; reusing it would mis-bucket every row. "other" is a real answer rather than a fallback -- the corpus holds RTX-2060 rows belonging to neither tier. Off-grid records (knn, delaunay_matern, the datacube_img* multi-band classes) and non-tier hardware get their own buckets: real measurements that are neither grid coverage nor noise. Malformed records are reported with file and index rather than skipped, surfacing the 4 that carry null hardware, class and instrument. The mode reports coverage only and never compares two timings. Compile timings are host-load-sensitive -- jax_compile/README.md records the first measurements being wrong by up to 7x (851s vs 117s for the same compile) because XLA compiles on the host cores -- so rows are comparable only within (hardware, jax_version, mixed_precision, cache state). Comparison waits on the pins in phase 2. ingest and triage reject --axis compile with exit 5 rather than ignoring it, so a compile flag can never silently return a runtime answer. The transform axis is read from probe.py's TRANSFORMS literal via the same ast route load_grid uses for CELLS, so the Brain cannot drift from the instrument. Adds tests/test_profiling_conductor.py -- profiling was the only conductor without a test file, so the "runtime axis unchanged" requirement had nothing to assert against. Hermetic: synthetic workspace fixtures, no real checkout. Runtime-axis output verified byte-identical against the real workspace. Phase 1 of 3; PyAutoMind draft/feature/profiling/. Closes #218 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
…atch The first glob never matched (jax_compile/ has no hpc/ dir) so the fallback always won, but had it matched it would have printed hpc/batch_gpu/<name> for a file that lives elsewhere. Use the same path the runtime campaign does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
jax_compile/ hosts export_probe.py and trace_profile.py, which append their own schema into the SAME results/<hardware>/ tree probe.py writes to. Their records have no hardware/dataset_class/instrument because they are a different record kind, not because they are corrupt -- so the 4 the mode was reporting as malformed would have sent someone to fix two files that work correctly. Split the two: missing the whole identity triple is a sibling instrument (reported per file, under its own bucket); missing only some key fields is genuine corruption and stays malformed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L4STU81pQP1GkMzZVvsMsv
Jammy2211
pushed a commit
that referenced
this pull request
Aug 10, 2026
# Conflicts: # agents/conductors/profiling/AGENTS.md # agents/conductors/profiling/_profiling.py # tests/test_profiling_conductor.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #218. Phase 1 of 3 of the compile-axis arc.
What this fixes
The Profiling Agent could not see the compile-time corpus
autolens_profilingisalready producing.
AGENTS.mdlisted "JAX compilation-time profiling of likelihoodfunctions" under Future modes,
_profiling.pyhad zero occurrences of "compile",and all three modes read only
results/runtime/— whilescripts/misc/jax_compile/held 93 committed probe records that nothing cross-referenced against the science
grid.
campaign --axis compilenow answers how much of the grid has compile data on a tier:That is the number the arc's other two phases are justified by: 2 of 21 grid cells
have any compile coverage,
hstonly, with no interferometer, datacube,jwstoraorow anywhere.
Design decisions worth review
Records are placed by their own fields, not by path. Results are filed under
<hardware>/<model_type>, which drops the dataset class and instrument entirely, soplacement uses the in-record
(dataset_class, model_type, instrument). A test files aninterferometer record inside
mge.jsonto pin this.The tier map is deliberately not
TIER_CONFIGS. That map keys off sweep confignames which fold precision into the name (
local_cpu_fp64/local_cpu_mp), whereas acompile record carries a raw
hardwarestring plus a separatemixed_precisionbool. Reusing it would mis-bucket every row. A test pins that an fp64 and an mp record
on one machine are one cell/transform run.
otheris a real answer, not a fallback. The corpus holds RTX-2060 rows belongingto neither tier; folding them into
localwould report coverage on hardware nobodyasked about. Same reasoning for the off-grid bucket —
knn,delaunay_maternandthe
datacube_img*multi-band classes are genuine Prodigy-census and multi-bandmeasurements, so they are neither grid coverage nor noise.
Malformed records are surfaced, not skipped — with file and index, exposing the 4
that carry
hardware,dataset_classandinstrumentallnull.Coverage only; it never compares two timings.
jax_compile/README.mdrecords thefirst measurements being wrong by up to 7× (851 s vs 117 s for the same compile)
purely from host load, because XLA compiles on the host cores. Rows are comparable only
within
(hardware, jax_version, mixed_precision, cache state), and comparison waits onthe pins in phase 2. A test asserts no timing value can leak into a coverage decision.
ingest/triagereject--axis compilewith exit 5 rather than ignoring theflag, so a compile flag can never silently return a runtime answer.
The transform axis is read from the workspace, via the same
astrouteload_griduses for
CELLS, so the Brain cannot drift fromprobe.py'sTRANSFORMS.Verification
tests/test_profiling_conductor.py. Profiling was the onlyconductor without a test file, so the "runtime axis unchanged" requirement had nothing
to assert against — the file is part of the work, not a bonus. Hermetic synthetic
workspace fixtures on the
test_hygiene_conductor.pypattern; stdlib + pytest only.both
campaignandingest), plus tests asserting the runtime decision shape gains noaxiskey and that the two tier vocabularies never cross.test_skill_install.pyfailures reproduce on cleanmainin this environment (they assert Codex-skill install behaviour that aweb-github / ci-onlyenvironment reports differently) and are untouched by thischange.
Not in scope
Any judgement about whether a compile time is good (phase 3), any pin/baseline/
dashboard (phase 2), and executing
probe.py— the agent reasons and emits a plan,never runs the workspace.
Follow-ups
compile_warm_baseline_dashboard.md: warm rows are not machine-identifiable today (warmness lives only in a free-text
tag, ~40 ad-hoc values, andcache_diris non-empty on cold rows too), soprobe.pyneeds an explicitcache_statebefore any warm-compile tracking is possible.compile_axis_triage_drift.md: classify drift, and close the arc bymoving compile-time profiling out of Future modes.
Generated by Claude Code