H2: model manifest registry + feature-schema hashing#5
Merged
Conversation
Activates src/sisyphus/ml/registry.py (previously raised NotImplementedError
for register/get) and ships .meta.json manifests for the 12 XGBoost
artifacts actually loaded by the pipeline.
Schema — docs/science/model_manifest_schema.md
- Sibling .meta.json convention per artifact.
- Required fields: version, target, trained_on{dataset_path,sha256},
feature_schema{name,n_features,sha256,description}, trained_at,
n_drugs_original, n_drugs_excluded, holdout_version, holdout_metric,
hyperparameters, retrained_reason. Missing fields are the literal
string "unknown_legacy" so provenance gaps are greppable.
- feature_schema.sha256 hashes compute_features(caffeine).tobytes(). A
mismatch between recorded hash and current code path means the feature
pipeline has drifted — the model may produce garbage. Warn-only.
Registry (src/sisyphus/ml/registry.py)
- ModelRegistry.get(path): loads + validates, logs warnings on missing
or incomplete manifests, returns ModelRecord or None. Non-blocking.
- ModelRegistry.register(path, manifest): STRICT — raises ValueError on
incomplete manifest. Guards new artifacts from shipping without
provenance.
- Free helpers: load_manifest, validate_manifest, compute_feature_hash_v1,
check_feature_hash, manifest_path_for.
Manifests backfilled (12 files)
- direct_pk: xgboost_cmax (migrated from models/direct_pk/meta.json which
is removed), xgboost_clf, xgboost_vdf.
- adme: xgboost_fup, xgboost_fup_v2, xgboost_clint, xgboost_rbp,
xgboost_vdss, xgboost_peff, xgboost_pka_acidic, xgboost_pka_basic,
logp_correction.
- 11 of 12 use compute_features_v1 (2057-element Morgan+RDKit pipeline,
sha256 dd014cd8e7df59980ac05cdf494dad42f015da8013c5b347ded92e25a804497b).
- logp_correction uses a distinct 6-feature pipeline (sha256
fdf88196a6c9a56c565b8aae6d2b2ee372b56263b401a1caf073cb140bed8114).
Scope exclusions
- dev-only training-script models (xgboost_vdss_v2, xgboost_bioavailability,
xgboost_thalf_v1, xgboost_clearance_v1, xgboost_cmax_v2_mw, etc.) not
backfilled — they are not loaded via src/ at inference time.
- .bak/.contaminated.bak artifacts left alone.
- Hard-error-on-mismatch policy deferred to a future cycle.
Gitignore
- `models/**/*.json` was blocking new .meta.json additions. Added
`!models/**/*.meta.json` negation so manifests ship cleanly.
Tests (tests/unit/test_model_registry.py, 47 cases)
- All 12 shipped manifests exist, validate clean, and feature-hash-match
the current compute_features output.
- Registry.get() returns a ModelRecord on a real manifest.
- Registry.get() on a missing manifest returns None and emits a warning.
- Registry.register() rejects incomplete manifests and writes complete
ones to the sibling path.
- check_feature_hash handles mismatch, missing, and match cases.
- Canonical caffeine SMILES guard.
Unit + integration + benchmark regressions: 11 integration pass + 1 xfail,
3 benchmark pass, registry 47/47 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 715fa02432
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+209
to
+211
| for w in validate_manifest(manifest): | ||
| logger.warning("%s: %s", model_path.name, w) | ||
| return ModelRecord.from_manifest(model_path, manifest) |
There was a problem hiding this comment.
ModelRegistry.get() only runs validate_manifest() and returns a ModelRecord; it never invokes compute_feature_hash_v1()/check_feature_hash(). That means a model trained with an older feature pipeline will load without any mismatch warning, so feature-schema drift can silently corrupt predictions despite shipping feature_schema.sha256 for this exact guardrail.
Useful? React with 👍 / 👎.
| warnings.append(f"manifest field missing: {field_name}") | ||
|
|
||
| feat = manifest.get("feature_schema") | ||
| if isinstance(feat, dict): |
There was a problem hiding this comment.
Validation only checks feature_schema subfields when feature_schema is a dict, so register() will accept malformed manifests like "feature_schema": "compute_features_v1" as “complete.” This breaks the documented strictness for new registrations and allows manifests that cannot carry n_features/sha256, undermining downstream compatibility checks.
Useful? React with 👍 / 👎.
PR #5 CI failed because the manifest feature_schema.sha256 values were computed under my local dev env (rdkit 2023.9.6, numpy 1.26.4) rather than the lockfile env (rdkit 2026.3.1, numpy 2.2.6). CI installs the lockfile, so its compute_features(caffeine) bytes — and therefore the sha256 — differ. 11 parametrized hash-match tests failed. Fix: - Regenerated canonical hashes in a fresh venv installed from requirements-lock.txt: compute_features_v1 → b41dddd7... (was dd014cd8...) logp_corr_6 → a8da0c08... (was fdf88196...) - Updated all 12 shipped manifests with the lockfile-env hashes. - docs/science/model_manifest_schema.md now documents that hashes are coupled to the pinned RDKit+numpy versions and explains the upgrade procedure. - tests/unit/test_model_registry.py: test_shipped_manifest_feature_hash_matches_v1 now skips (with a clear message) when the runtime env does not match the lockfile-pinned hash, instead of failing. CI still exercises it; local dev does not require matching the full lockfile. Verified: - Lockfile venv: 47/47 registry tests pass. - Local dev env: 36 pass, 11 skip (hash-mismatch tests) — no failures. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jam-sudo
added a commit
that referenced
this pull request
May 2, 2026
* feat(ml): model manifest registry with feature-schema hashing (H2)
Activates src/sisyphus/ml/registry.py (previously raised NotImplementedError
for register/get) and ships .meta.json manifests for the 12 XGBoost
artifacts actually loaded by the pipeline.
Schema — docs/science/model_manifest_schema.md
- Sibling .meta.json convention per artifact.
- Required fields: version, target, trained_on{dataset_path,sha256},
feature_schema{name,n_features,sha256,description}, trained_at,
n_drugs_original, n_drugs_excluded, holdout_version, holdout_metric,
hyperparameters, retrained_reason. Missing fields are the literal
string "unknown_legacy" so provenance gaps are greppable.
- feature_schema.sha256 hashes compute_features(caffeine).tobytes(). A
mismatch between recorded hash and current code path means the feature
pipeline has drifted — the model may produce garbage. Warn-only.
Registry (src/sisyphus/ml/registry.py)
- ModelRegistry.get(path): loads + validates, logs warnings on missing
or incomplete manifests, returns ModelRecord or None. Non-blocking.
- ModelRegistry.register(path, manifest): STRICT — raises ValueError on
incomplete manifest. Guards new artifacts from shipping without
provenance.
- Free helpers: load_manifest, validate_manifest, compute_feature_hash_v1,
check_feature_hash, manifest_path_for.
Manifests backfilled (12 files)
- direct_pk: xgboost_cmax (migrated from models/direct_pk/meta.json which
is removed), xgboost_clf, xgboost_vdf.
- adme: xgboost_fup, xgboost_fup_v2, xgboost_clint, xgboost_rbp,
xgboost_vdss, xgboost_peff, xgboost_pka_acidic, xgboost_pka_basic,
logp_correction.
- 11 of 12 use compute_features_v1 (2057-element Morgan+RDKit pipeline,
sha256 dd014cd8e7df59980ac05cdf494dad42f015da8013c5b347ded92e25a804497b).
- logp_correction uses a distinct 6-feature pipeline (sha256
fdf88196a6c9a56c565b8aae6d2b2ee372b56263b401a1caf073cb140bed8114).
Scope exclusions
- dev-only training-script models (xgboost_vdss_v2, xgboost_bioavailability,
xgboost_thalf_v1, xgboost_clearance_v1, xgboost_cmax_v2_mw, etc.) not
backfilled — they are not loaded via src/ at inference time.
- .bak/.contaminated.bak artifacts left alone.
- Hard-error-on-mismatch policy deferred to a future cycle.
Gitignore
- `models/**/*.json` was blocking new .meta.json additions. Added
`!models/**/*.meta.json` negation so manifests ship cleanly.
Tests (tests/unit/test_model_registry.py, 47 cases)
- All 12 shipped manifests exist, validate clean, and feature-hash-match
the current compute_features output.
- Registry.get() returns a ModelRecord on a real manifest.
- Registry.get() on a missing manifest returns None and emits a warning.
- Registry.register() rejects incomplete manifests and writes complete
ones to the sibling path.
- check_feature_hash handles mismatch, missing, and match cases.
- Canonical caffeine SMILES guard.
Unit + integration + benchmark regressions: 11 integration pass + 1 xfail,
3 benchmark pass, registry 47/47 pass.
* fix(ml): pin manifest feature hashes to lockfile env (H2 follow-up)
PR #5 CI failed because the manifest feature_schema.sha256 values were
computed under my local dev env (rdkit 2023.9.6, numpy 1.26.4) rather
than the lockfile env (rdkit 2026.3.1, numpy 2.2.6). CI installs the
lockfile, so its compute_features(caffeine) bytes — and therefore the
sha256 — differ. 11 parametrized hash-match tests failed.
Fix:
- Regenerated canonical hashes in a fresh venv installed from
requirements-lock.txt:
compute_features_v1 → b41dddd7... (was dd014cd8...)
logp_corr_6 → a8da0c08... (was fdf88196...)
- Updated all 12 shipped manifests with the lockfile-env hashes.
- docs/science/model_manifest_schema.md now documents that hashes are
coupled to the pinned RDKit+numpy versions and explains the upgrade
procedure.
- tests/unit/test_model_registry.py: test_shipped_manifest_feature_hash_matches_v1
now skips (with a clear message) when the runtime env does not match
the lockfile-pinned hash, instead of failing. CI still exercises it;
local dev does not require matching the full lockfile.
Verified:
- Lockfile venv: 47/47 registry tests pass.
- Local dev env: 36 pass, 11 skip (hash-mismatch tests) — no failures.
---------
Co-authored-by: jam <jam@sisyphus.dev>
jam-sudo
added a commit
that referenced
this pull request
May 3, 2026
* feat(ml): model manifest registry with feature-schema hashing (H2)
Activates src/sisyphus/ml/registry.py (previously raised NotImplementedError
for register/get) and ships .meta.json manifests for the 12 XGBoost
artifacts actually loaded by the pipeline.
Schema — docs/science/model_manifest_schema.md
- Sibling .meta.json convention per artifact.
- Required fields: version, target, trained_on{dataset_path,sha256},
feature_schema{name,n_features,sha256,description}, trained_at,
n_drugs_original, n_drugs_excluded, holdout_version, holdout_metric,
hyperparameters, retrained_reason. Missing fields are the literal
string "unknown_legacy" so provenance gaps are greppable.
- feature_schema.sha256 hashes compute_features(caffeine).tobytes(). A
mismatch between recorded hash and current code path means the feature
pipeline has drifted — the model may produce garbage. Warn-only.
Registry (src/sisyphus/ml/registry.py)
- ModelRegistry.get(path): loads + validates, logs warnings on missing
or incomplete manifests, returns ModelRecord or None. Non-blocking.
- ModelRegistry.register(path, manifest): STRICT — raises ValueError on
incomplete manifest. Guards new artifacts from shipping without
provenance.
- Free helpers: load_manifest, validate_manifest, compute_feature_hash_v1,
check_feature_hash, manifest_path_for.
Manifests backfilled (12 files)
- direct_pk: xgboost_cmax (migrated from models/direct_pk/meta.json which
is removed), xgboost_clf, xgboost_vdf.
- adme: xgboost_fup, xgboost_fup_v2, xgboost_clint, xgboost_rbp,
xgboost_vdss, xgboost_peff, xgboost_pka_acidic, xgboost_pka_basic,
logp_correction.
- 11 of 12 use compute_features_v1 (2057-element Morgan+RDKit pipeline,
sha256 dd014cd8e7df59980ac05cdf494dad42f015da8013c5b347ded92e25a804497b).
- logp_correction uses a distinct 6-feature pipeline (sha256
fdf88196a6c9a56c565b8aae6d2b2ee372b56263b401a1caf073cb140bed8114).
Scope exclusions
- dev-only training-script models (xgboost_vdss_v2, xgboost_bioavailability,
xgboost_thalf_v1, xgboost_clearance_v1, xgboost_cmax_v2_mw, etc.) not
backfilled — they are not loaded via src/ at inference time.
- .bak/.contaminated.bak artifacts left alone.
- Hard-error-on-mismatch policy deferred to a future cycle.
Gitignore
- `models/**/*.json` was blocking new .meta.json additions. Added
`!models/**/*.meta.json` negation so manifests ship cleanly.
Tests (tests/unit/test_model_registry.py, 47 cases)
- All 12 shipped manifests exist, validate clean, and feature-hash-match
the current compute_features output.
- Registry.get() returns a ModelRecord on a real manifest.
- Registry.get() on a missing manifest returns None and emits a warning.
- Registry.register() rejects incomplete manifests and writes complete
ones to the sibling path.
- check_feature_hash handles mismatch, missing, and match cases.
- Canonical caffeine SMILES guard.
Unit + integration + benchmark regressions: 11 integration pass + 1 xfail,
3 benchmark pass, registry 47/47 pass.
* fix(ml): pin manifest feature hashes to lockfile env (H2 follow-up)
PR #5 CI failed because the manifest feature_schema.sha256 values were
computed under my local dev env (rdkit 2023.9.6, numpy 1.26.4) rather
than the lockfile env (rdkit 2026.3.1, numpy 2.2.6). CI installs the
lockfile, so its compute_features(caffeine) bytes — and therefore the
sha256 — differ. 11 parametrized hash-match tests failed.
Fix:
- Regenerated canonical hashes in a fresh venv installed from
requirements-lock.txt:
compute_features_v1 → b41dddd7... (was dd014cd8...)
logp_corr_6 → a8da0c08... (was fdf88196...)
- Updated all 12 shipped manifests with the lockfile-env hashes.
- docs/science/model_manifest_schema.md now documents that hashes are
coupled to the pinned RDKit+numpy versions and explains the upgrade
procedure.
- tests/unit/test_model_registry.py: test_shipped_manifest_feature_hash_matches_v1
now skips (with a clear message) when the runtime env does not match
the lockfile-pinned hash, instead of failing. CI still exercises it;
local dev does not require matching the full lockfile.
Verified:
- Lockfile venv: 47/47 registry tests pass.
- Local dev env: 36 pass, 11 skip (hash-mismatch tests) — no failures.
---------
Co-authored-by: jam <jam@sisyphus.dev>
jam-sudo
added a commit
that referenced
this pull request
May 31, 2026
…ent flux drop (#53) * fix(engine): JAX RHS rejects unsupported flux types instead of silent drop ProdrugActivationFluxSpec and OneCompartmentEliminationFluxSpec had no branch in make_jax_rhs's isinstance chain and no terminal else, so a graph with a prodrug-activation or active-metabolite edge would have that flux silently omitted from the JAX RHS (the SciPy backend handles them correctly). Add a pure-Python _unsupported_flux_specs() guard that make_jax_rhs calls before the build loop, raising NotImplementedError instead. The guard imports no JAX, so the 'no silent drop' invariant is unit-tested even though JAX is optional and absent from requirements-lock.txt. Audit follow-up; no production path uses backend='jax'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(data): close pravastatin holdout->MMPK training leak; guard invariant #5 Pravastatin was the only holdout drug surviving BOTH filters in ml_cmax_improvement.load_mmpk_data: it had in_holdout=False rows AND its MMPK canon_smiles InChIKey-14 (TUZYXOIXSAXUGO) differs in connectivity from its clinical_pk SMILES (GOSGZXISMCZCDW), so the ho_ik filter missed it too. The other ~70 holdout drugs present in the corpus are correctly excluded by the InChIKey filter. - Correct the in_holdout flag (False->True) on pravastatin's 3 rows in both mmpk_expanded_{full,v2}.csv — the universal first-line filter every consumer respects (ml_cmax_improvement, build_clf_training_data, etc.). - Add load_holdout_names() + a name-based exclusion to load_mmpk_data (defense-in-depth, robust to SMILES-representation drift; mirrors build_n50_exclusion.py's name policy). - Add tests/regression/test_mmpk_holdout_leak.py pinning invariant #5. Forward-looking only: the shipped xgboost_cmax.json was trained on Omega's mmpk_clean.csv with its own N=107 3-key exclusion (not via this loader), so the current headline cache is unaffected. Audit follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(experiment-log): record 2026-05-31 completeness audit + 3 hardening fixes 29-agent adversarial audit (overall B+/~77; invariants hold). Logs Fix 1 (CLAUDE.md headline reconcile to cache 2.698/N=79), Fix 2 (pravastatin holdout->MMPK leak — corrected severity: forward-looking, shipped model is Omega-trained), Fix 3 (JAX RHS silent-drop guard). No metric change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: jam <jam@sisyphus.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jam-sudo
added a commit
that referenced
this pull request
May 31, 2026
* feat(ml): model manifest registry with feature-schema hashing (H2)
Activates src/sisyphus/ml/registry.py (previously raised NotImplementedError
for register/get) and ships .meta.json manifests for the 12 XGBoost
artifacts actually loaded by the pipeline.
Schema — docs/science/model_manifest_schema.md
- Sibling .meta.json convention per artifact.
- Required fields: version, target, trained_on{dataset_path,sha256},
feature_schema{name,n_features,sha256,description}, trained_at,
n_drugs_original, n_drugs_excluded, holdout_version, holdout_metric,
hyperparameters, retrained_reason. Missing fields are the literal
string "unknown_legacy" so provenance gaps are greppable.
- feature_schema.sha256 hashes compute_features(caffeine).tobytes(). A
mismatch between recorded hash and current code path means the feature
pipeline has drifted — the model may produce garbage. Warn-only.
Registry (src/sisyphus/ml/registry.py)
- ModelRegistry.get(path): loads + validates, logs warnings on missing
or incomplete manifests, returns ModelRecord or None. Non-blocking.
- ModelRegistry.register(path, manifest): STRICT — raises ValueError on
incomplete manifest. Guards new artifacts from shipping without
provenance.
- Free helpers: load_manifest, validate_manifest, compute_feature_hash_v1,
check_feature_hash, manifest_path_for.
Manifests backfilled (12 files)
- direct_pk: xgboost_cmax (migrated from models/direct_pk/meta.json which
is removed), xgboost_clf, xgboost_vdf.
- adme: xgboost_fup, xgboost_fup_v2, xgboost_clint, xgboost_rbp,
xgboost_vdss, xgboost_peff, xgboost_pka_acidic, xgboost_pka_basic,
logp_correction.
- 11 of 12 use compute_features_v1 (2057-element Morgan+RDKit pipeline,
sha256 dd014cd8e7df59980ac05cdf494dad42f015da8013c5b347ded92e25a804497b).
- logp_correction uses a distinct 6-feature pipeline (sha256
fdf88196a6c9a56c565b8aae6d2b2ee372b56263b401a1caf073cb140bed8114).
Scope exclusions
- dev-only training-script models (xgboost_vdss_v2, xgboost_bioavailability,
xgboost_thalf_v1, xgboost_clearance_v1, xgboost_cmax_v2_mw, etc.) not
backfilled — they are not loaded via src/ at inference time.
- .bak/.contaminated.bak artifacts left alone.
- Hard-error-on-mismatch policy deferred to a future cycle.
Gitignore
- `models/**/*.json` was blocking new .meta.json additions. Added
`!models/**/*.meta.json` negation so manifests ship cleanly.
Tests (tests/unit/test_model_registry.py, 47 cases)
- All 12 shipped manifests exist, validate clean, and feature-hash-match
the current compute_features output.
- Registry.get() returns a ModelRecord on a real manifest.
- Registry.get() on a missing manifest returns None and emits a warning.
- Registry.register() rejects incomplete manifests and writes complete
ones to the sibling path.
- check_feature_hash handles mismatch, missing, and match cases.
- Canonical caffeine SMILES guard.
Unit + integration + benchmark regressions: 11 integration pass + 1 xfail,
3 benchmark pass, registry 47/47 pass.
* fix(ml): pin manifest feature hashes to lockfile env (H2 follow-up)
PR #5 CI failed because the manifest feature_schema.sha256 values were
computed under my local dev env (rdkit 2023.9.6, numpy 1.26.4) rather
than the lockfile env (rdkit 2026.3.1, numpy 2.2.6). CI installs the
lockfile, so its compute_features(caffeine) bytes — and therefore the
sha256 — differ. 11 parametrized hash-match tests failed.
Fix:
- Regenerated canonical hashes in a fresh venv installed from
requirements-lock.txt:
compute_features_v1 → b41dddd7... (was dd014cd8...)
logp_corr_6 → a8da0c08... (was fdf88196...)
- Updated all 12 shipped manifests with the lockfile-env hashes.
- docs/science/model_manifest_schema.md now documents that hashes are
coupled to the pinned RDKit+numpy versions and explains the upgrade
procedure.
- tests/unit/test_model_registry.py: test_shipped_manifest_feature_hash_matches_v1
now skips (with a clear message) when the runtime env does not match
the lockfile-pinned hash, instead of failing. CI still exercises it;
local dev does not require matching the full lockfile.
Verified:
- Lockfile venv: 47/47 registry tests pass.
- Local dev env: 36 pass, 11 skip (hash-mismatch tests) — no failures.
---------
Co-authored-by: jam <jam@sisyphus.dev>
jam-sudo
added a commit
that referenced
this pull request
May 31, 2026
…ent flux drop (#53) * fix(engine): JAX RHS rejects unsupported flux types instead of silent drop ProdrugActivationFluxSpec and OneCompartmentEliminationFluxSpec had no branch in make_jax_rhs's isinstance chain and no terminal else, so a graph with a prodrug-activation or active-metabolite edge would have that flux silently omitted from the JAX RHS (the SciPy backend handles them correctly). Add a pure-Python _unsupported_flux_specs() guard that make_jax_rhs calls before the build loop, raising NotImplementedError instead. The guard imports no JAX, so the 'no silent drop' invariant is unit-tested even though JAX is optional and absent from requirements-lock.txt. Audit follow-up; no production path uses backend='jax'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(data): close pravastatin holdout->MMPK training leak; guard invariant #5 Pravastatin was the only holdout drug surviving BOTH filters in ml_cmax_improvement.load_mmpk_data: it had in_holdout=False rows AND its MMPK canon_smiles InChIKey-14 (TUZYXOIXSAXUGO) differs in connectivity from its clinical_pk SMILES (GOSGZXISMCZCDW), so the ho_ik filter missed it too. The other ~70 holdout drugs present in the corpus are correctly excluded by the InChIKey filter. - Correct the in_holdout flag (False->True) on pravastatin's 3 rows in both mmpk_expanded_{full,v2}.csv — the universal first-line filter every consumer respects (ml_cmax_improvement, build_clf_training_data, etc.). - Add load_holdout_names() + a name-based exclusion to load_mmpk_data (defense-in-depth, robust to SMILES-representation drift; mirrors build_n50_exclusion.py's name policy). - Add tests/regression/test_mmpk_holdout_leak.py pinning invariant #5. Forward-looking only: the shipped xgboost_cmax.json was trained on Omega's mmpk_clean.csv with its own N=107 3-key exclusion (not via this loader), so the current headline cache is unaffected. Audit follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(experiment-log): record 2026-05-31 completeness audit + 3 hardening fixes 29-agent adversarial audit (overall B+/~77; invariants hold). Logs Fix 1 (CLAUDE.md headline reconcile to cache 2.698/N=79), Fix 2 (pravastatin holdout->MMPK leak — corrected severity: forward-looking, shipped model is Omega-trained), Fix 3 (JAX RHS silent-drop guard). No metric change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: jam <jam@sisyphus.dev> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
jam-sudo
added a commit
that referenced
this pull request
Jun 9, 2026
…docs) Response to the 10-finding review of PR #69. Guards/bugs: - _recover_f_engine: raise instead of silently returning F=0.5 when the measured-F routing did not scale the engine (failed oral IV-reference solve); point to cl_latent=True (#2). - CLGrid.conc_at: raise when an observation time is past the grid horizon rather than letting np.interp edge-clamp a TDM trough (#3). - _softmax_resample: log a degeneracy warning when n_eff < 0.5% of draws (#4). - predict_posterior: reject non-oral route (the F latent is oral-specific; IV has F equivalent to 1) (#8). - build_cl_grid: raise on total engine failure instead of returning a silent NaN grid; _nearest_finite_backfill replaces the adjacent-copy that could copy a still-NaN neighbor (#9). Grid faithfulness: - build_cl_grid now mirrors predict()'s OATP-ECM + non-CYP (UGT/NAT) disposition wiring, so the s=1 grid reproduces predict() for non-CYP substrates (codeine 18% -> <3% divergence; new faithfulness test) (#1). - document the clint-scale latent as metabolic-only (renal/biliary held fixed), not a total-CL scale (#5). Docs: - cmax_90ci flagged as a-priori-calibrated / conservative-when-conditioned (#6). - document the (F, clint-scale) identifiability ridge: a curve-shape observation is required to separate the two latents (#7). 12 new unit tests, all written test-first (RED->GREEN). Full mipd suite 52/52.
jam-sudo
added a commit
that referenced
this pull request
Jul 2, 2026
Ships the canonical CI-stack artifacts for the CL/F-track InChIKey-14 holdout-leak fix (PR #90), which removed 5 name-evading stereo/salt holdout collisions from clf_training.csv (1131 → 1126). This PR retrains CLF/VDF leak-free and re-pins the headline. Dual-arm regen, one CI run (clf-leakfree-regen.yml / regen_clf_leakfree_canonical.py): baseline retrain (leaky csv) Meta AAFE 2.73104 — reproduces the committed 2.731044 to ±0.00004, so the delta is cleanly attributable to the leak fix leak-free retrain Meta AAFE 2.73531 delta_leak = +0.00427, well inside the bootstrap CI (half-width ~0.42). The sign is stack-dependent (a local macOS retrain moved it -0.004): the leak effect sits at the retrain-noise floor, not a distinguishable accuracy change. The leaked drugs are poorly predicted (methylphenidate 12x, quinine 8.7x), so the leak never inflated accuracy. Correctness-first per Invariant #5 (holdout is inviolable): correct data hygiene over a marginally-lower number. Canonical artifacts (CI-generated): clf_training.csv (1126), models/direct_pk/xgboost_{clf,vdf}.json, 4track_holdout_predictions.json (2.735), 4track_ci_2026-07-02_clf_leakfree.json, prodrug_v3_pre_baseline.json. Re-pins: test_cached_holdout_aafe_is_2p731 -> _2p735 (+ node-id/name refs in 5 probe/invariance tests); headline assertions 2.731 -> 2.735 in cr_mc / concentration_response / multispecies / gsh probes. Engine 4.244 / ML 2.998 unchanged; in-domain 2.777 -> 2.781 (N=81); tebipenem pin unchanged (CL/F does not touch the prodrug arm). README headline + reproducibility band + experiment-log updated. The agent-write-protected private instructions file's headline block still reads 2.731 and needs a manual maintainer update. Co-authored-by: jam-sudo <jam-sudo@users.noreply.github.com>
This was referenced Jul 6, 2026
docs(oatp): formalize the +BSA PSu,inf deferral (data-blocked) after closing the auto-ECM xfail
#104
Merged
jam-sudo
added a commit
that referenced
this pull request
Jul 7, 2026
… + invivo-F-prior (DE-56) (#105) Ran the two cheap pre-registered kill-tests DE-44 flagged as UQ/accuracy survivors. Both foreclosed on the 107-holdout (local public-clone). DE-55 (adaptive conformal PI): no per-drug difficulty signal predicts the meta fold-error — Spearman rho is ~0 for every obs-free signal (divergence x3, AD flags, magnitude, and the named MC parameter-uncertainty half-width rho=0.039; max |rho|=0.069). Mondrian-by-compound-type widens +86% (small-n per class -> conformal q lands at class-max). The flat /÷12.9 PI is near-optimal under Invariant #5; structural error is per-drug non-discriminable (DE-41). DE-56 (invivo-F-prior): the R2=-0.09 structure->F predictor applied via measured-F routing made the meta AAFE FALL (2.743->2.646 at w=0.5), the pre-registered surprise branch. Placebo controls settle it: a CONSTANT F (-0.133) and a SHUFFLED F (-0.116) both reproduce/exceed the real per-drug F (-0.097) -> zero per-drug F info. It is the DE-42 flat-scalar median-bias null (engine under-predicts F, any upward scalar nulls the median bias). Triple-dead: placebo-artifact + Invariant #8 (holdout-fit scalar) + within-CI + non-generalizing. The pre-registration + placebo discipline caught a would-be 'broke the ceiling' false positive. Adds DE-55, DE-56, a 2026-07-07 experiment-log entry, and the kill-test artifact. Headline 2.743 untouched. Co-authored-by: jam-sudo <jam-sudo@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
H2 per
docs/claude/hardening_backlog.md(amended). Activates the previously-stubsrc/sisyphus/ml/registry.pyand ships.meta.jsonmanifests for the 12 XGBoost artifacts that the inference pipeline actually loads. Adds a reproduciblefeature_schema.sha256fingerprint so a drift in the feature extractor cannot silently load garbage predictions.What ships
Schema (
docs/science/model_manifest_schema.md).meta.jsonconvention; required fields;"unknown_legacy"sentinel for unrecoverable provenance; canonical caffeine SMILES for feature hashing.Registry (
src/sisyphus/ml/registry.py)ModelRegistry.get(path)— warn-only load + validate.ModelRegistry.register(path, manifest)— strict;ValueErroron incomplete manifest.load_manifest,validate_manifest,compute_feature_hash_v1,check_feature_hash,manifest_path_for.12 backfilled manifests
models/direct_pk/:xgboost_cmax.meta.json(migrated from existingmeta.jsonwhich is removed),xgboost_clf.meta.json,xgboost_vdf.meta.json.models/adme/:xgboost_fup,xgboost_fup_v2,xgboost_clint,xgboost_rbp,xgboost_vdss,xgboost_peff,xgboost_pka_acidic,xgboost_pka_basic,logp_correction.compute_features_v1hashdd014cd8….logp_correctionuses its own 6-feature pipeline hashfdf88196….Gitignore
!models/**/*.meta.jsonnegation; base rulemodels/**/*.jsonwas blocking manifest additions.Non-goals
xgboost_vdss_v2,xgboost_bioavailability,xgboost_thalf_v1,xgboost_clearance_v1,xgboost_cmax_v2_mw) not backfilled — they are not loaded viasrc/at inference time..bak/.contaminated.bakartifacts unchanged.Invariants preserved
No engine, no metric, no holdout, no pipeline topology changes. Pure metadata + infra.
Test plan
tests/unit/test_model_registry.py47 tests (registry logic + 12 parametrized shipped-manifest checks)test_engine_validation.py11 passed + 1 xfailtest_holdout.py3 passed🤖 Generated with Claude Code