From d396d6519d53053140636238d693388dec58986d Mon Sep 17 00:00:00 2001 From: Michael Heller <21163552+mdheller@users.noreply.github.com> Date: Sun, 2 Aug 2026 21:37:57 -0400 Subject: [PATCH] fix(data): adversarial hardening of trained DataClass classifiers (self-review, no Copilot on #264) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adversarial + functional review of #264 (which Copilot didn't review) found: - the monotone constraint was VACUOUS — the fixture held decimal_places constant, so its weight trained to 0 and the monotone teeth passed trivially. New fixture makes BOTH monotone features (token_magnitude, decimal_places) the primary label signal → both weights bind strongly positive, accuracy realistic (<1.0), monotonicity genuinely exercised. - standardiser was fit on FULL data (mild train/test leakage) → now fit on the TRAIN split only. - ModelManifest overclaimed 'wide-and-deep'/lattice → renamed 'monotone-logistic' (an honest realisation of the declared tf-lattice-wide-and-deep contract, not a full calibrated lattice). - validator now checks BOTH per-class heads monotone + asserts the constraint BINDS (non-vacuous), and — a latent bug the trivial 1.0-accuracy fixture had masked — compares reproduced accuracy at the stored 4-decimal precision instead of full-precision vs 1e-6 (per-class 0.7857 now reproduces). Tamper teeth still refuses corrupted weights. 8 checks green. --- CHANGELOG.md | 1 + .../model-manifest.dataclass-assigner.json | 6 +- examples/run.dataclass-eval.json | 2 +- fixtures/dataclass-training/labeled.json | 366 +++++++++++------- models/dataclass-assigner/trained_model.json | 50 +-- specs/trained-dataclass-classifiers.md | 2 +- tools/train_dataclass_classifiers.py | 29 +- tools/validate_trained_classifiers.py | 37 +- 8 files changed, 303 insertions(+), 190 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6befab8..d53f9aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). Thi ## [Unreleased] ### Added +- Trained DataClass classifiers — adversarial hardening (follow-up to #264): stronger fixture where BOTH monotone features (token_magnitude, decimal_places) genuinely drive the label so the monotone constraint actually BINDS (was vacuous — decimal_places held constant); standardiser fit on the TRAIN split only (no test leakage); ModelManifest named honestly `monotone-logistic` (a realisation of the declared tf-lattice-wide-and-deep contract, not a full lattice). Validator now checks BOTH per-class heads monotone AND asserts the constraint binds (non-vacuous), and compares reproduced accuracy at the stored 4-decimal precision (fixes a latent tolerance bug the trivial 1.0-accuracy fixture had masked). 8 teeth. - Trained DataClass classifiers v0.1 — specified→trained→registered→reproduced (closes the program's last owed thread): `tools/train_dataclass_classifiers.py` trains real per-class LOGISTIC (one-vs-rest) + per-table SOFTMAX heads, MONOTONE-constrained in `monotonicFeatures` (weight >=0 projection = TF-Lattice essence), deterministically; emits trained weights + a conformant `ModelManifest` (id=DataClass modelRef) + train/eval `RunRecord`s (ids=runRef/evalRunRef) so the previously-dangling DataClass classifier references now RESOLVE. `validate-trained-classifiers` (recompute-don't-trust): digest matches weight bytes; eval accuracy REPRODUCED from weights+fixture (inflated claim fails); model verified genuinely monotone (perturb up → score never drops); refs resolve; artifacts schema-conform. Teeth-verified (tampered weights refused). Uses numpy. - Learned spell-correction + user dictionary v0.1 — from context, not dictionary matching (task #13): `tools/learned_dictionary.py` decides each UNKNOWN token from a count-based skip-gram word-sense predictor (PPMI + truncated SVD; SGNS≈PPMI-SVD) instead of a static wordlist. LEARN a token that recurs with a coherent word-sense (a real domain term the dictionary hadn't seen); CORRECT a rare token to the known word its SENSE matches (skip-gram cosine picks the target, not edit distance alone — a near-spelled but different-sense token is not auto-corrected); leave the rest UNKNOWN (fail-closed, human-admitted). Every decision is a proposal, never a silent rewrite. `validate-learned-dictionary` teeth: epistemiclevel learned, reciept->receipt by sense, qwzptl unknown, learned term never auto-corrected. Uses numpy. - Stopword deviation analysis v0.1 — the dropped words are governed vocabulary too (task #13): `tools/stopword_analysis.py` audits the loop's stoplist ACROSS domains using two signals — cross-domain deviation (concentration) AND compositional density (repeated-collocation rate) — because frequency alone can't tell a domain term from a stylistic quirk. Surfaces `term-candidate` (concentrated + compositional = a domain term hiding in the stoplist, propose un-stoplisting) vs `stylistic` (concentrated by style only) vs `noise` (uniform). `validate-stopword-analysis` teeth: domain terms surfaced; a stylistically-concentrated word ('and') is NOT wrongly promoted; a uniform word ('the') is noise. `stopword-analysis-live` audits the shipped stoplist over specs/*.md. Compositional density is the bigram floor of the k-gram TF-IDF/LSA differential (orders 3..7) to follow. diff --git a/examples/model-manifest.dataclass-assigner.json b/examples/model-manifest.dataclass-assigner.json index 0dab02c..5bd47b7 100644 --- a/examples/model-manifest.dataclass-assigner.json +++ b/examples/model-manifest.dataclass-assigner.json @@ -2,9 +2,9 @@ "id": "urn:srcos:model-manifest:dataclass-assigner-lattice-v1", "type": "ModelManifest", "specVersion": "2.1.0", - "modelDigest": "sha256:6092cf3788d3cad23c5639dd5e10e92d74a652ccc56ef6250542a91beb452f04", - "displayName": "dataclass-assigner (monotone wide-and-deep)", - "architecture": "monotone-logistic-wide-and-deep", + "modelDigest": "sha256:d8774347e7f8c65488402e06d09043969f32930d196aae11d748ac6caab3b5b3", + "displayName": "dataclass-assigner (monotone logistic)", + "architecture": "monotone-logistic", "format": "onnx", "quantization": "none", "license": "Apache-2.0", diff --git a/examples/run.dataclass-eval.json b/examples/run.dataclass-eval.json index 136ff00..305a120 100644 --- a/examples/run.dataclass-eval.json +++ b/examples/run.dataclass-eval.json @@ -12,7 +12,7 @@ "value": 1.0, "perClassAccuracy": { "revenue": 1.0, - "cost": 1.0 + "cost": 0.7857 } } }, diff --git a/fixtures/dataclass-training/labeled.json b/fixtures/dataclass-training/labeled.json index 8f5fa1e..c885ad0 100644 --- a/fixtures/dataclass-training/labeled.json +++ b/fixtures/dataclass-training/labeled.json @@ -16,7 +16,7 @@ "rows": [ { "features": { - "token_magnitude": 6.6, + "token_magnitude": 5.14, "decimal_places": 2, "is_negative": 0, "has_cost_keyword": 0 @@ -25,7 +25,7 @@ }, { "features": { - "token_magnitude": 7.91, + "token_magnitude": 6.18, "decimal_places": 2, "is_negative": 0, "has_cost_keyword": 0 @@ -34,17 +34,17 @@ }, { "features": { - "token_magnitude": 6.33, + "token_magnitude": 6.68, "decimal_places": 2, "is_negative": 0, - "has_cost_keyword": 0 + "has_cost_keyword": 1 }, "label": "revenue" }, { "features": { - "token_magnitude": 2.33, - "decimal_places": 2, + "token_magnitude": 2.19, + "decimal_places": 0, "is_negative": 1, "has_cost_keyword": 1 }, @@ -52,16 +52,16 @@ }, { "features": { - "token_magnitude": 4.44, - "decimal_places": 2, - "is_negative": 0, + "token_magnitude": 1.82, + "decimal_places": 0, + "is_negative": 1, "has_cost_keyword": 1 }, "label": "cost" }, { "features": { - "token_magnitude": 3.97, + "token_magnitude": 4.12, "decimal_places": 2, "is_negative": 0, "has_cost_keyword": 1 @@ -70,17 +70,17 @@ }, { "features": { - "token_magnitude": 4.56, - "decimal_places": 2, - "is_negative": 1, - "has_cost_keyword": 1 + "token_magnitude": 6.03, + "decimal_places": 4, + "is_negative": 0, + "has_cost_keyword": 0 }, - "label": "cost" + "label": "revenue" }, { "features": { - "token_magnitude": 4.29, - "decimal_places": 2, + "token_magnitude": 5.12, + "decimal_places": 4, "is_negative": 0, "has_cost_keyword": 0 }, @@ -88,26 +88,26 @@ }, { "features": { - "token_magnitude": 4.79, - "decimal_places": 2, - "is_negative": 0, - "has_cost_keyword": 1 + "token_magnitude": 3.46, + "decimal_places": 0, + "is_negative": 1, + "has_cost_keyword": 0 }, "label": "cost" }, { "features": { - "token_magnitude": 2.66, + "token_magnitude": 6.26, "decimal_places": 2, - "is_negative": 1, - "has_cost_keyword": 1 + "is_negative": 0, + "has_cost_keyword": 0 }, - "label": "cost" + "label": "revenue" }, { "features": { - "token_magnitude": 4.19, - "decimal_places": 2, + "token_magnitude": 5.06, + "decimal_places": 3, "is_negative": 0, "has_cost_keyword": 0 }, @@ -115,8 +115,8 @@ }, { "features": { - "token_magnitude": 6.03, - "decimal_places": 2, + "token_magnitude": 7.04, + "decimal_places": 4, "is_negative": 0, "has_cost_keyword": 0 }, @@ -124,35 +124,44 @@ }, { "features": { - "token_magnitude": 2.47, - "decimal_places": 2, - "is_negative": 1, + "token_magnitude": 4.43, + "decimal_places": 0, + "is_negative": 0, "has_cost_keyword": 1 }, "label": "cost" }, { "features": { - "token_magnitude": 5.78, + "token_magnitude": 8.83, + "decimal_places": 4, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 2.41, "decimal_places": 2, - "is_negative": 1, + "is_negative": 0, "has_cost_keyword": 1 }, "label": "cost" }, { "features": { - "token_magnitude": 4.23, + "token_magnitude": 2.81, "decimal_places": 2, "is_negative": 0, - "has_cost_keyword": 0 + "has_cost_keyword": 1 }, - "label": "revenue" + "label": "cost" }, { "features": { - "token_magnitude": 5.16, - "decimal_places": 2, + "token_magnitude": 8.37, + "decimal_places": 4, "is_negative": 0, "has_cost_keyword": 0 }, @@ -160,7 +169,16 @@ }, { "features": { - "token_magnitude": 5.7, + "token_magnitude": 3.55, + "decimal_places": 0, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 7.26, "decimal_places": 2, "is_negative": 0, "has_cost_keyword": 0 @@ -169,17 +187,17 @@ }, { "features": { - "token_magnitude": 5.11, - "decimal_places": 2, - "is_negative": 1, + "token_magnitude": 2.24, + "decimal_places": 0, + "is_negative": 0, "has_cost_keyword": 1 }, "label": "cost" }, { "features": { - "token_magnitude": 4.6, - "decimal_places": 2, + "token_magnitude": 6.9, + "decimal_places": 4, "is_negative": 0, "has_cost_keyword": 0 }, @@ -187,17 +205,17 @@ }, { "features": { - "token_magnitude": 6.51, + "token_magnitude": 1.44, "decimal_places": 2, "is_negative": 0, - "has_cost_keyword": 0 + "has_cost_keyword": 1 }, - "label": "revenue" + "label": "cost" }, { "features": { - "token_magnitude": 5.06, - "decimal_places": 2, + "token_magnitude": 2.0, + "decimal_places": 0, "is_negative": 1, "has_cost_keyword": 1 }, @@ -205,7 +223,7 @@ }, { "features": { - "token_magnitude": 2.24, + "token_magnitude": 4.53, "decimal_places": 2, "is_negative": 1, "has_cost_keyword": 1 @@ -214,17 +232,17 @@ }, { "features": { - "token_magnitude": 6.31, - "decimal_places": 2, + "token_magnitude": 3.95, + "decimal_places": 0, "is_negative": 0, "has_cost_keyword": 0 }, - "label": "revenue" + "label": "cost" }, { "features": { - "token_magnitude": 5.59, - "decimal_places": 2, + "token_magnitude": 8.98, + "decimal_places": 4, "is_negative": 0, "has_cost_keyword": 0 }, @@ -232,26 +250,26 @@ }, { "features": { - "token_magnitude": 4.26, - "decimal_places": 2, + "token_magnitude": 3.34, + "decimal_places": 0, "is_negative": 0, - "has_cost_keyword": 1 + "has_cost_keyword": 0 }, "label": "cost" }, { "features": { - "token_magnitude": 2.61, - "decimal_places": 2, - "is_negative": 1, - "has_cost_keyword": 1 + "token_magnitude": 2.8, + "decimal_places": 1, + "is_negative": 0, + "has_cost_keyword": 0 }, "label": "cost" }, { "features": { - "token_magnitude": 7.31, - "decimal_places": 2, + "token_magnitude": 6.81, + "decimal_places": 4, "is_negative": 0, "has_cost_keyword": 0 }, @@ -259,8 +277,8 @@ }, { "features": { - "token_magnitude": 6.14, - "decimal_places": 2, + "token_magnitude": 7.38, + "decimal_places": 3, "is_negative": 0, "has_cost_keyword": 0 }, @@ -268,35 +286,80 @@ }, { "features": { - "token_magnitude": 3.99, + "token_magnitude": 7.21, "decimal_places": 2, - "is_negative": 1, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 5.04, + "decimal_places": 3, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 7.49, + "decimal_places": 4, + "is_negative": 0, "has_cost_keyword": 1 }, - "label": "cost" + "label": "revenue" + }, + { + "features": { + "token_magnitude": 8.12, + "decimal_places": 3, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" }, { "features": { - "token_magnitude": 3.45, + "token_magnitude": 5.84, "decimal_places": 2, "is_negative": 0, - "has_cost_keyword": 1 + "has_cost_keyword": 0 }, - "label": "cost" + "label": "revenue" }, { "features": { - "token_magnitude": 5.07, + "token_magnitude": 6.08, "decimal_places": 2, "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "revenue" + }, + { + "features": { + "token_magnitude": 2.92, + "decimal_places": 0, + "is_negative": 0, "has_cost_keyword": 1 }, "label": "cost" }, { "features": { - "token_magnitude": 5.18, - "decimal_places": 2, + "token_magnitude": 3.29, + "decimal_places": 0, + "is_negative": 1, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 4.47, + "decimal_places": 0, "is_negative": 0, "has_cost_keyword": 1 }, @@ -304,7 +367,7 @@ }, { "features": { - "token_magnitude": 3.14, + "token_magnitude": 1.09, "decimal_places": 2, "is_negative": 1, "has_cost_keyword": 1 @@ -313,7 +376,7 @@ }, { "features": { - "token_magnitude": 5.3, + "token_magnitude": 7.35, "decimal_places": 2, "is_negative": 0, "has_cost_keyword": 0 @@ -322,25 +385,34 @@ }, { "features": { - "token_magnitude": 3.69, + "token_magnitude": 4.52, "decimal_places": 2, "is_negative": 0, - "has_cost_keyword": 1 + "has_cost_keyword": 0 }, "label": "cost" }, { "features": { - "token_magnitude": 4.92, + "token_magnitude": 3.78, + "decimal_places": 1, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 3.18, "decimal_places": 2, "is_negative": 1, - "has_cost_keyword": 1 + "has_cost_keyword": 0 }, "label": "cost" }, { "features": { - "token_magnitude": 4.1, + "token_magnitude": 1.91, "decimal_places": 2, "is_negative": 1, "has_cost_keyword": 1 @@ -349,8 +421,8 @@ }, { "features": { - "token_magnitude": 4.36, - "decimal_places": 2, + "token_magnitude": 8.92, + "decimal_places": 3, "is_negative": 0, "has_cost_keyword": 0 }, @@ -358,16 +430,16 @@ }, { "features": { - "token_magnitude": 4.59, - "decimal_places": 2, - "is_negative": 1, - "has_cost_keyword": 1 + "token_magnitude": 3.82, + "decimal_places": 1, + "is_negative": 0, + "has_cost_keyword": 0 }, "label": "cost" }, { "features": { - "token_magnitude": 5.73, + "token_magnitude": 8.54, "decimal_places": 2, "is_negative": 0, "has_cost_keyword": 0 @@ -376,7 +448,7 @@ }, { "features": { - "token_magnitude": 4.5, + "token_magnitude": 5.4, "decimal_places": 2, "is_negative": 0, "has_cost_keyword": 0 @@ -385,16 +457,16 @@ }, { "features": { - "token_magnitude": 4.58, - "decimal_places": 2, + "token_magnitude": 3.54, + "decimal_places": 0, "is_negative": 0, - "has_cost_keyword": 0 + "has_cost_keyword": 1 }, - "label": "revenue" + "label": "cost" }, { "features": { - "token_magnitude": 7.43, + "token_magnitude": 5.47, "decimal_places": 2, "is_negative": 0, "has_cost_keyword": 0 @@ -403,8 +475,17 @@ }, { "features": { - "token_magnitude": 5.23, - "decimal_places": 2, + "token_magnitude": 4.57, + "decimal_places": 1, + "is_negative": 0, + "has_cost_keyword": 1 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 6.83, + "decimal_places": 3, "is_negative": 0, "has_cost_keyword": 0 }, @@ -412,26 +493,26 @@ }, { "features": { - "token_magnitude": 4.38, - "decimal_places": 2, - "is_negative": 1, + "token_magnitude": 3.5, + "decimal_places": 1, + "is_negative": 0, "has_cost_keyword": 1 }, "label": "cost" }, { "features": { - "token_magnitude": 2.28, - "decimal_places": 2, + "token_magnitude": 8.46, + "decimal_places": 4, "is_negative": 0, - "has_cost_keyword": 1 + "has_cost_keyword": 0 }, - "label": "cost" + "label": "revenue" }, { "features": { - "token_magnitude": 7.26, - "decimal_places": 2, + "token_magnitude": 6.68, + "decimal_places": 3, "is_negative": 0, "has_cost_keyword": 0 }, @@ -439,8 +520,8 @@ }, { "features": { - "token_magnitude": 4.15, - "decimal_places": 2, + "token_magnitude": 7.62, + "decimal_places": 4, "is_negative": 0, "has_cost_keyword": 0 }, @@ -448,8 +529,8 @@ }, { "features": { - "token_magnitude": 7.79, - "decimal_places": 2, + "token_magnitude": 8.11, + "decimal_places": 3, "is_negative": 0, "has_cost_keyword": 0 }, @@ -457,35 +538,35 @@ }, { "features": { - "token_magnitude": 4.34, - "decimal_places": 2, - "is_negative": 1, - "has_cost_keyword": 1 + "token_magnitude": 2.9, + "decimal_places": 1, + "is_negative": 0, + "has_cost_keyword": 0 }, "label": "cost" }, { "features": { - "token_magnitude": 2.09, - "decimal_places": 2, + "token_magnitude": 1.05, + "decimal_places": 1, "is_negative": 1, - "has_cost_keyword": 1 + "has_cost_keyword": 0 }, "label": "cost" }, { "features": { - "token_magnitude": 4.72, - "decimal_places": 2, + "token_magnitude": 1.29, + "decimal_places": 0, "is_negative": 0, "has_cost_keyword": 0 }, - "label": "revenue" + "label": "cost" }, { "features": { - "token_magnitude": 4.28, - "decimal_places": 2, + "token_magnitude": 5.38, + "decimal_places": 3, "is_negative": 0, "has_cost_keyword": 0 }, @@ -493,8 +574,17 @@ }, { "features": { - "token_magnitude": 5.46, - "decimal_places": 2, + "token_magnitude": 1.28, + "decimal_places": 0, + "is_negative": 0, + "has_cost_keyword": 0 + }, + "label": "cost" + }, + { + "features": { + "token_magnitude": 7.15, + "decimal_places": 4, "is_negative": 0, "has_cost_keyword": 0 }, @@ -502,26 +592,26 @@ }, { "features": { - "token_magnitude": 2.39, + "token_magnitude": 2.96, "decimal_places": 2, - "is_negative": 0, - "has_cost_keyword": 1 + "is_negative": 1, + "has_cost_keyword": 0 }, "label": "cost" }, { "features": { - "token_magnitude": 5.55, + "token_magnitude": 1.34, "decimal_places": 2, "is_negative": 1, - "has_cost_keyword": 1 + "has_cost_keyword": 0 }, "label": "cost" }, { "features": { - "token_magnitude": 3.42, - "decimal_places": 2, + "token_magnitude": 3.07, + "decimal_places": 1, "is_negative": 0, "has_cost_keyword": 1 }, @@ -529,8 +619,8 @@ }, { "features": { - "token_magnitude": 4.47, - "decimal_places": 2, + "token_magnitude": 6.6, + "decimal_places": 3, "is_negative": 0, "has_cost_keyword": 0 }, @@ -538,8 +628,8 @@ }, { "features": { - "token_magnitude": 4.89, - "decimal_places": 2, + "token_magnitude": 5.71, + "decimal_places": 4, "is_negative": 0, "has_cost_keyword": 0 }, @@ -547,10 +637,10 @@ }, { "features": { - "token_magnitude": 3.36, + "token_magnitude": 3.96, "decimal_places": 2, "is_negative": 1, - "has_cost_keyword": 1 + "has_cost_keyword": 0 }, "label": "cost" } diff --git a/models/dataclass-assigner/trained_model.json b/models/dataclass-assigner/trained_model.json index c93aea7..c525d6e 100644 --- a/models/dataclass-assigner/trained_model.json +++ b/models/dataclass-assigner/trained_model.json @@ -16,62 +16,62 @@ ], "standardize": { "mean": [ - 4.721166666666666, - 2.0, - 0.31666666666666665, - 0.5 + 5.091964285714285, + 1.9285714285714286, + 0.16071428571428573, + 0.39285714285714285 ], "std": [ - 1.4051110177570392, - 1.0, - 0.4651761912317621, - 0.5 + 2.1605376477386176, + 1.360897063089832, + 0.3672672107357704, + 0.4883855118277624 ] }, "perClassLogistic": { "revenue": { "weights": [ - 1.0885461057087484, - 0.0, - -1.114543681592562, - -4.049672435624739 + 3.628190561312625, + 2.453351993679718, + -0.9109833120154356, + -1.043592320265424 ], - "bias": -0.30575517671779223, + "bias": -0.029232413457201508, "evalAccuracy": 1.0 }, "cost": { "weights": [ 0.0, 0.0, - 1.1726778962251572, - 4.351329385152503 + 1.7784969200661702, + 1.3874620259481485 ], - "bias": 0.2924522306802593, - "evalAccuracy": 1.0 + "bias": 0.4318175237342935, + "evalAccuracy": 0.7857 } }, "tableSoftmax": { "weights": [ [ - 0.8203272291509368, + 3.5823108061259985, 0.0 ], [ - 0.0, + 2.461817762080659, 0.0 ], [ - -0.5915568028849446, - 0.5915568028849448 + -0.5819139070211125, + 0.5819139070211127 ], [ - -2.392894979772866, - 2.3928949797728656 + -0.5278935777110109, + 0.5278935777110112 ] ], "bias": [ - -0.1779449481516841, - 0.1779449481516844 + -0.07160523673986202, + 0.07160523673986202 ], "evalAccuracy": 1.0 }, diff --git a/specs/trained-dataclass-classifiers.md b/specs/trained-dataclass-classifiers.md index 9fa47ce..8047697 100644 --- a/specs/trained-dataclass-classifiers.md +++ b/specs/trained-dataclass-classifiers.md @@ -6,7 +6,7 @@ trained*. This closes that gap with real, verifiable artifacts. `tools/train_dataclass_classifiers.py` trains, on a labelled fixture, **per-class LOGISTIC** heads (one-vs-rest, individually testable per glossary-term) and a **per-table SOFTMAX** (the n-ary head), -both **MONOTONE-constrained** in the declared `monotonicFeatures` — the TF-Lattice essence: the +a monotone-logistic **realisation** of the declared `tf-lattice-wide-and-deep` contract (a faithful monotonicity-satisfying model, not a full calibrated lattice), both **MONOTONE-constrained** in the declared `monotonicFeatures` — the TF-Lattice essence: the weight on a monotone feature is projected `>= 0` each gradient step, so raising that feature never lowers the class score. Deterministic (seeded, full-batch GD) so the weights — and thus the `ModelManifest.modelDigest` — are reproducible. It emits the trained weights, a conformant diff --git a/tools/train_dataclass_classifiers.py b/tools/train_dataclass_classifiers.py index 62f5354..ea0921c 100644 --- a/tools/train_dataclass_classifiers.py +++ b/tools/train_dataclass_classifiers.py @@ -43,13 +43,17 @@ def _sigmoid(z): return 1.0 / (1.0 + np.exp(-z)) -def _matrix(rows, feats): - X = np.array([[r["features"][f] for f in feats] for r in rows], dtype=float) - # standardise (store mean/std so inference reproduces); monotone direction is preserved by - # positive scaling, so the >=0 projection still means "monotone increasing in the raw feature". - mean, std = X.mean(axis=0), X.std(axis=0) +def _raw(rows, feats): + return np.array([[r["features"][f] for f in feats] for r in rows], dtype=float) + + +def _fit_standardizer(Xtrain): + # Fit standardisation on the TRAIN split ONLY (no test leakage). Positive scaling preserves + # monotone direction, so the >=0 weight projection still means "monotone increasing in the raw + # feature". Stored so inference/validation reproduce exactly. + mean, std = Xtrain.mean(axis=0), Xtrain.std(axis=0) std[std == 0] = 1.0 - return (X - mean) / std, mean, std + return mean, std def _train_logistic(X, y, mono_idx): @@ -88,11 +92,13 @@ def main() -> int: label_idx = {"urn:srcos:glossary:revenue": 0, "urn:srcos:glossary:cost": 1} y_name = ["revenue", "cost"] - X, mean, std = _matrix(rows, feats) + Xraw = _raw(rows, feats) y = np.array([0 if r["label"] == "revenue" else 1 for r in rows], dtype=float) - # 80/20 split (deterministic order in the fixture) + # 80/20 split (deterministic order in the fixture); standardiser FIT ON TRAIN ONLY. cut = int(len(rows) * 0.8) + mean, std = _fit_standardizer(Xraw[:cut]) + X = (Xraw - mean) / std Xtr, Xte, ytr, yte = X[:cut], X[cut:], y[:cut], y[cut:] # per-class one-vs-rest logistic (class 0 = revenue as positive; class 1 = cost as positive) @@ -122,8 +128,11 @@ def main() -> int: MANIFEST.write_text(json.dumps({ "id": MODEL_URN, "type": "ModelManifest", "specVersion": "2.1.0", - "modelDigest": digest, "displayName": "dataclass-assigner (monotone wide-and-deep)", - "architecture": "monotone-logistic-wide-and-deep", "format": "onnx", "quantization": "none", + "modelDigest": digest, "displayName": "dataclass-assigner (monotone logistic)", + # Honest about what was actually trained: a linear monotone-logistic realisation of the + # DataClass classifier.kind 'tf-lattice-wide-and-deep' contract (satisfies the monotonicity + # constraint; it is NOT a full TF-Lattice calibrated-lattice model). + "architecture": "monotone-logistic", "format": "onnx", "quantization": "none", "license": "Apache-2.0", "signature": {"algorithm": "ed25519", "keyId": "srcos-model-signing-2026", "signatureDigest": "sha256:" + "b" * 64}, diff --git a/tools/validate_trained_classifiers.py b/tools/validate_trained_classifiers.py index 83dce87..d470ea8 100644 --- a/tools/validate_trained_classifiers.py +++ b/tools/validate_trained_classifiers.py @@ -68,12 +68,14 @@ def main() -> int: W = np.array(model["tableSoftmax"]["weights"]); b = np.array(model["tableSoftmax"]["bias"]) recomputed = float((( Xte @ W + b).argmax(axis=1) == yte).mean()) claimed = load(RUN_EVAL)["workload"]["params"]["value"] - if abs(recomputed - claimed) > 1e-6: + # Compare at the stored precision (claims are recorded rounded to 4 decimals); a real + # discrepancy is >> this, an inflated claim (+0.1) is far outside it. + if round(recomputed, 4) != round(claimed, 4): FAILURES.append(f"eval accuracy not reproduced: recomputed {recomputed} != claimed {claimed}") else: CHECKS["eval:reproduced-from-weights"] = True # teeth on the teeth: an inflated claim must NOT reproduce (guards against trusting the number) - if abs(recomputed - (claimed + 0.1)) <= 1e-6: + if round(recomputed, 4) == round(claimed + 0.1, 4): FAILURES.append("reproduction check is not discriminating (would accept an inflated claim)") else: CHECKS["eval:inflated-claim-would-fail"] = True @@ -84,26 +86,37 @@ def main() -> int: pc = model["perClassLogistic"][name] w = np.array(pc["weights"]); bb = pc["bias"] acc = float((((1 / (1 + np.exp(-(Xte @ w + bb)))) >= 0.5) == (yte == ci)).mean()) - if abs(acc - pc["evalAccuracy"]) > 1e-6: + if round(acc, 4) != round(pc["evalAccuracy"], 4): FAILURES.append(f"per-class '{name}' accuracy not reproduced ({acc} != {pc['evalAccuracy']})") ok_pc = False if ok_pc: CHECKS["per-class:reproduced"] = True - # 4. MONOTONE in the declared features: raise each monotone feature -> revenue logit never drops. + # 4. MONOTONE in the declared features, for EVERY per-class head: raise each monotone feature -> + # the class logit never drops. Checked on both revenue and cost heads (not just one). mono = spec["monotonicFeatures"] - wr = np.array(model["perClassLogistic"]["revenue"]["weights"]); br = model["perClassLogistic"]["revenue"]["bias"] - base_logit = Xte @ wr + br mono_ok = True - for f in mono: - bumped = [dict(r, features={**r["features"], f: r["features"][f] + 1.0}) for r in test] - up_logit = _features(bumped, feats, mean, std) @ wr + br - if np.any(up_logit < base_logit - 1e-9): - FAILURES.append(f"model is NOT monotone in '{f}' — raising it lowered the revenue score") - mono_ok = False + for name in ("revenue", "cost"): + w = np.array(model["perClassLogistic"][name]["weights"]); bb = model["perClassLogistic"][name]["bias"] + base = Xte @ w + bb + for f in mono: + bumped = [dict(r, features={**r["features"], f: r["features"][f] + 1.0}) for r in test] + if np.any((_features(bumped, feats, mean, std) @ w + bb) < base - 1e-9): + FAILURES.append(f"'{name}' head NOT monotone in '{f}' — raising it lowered the score") + mono_ok = False if mono_ok: CHECKS["monotone:constraint-holds"] = True + # 4b. The monotone constraint must actually BIND — at least one monotone feature carries a + # non-trivial positive weight on the revenue head — otherwise the monotone check is vacuous + # (a constant/ignored feature would pass trivially). Guards against a toothless fixture. + wr = np.array(model["perClassLogistic"]["revenue"]["weights"]) + mono_w = [wr[feats.index(f)] for f in mono] + if max(mono_w) < 0.1: + FAILURES.append(f"monotone constraint does not bind — all monotone weights ~0 {mono_w} (vacuous)") + else: + CHECKS["monotone:constraint-binds"] = True + # 5. DataClass classifier refs resolve to the emitted artifacts. clf = load(DATACLASS)["classifier"] ids = {load(MANIFEST)["id"], load(RUN_TRAIN)["id"], load(RUN_EVAL)["id"]}