v0.2.0 (preview)

TriggerMap v0.2.0 (preview)

Still a preview, not a quality-gated release. The acceptance gates in
docs/targets.md are judged on hand-labelled test records, and only 4 exist.
No more hand labels will come. The declared held-out evaluation therefore reports the gates as
not judgeable and gives test results as agreement with model drafts.

The default model does not change. v0.2.0 serves the same CRF bundle as v0.1.0, and the
evaluation, reproducibility, demo and documentation work since v0.1.0 ships with it.

What is in it

• Serving image: built by release.yml from this tag with EXTRAS="serve baselines". Its
  digest is in the attached release-image.json.
• Default model: CRF crf-batches-01-02, unchanged from v0.1.0. Bundle weights SHA-256
  d1f8cb5a94b0ba4540c585d902543e13e8b4252b9c21891f29bf6381b0c07056, archive
  crf-batches-01-02.tgz (attached, SHA-256
  65677a7413e02a8da4ba3b920ac7eb8e02a8cba874876d93810d4ae5fc76d29a).
• Review policy: review-policy-0, uncalibrated, so every prediction is flagged Review
  required.
• Optional transformer: the run-3 DeBERTa-v3-small is a separate model-only pre-release,
  model-deberta-3-run3.
  It is not served by this image.

Changes since v0.1.0

• Held-out test evaluation, declared by the owner on 2026-11-15 and run once
  (reports/test-evaluation/; every run is in
  test-access-log.md).
• Two-reference scoring: a second model draft of batch-07, drafter-consistency
  measurement, and triggermap multiref for scoring systems against both drafts
  (reports/drafter-consistency/,
  reports/batch-07-comparison/).
• Exported runtimes on the trained DeBERTa: FP32 ONNX is exact; dynamic INT8 fails the
  ≤ 1-point gate (reports/onnx-int8/).
• Prospective check: 259 advisories published after the snapshot, behaviour only
  (reports/prospective/).
• Reproducibility: scripts/reproduce/clean_clone.sh rebuilds the corpus, retrains the CRF
  and regenerates the reports byte for byte on macOS arm64 and Linux x86-64. A git-version
  dependence in the snapshot manifest was fixed
  (reproducibility.md).
• Public demo: static review pages for an ordinary case, a hard case and an abstention, at
  https://professor3333.github.io/TriggerMap/.
• Attribution: advisory-text footers in the review page and the app now link the CC BY 4.0
  licence and say how the text was modified.
• Documentation: model-card downloads with checksums, and README commands rechecked against
  the project.

What was measured

Test split, declared evaluation: 146 model-drafted records, so agreement, not accuracy:

System	Typed-condition F1	Span precision	Span recall	False-prerequisite rate
CRF crf-batches-01-02 (default)	0.193 [0.154, 0.238]	0.263	0.219	0.482
DeBERTa-v3-small run 3	0.154 [0.125, 0.186]	0.114	0.393	0.650
rules-2	0.048 [0.021, 0.082]	0.101	0.053	0.646

• CRF against DeBERTa: DeBERTa − CRF typed F1 −0.039 [−0.080, −0.000], clear but only just.
• Hand-labelled test records: on the 4 records (7 conditions), no difference is clear.
• Dev: agreement on 172 model-drafted records, CRF typed F1 0.145 [0.105, 0.185]. Hand-labelled
  accuracy on 10 documents: 0.370 [0.000, 0.640].
• Serving, measured for v0.1.0 on the 2 vCPU / 4 GB target host (same model): p95
  0.062 s / 0.145 s, 1,000 of 1,000 load-test requests succeeded
  (benchmarks.md).

Known limitations

• Labels after the pilot are model drafts. Two drafting runs of the same documents agree at
  only 0.375 typed F1, so agreement scores rest on unstable references.
• Group exact match is 0 for every system, and EXCEPTION conditions are rarely found.
• New advisories are much longer than the annotated ones (median 515 words against 79 on dev),
  so the measured scores may not carry over to current advisories.
• No usability study has been run.

Release checks (2026-11-15)

• release.yml on the tag passed lint, types and tests, then built the image, passed the
  container smoke test and pushed it: ghcr.io/professor3333/triggermap@sha256:cc1578f9a488a09da717a298db5f0018350272fb5b540157f3d1016b04f376ce
  (from commit 1c2d0cc). The image can be pulled without logging in.
• The attached crf-batches-01-02.tgz is byte-identical to the v0.1.0 asset (SHA-256 above).
• Rollback drill on local Docker: v0.1.0 → v0.2.0 → v0.1.0. Every step was verified (ready check,
  extraction, idempotent review save, retrieval, export), reviews were kept, and the run took
  51 seconds.

Deploying it

Follow docs/deployment.md with the image digest from
release-image.json:

gh release download v0.2.0 --pattern crf-batches-01-02.tgz --pattern release-image.json
shasum -a 256 crf-batches-01-02.tgz      # must match the SHA-256 above
tar -xzf crf-batches-01-02.tgz
uv run triggermap release create v0.2.0 --image ghcr.io/professor3333/triggermap@sha256:<digest> \
    --model crf --model-dir crf-batches-01-02
uv run triggermap release activate v0.2.0 --apply

Rolling back to v0.1.0 restores its image and its model bundle together:
uv run triggermap release activate v0.1.0 --apply.

Model: DeBERTa-v3-small, run 3 (not the default)

Model-only pre-release. Not the default model. The serving release is still v0.1.0 with the CRF crf-batches-01-02. This pre-release makes the fine-tuned DeBERTa-v3-small from run 3 downloadable. It is the leading transformer candidate, but under the pre-set switching rule it does not replace the CRF.

Asset

File	Size	SHA-256
deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2.tgz	422,269,499 bytes	45b34cabfa51bc58e2eed6e4d0c6803693421b9ec93c8c572993cd2217caf4a9

The archive contains:

• the model bundle (weights, tokenizer, label inventory, metadata with provenance);
• the Platt calibration fitted on the dev selection half (calibration/deberta-platt.yaml);
• a NOTICE and a README.md.

model.safetensors SHA-256: 1d09c400a41da2c2e92b2fb21758c41c6f84c6fa1a148f3c3b2c5f1879b7baf6.

sha256sum deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2.tgz   # compare with the table
tar xzf deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2.tgz
uv run triggermap predict --model transformer \
    --model-dir deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2 --out predictions.jsonl

Provenance

• Base model: microsoft/deberta-v3-small at revision a36c739020e01763fe789b4b85e2df55d6180012, MIT License (Microsoft). The licence text is in NOTICE.
• Recipe: run 3: sqrt span-tag weights, learning rate 5e-5, seed 2, at most 20 epochs with at least 6, patience 3, on one Kaggle T4.
• Training data: 239 train-split records of pilot-2, batch-01 and batch-02. Of these, 7 are manual, 219 model_draft_accepted and 13 model_draft_revised. Training data is derived from the GitHub Advisory Database (CC BY 4.0, modified).
• Model selection: early stopping used only the selection half of the model-drafted dev batches 03 and 06 (splits/selection-v1).
• Reproduction: loaded on CPU, the bundle reproduces its committed predictions on the 60 batch-06 documents with identical conditions, and confidences within 0.0001.

Measured (dev only; agreement with model drafts, not accuracy, unless stated)

	Result
Held-out dev half (61 documents), typed F1	0.185 [0.145, 0.235] (CRF 0.127; paired +0.058, not clear)
Hand-labelled dev (10 documents, 17 conditions), typed F1 (accuracy)	0.346
batch-07, typed F1 against draft 1 / draft 2	0.186 / 0.245 (CRF 0.158 / 0.136)
batch-07, invented prerequisites against both drafts	0.327 (CRF 0.380, not clearly different)
ONNX FP32 export	identical to PyTorch on 372 documents
ONNX dynamic INT8	fails the ≤ 1-point gate (−2.1 points on hand-labelled dev)

Details: reports/neural/run-3/, reports/calibration/, reports/batch-07-comparison/ and reports/onnx-int8/ in the repository. No test-split result exists for this model.

Limitations

• Most training labels are model drafts, not hand labels. Only 22 records in the whole dataset are hand-labelled.
• These are suggested checklist items for a person to review. They are not a judgement of whether a deployment is vulnerable.
• Use PyTorch or FP32 ONNX. Dynamic INT8 changes most predictions.

v0.1.0 (preview)

Serving image: ghcr.io/professor3333/triggermap@sha256:0b9e2f30c328959d0ed8fe82677757b6276bef89ce322ebcc6b0d07ffce9adac (also in release-image.json).

A preview release, not a quality-gated one. The acceptance gates in
docs/targets.md are judged on hand-labelled test records. Only 4 exist,
because every annotation batch after the pilot was submitted from model drafts. No gate has
been evaluated, and the test split has not been read
(docs/test-access-log.md has no entries).

What is in it

• Serving image: built by release.yml from this tag with EXTRAS="serve baselines", so it
  serves both the rules model and the CRF. Its digest is in the attached release-image.json.
• Default model: crf-batches-01-02, a CRF tagger with logistic-regression category, role and
  group classifiers. Its bundle weights SHA-256 is
  d1f8cb5a94b0ba4540c585d902543e13e8b4252b9c21891f29bf6381b0c07056.
  • Trained on 239 train-split records: 7 hand-labelled, 232 model drafts.
  • Preprocessing version 1.
• Review policy: review-policy-0, uncalibrated, so every prediction is flagged
  Review required. Thresholds fitted on model drafts measure agreement, not accuracy, so
  none is applied.

What was measured (dev split only)

• Agreement with model drafts, held-out half (61 documents; not accuracy):
  • CRF typed F1 0.127 [0.054, 0.201];
  • DeBERTa-v3-small 0.185 [0.145, 0.235]; the paired difference is not clear.
• Hand-labelled accuracy (10 dev documents): CRF typed F1 0.370 [0.000, 0.640].
• Why the CRF, not DeBERTa: after calibration DeBERTa keeps more agreeing conditions at the
  same precision. It is not clearly more precise, and it does not invent fewer prerequisites,
  which was the pre-set condition for switching
  (reports/calibration/).
• Deployment checks, local Docker on 2026-11-06:
  • the container smoke test passed;
  • app verify passed: ready check against the manifest, extraction, idempotent review save,
    retrieval and export;
  • the rollback drill r1 (rules) → v0.1.0 (CRF) → r1 passed in 15 seconds, with
    reviews kept.
• Not measured: the 2 vCPU / 4 GB target-host benchmarks.

Model bundle

crf-batches-01-02.tgz (attached to the release) holds the bundle directory
crf-batches-01-02/, with tagger.crfsuite, classifiers.joblib and metadata.json.

• Archive SHA-256: 65677a7413e02a8da4ba3b920ac7eb8e02a8cba874876d93810d4ae5fc76d29a.
• Bundle weights digest: d1f8cb5a…07056, which release.json records and the app checks
  at start-up.
• Trust: classifiers.joblib is a pickle, so load it only from this release, after
  checking the SHA-256.
• License: the CRF's feature vocabulary is derived from GitHub Advisory Database text and
  the TriggerMap annotations (CC BY 4.0, modified). Attribution and licence details are in
  annotations/README.md.

Deploying it

Follow docs/deployment.md with the image digest from
release-image.json:

gh release download v0.1.0 --pattern crf-batches-01-02.tgz
shasum -a 256 crf-batches-01-02.tgz      # must match the SHA-256 above
tar -xzf crf-batches-01-02.tgz
uv run triggermap release create v0.1.0 --image ghcr.io/professor3333/triggermap@sha256:<digest> \
    --model crf --model-dir crf-batches-01-02