Skip to content

v0.2.0 (preview)

Pre-release
Pre-release

Choose a tag to compare

View all tags
@github-actions github-actions released this 15 Nov 03:42
· 3 commits to main since this release
v0.2.0
1c2d0cc

TriggerMap v0.2.0 (preview)

Still a preview, not a quality-gated release. The acceptance gates in
docs/targets.md are judged on hand-labelled test records, and only 4 exist.
No more hand labels will come. The declared held-out evaluation therefore reports the gates as
not judgeable and gives test results as agreement with model drafts.

The default model does not change. v0.2.0 serves the same CRF bundle as v0.1.0, and the
evaluation, reproducibility, demo and documentation work since v0.1.0 ships with it.

What is in it

  • Serving image: built by release.yml from this tag with EXTRAS="serve baselines". Its
    digest is in the attached release-image.json.
  • Default model: CRF crf-batches-01-02, unchanged from v0.1.0. Bundle weights SHA-256
    d1f8cb5a94b0ba4540c585d902543e13e8b4252b9c21891f29bf6381b0c07056, archive
    crf-batches-01-02.tgz (attached, SHA-256
    65677a7413e02a8da4ba3b920ac7eb8e02a8cba874876d93810d4ae5fc76d29a).
  • Review policy: review-policy-0, uncalibrated, so every prediction is flagged Review
    required    .
  • Optional transformer: the run-3 DeBERTa-v3-small is a separate model-only pre-release,
    model-deberta-3-run3.
    It is not served by this image.

Changes since v0.1.0

  • Held-out test evaluation, declared by the owner on 2026-11-15 and run once
    (reports/test-evaluation/; every run is in
    test-access-log.md).
  • Two-reference scoring: a second model draft of batch-07, drafter-consistency
    measurement, and triggermap multiref for scoring systems against both drafts
    (reports/drafter-consistency/,
    reports/batch-07-comparison/).
  • Exported runtimes on the trained DeBERTa: FP32 ONNX is exact; dynamic INT8 fails the
    ≤ 1-point gate (reports/onnx-int8/).
  • Prospective check: 259 advisories published after the snapshot, behaviour only
    (reports/prospective/).
  • Reproducibility: scripts/reproduce/clean_clone.sh rebuilds the corpus, retrains the CRF
    and regenerates the reports byte for byte on macOS arm64 and Linux x86-64. A git-version
    dependence in the snapshot manifest was fixed
    (reproducibility.md).
  • Public demo: static review pages for an ordinary case, a hard case and an abstention, at
    https://professor3333.github.io/TriggerMap/.
  • Attribution: advisory-text footers in the review page and the app now link the CC BY 4.0
    licence and say how the text was modified.
  • Documentation: model-card downloads with checksums, and README commands rechecked against
    the project.

What was measured

Test split, declared evaluation: 146 model-drafted records, so agreement, not accuracy:

System Typed-condition F1 Span precision Span recall False-prerequisite rate
CRF crf-batches-01-02 (default) 0.193 [0.154, 0.238] 0.263 0.219 0.482
DeBERTa-v3-small run 3 0.154 [0.125, 0.186] 0.114 0.393 0.650
rules-2 0.048 [0.021, 0.082] 0.101 0.053 0.646
  • CRF against DeBERTa: DeBERTa − CRF typed F1 −0.039 [−0.080, −0.000], clear but only just.
  • Hand-labelled test records: on the 4 records (7 conditions), no difference is clear.
  • Dev: agreement on 172 model-drafted records, CRF typed F1 0.145 [0.105, 0.185]. Hand-labelled
    accuracy on 10 documents: 0.370 [0.000, 0.640].
  • Serving, measured for v0.1.0 on the 2 vCPU / 4 GB target host (same model): p95
    0.062 s / 0.145 s, 1,000 of 1,000 load-test requests succeeded
    (benchmarks.md).

Known limitations

  • Labels after the pilot are model drafts. Two drafting runs of the same documents agree at
    only 0.375 typed F1, so agreement scores rest on unstable references.
  • Group exact match is 0 for every system, and EXCEPTION conditions are rarely found.
  • New advisories are much longer than the annotated ones (median 515 words against 79 on dev),
    so the measured scores may not carry over to current advisories.
  • No usability study has been run.

Release checks (2026-11-15)

  • release.yml on the tag passed lint, types and tests, then built the image, passed the
    container smoke test and pushed it: ghcr.io/professor3333/triggermap@sha256:cc1578f9a488a09da717a298db5f0018350272fb5b540157f3d1016b04f376ce
    (from commit 1c2d0cc). The image can be pulled without logging in.
  • The attached crf-batches-01-02.tgz is byte-identical to the v0.1.0 asset (SHA-256 above).
  • Rollback drill on local Docker: v0.1.0 → v0.2.0 → v0.1.0. Every step was verified (ready check,
    extraction, idempotent review save, retrieval, export), reviews were kept, and the run took
    51 seconds.

Deploying it

Follow docs/deployment.md with the image digest from
release-image.json:

gh release download v0.2.0 --pattern crf-batches-01-02.tgz --pattern release-image.json
shasum -a 256 crf-batches-01-02.tgz      # must match the SHA-256 above
tar -xzf crf-batches-01-02.tgz
uv run triggermap release create v0.2.0 --image ghcr.io/professor3333/triggermap@sha256:<digest> \
    --model crf --model-dir crf-batches-01-02
uv run triggermap release activate v0.2.0 --apply

Rolling back to v0.1.0 restores its image and its model bundle together:
uv run triggermap release activate v0.1.0 --apply.

Assets 4
Loading