Skip to content

Model: DeBERTa-v3-small, run 3 (not the default)

Pre-release
Pre-release

Choose a tag to compare

View all tags
@professor3333 professor3333 released this 15 Nov 02:46
· 8 commits to main since this release
model-deberta-3-run3
cf5e4c6

Model-only pre-release. Not the default model. The serving release is still v0.1.0 with the CRF crf-batches-01-02. This pre-release makes the fine-tuned DeBERTa-v3-small from run 3 downloadable. It is the leading transformer candidate, but under the pre-set switching rule it does not replace the CRF.

Asset

File Size SHA-256
deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2.tgz 422,269,499 bytes 45b34cabfa51bc58e2eed6e4d0c6803693421b9ec93c8c572993cd2217caf4a9

The archive contains:

  • the model bundle (weights, tokenizer, label inventory, metadata with provenance);
  • the Platt calibration fitted on the dev selection half (calibration/deberta-platt.yaml);
  • a NOTICE and a README.md.

model.safetensors SHA-256: 1d09c400a41da2c2e92b2fb21758c41c6f84c6fa1a148f3c3b2c5f1879b7baf6.

sha256sum deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2.tgz   # compare with the table
tar xzf deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2.tgz
uv run triggermap predict --model transformer \
    --model-dir deberta-3-base-bio_weightingsqrt-learning_rate5e-05-seed2 --out predictions.jsonl

Provenance

  • Base model: microsoft/deberta-v3-small at revision a36c739020e01763fe789b4b85e2df55d6180012, MIT License (Microsoft). The licence text is in NOTICE.
  • Recipe: run 3: sqrt span-tag weights, learning rate 5e-5, seed 2, at most 20 epochs with at least 6, patience 3, on one Kaggle T4.
  • Training data: 239 train-split records of pilot-2, batch-01 and batch-02. Of these, 7 are manual, 219 model_draft_accepted and 13 model_draft_revised. Training data is derived from the GitHub Advisory Database (CC BY 4.0, modified).
  • Model selection: early stopping used only the selection half of the model-drafted dev batches 03 and 06 (splits/selection-v1).
  • Reproduction: loaded on CPU, the bundle reproduces its committed predictions on the 60 batch-06 documents with identical conditions, and confidences within 0.0001.

Measured (dev only; agreement with model drafts, not accuracy, unless stated)

Result
Held-out dev half (61 documents), typed F1 0.185 [0.145, 0.235] (CRF 0.127; paired +0.058, not clear)
Hand-labelled dev (10 documents, 17 conditions), typed F1 (accuracy) 0.346
batch-07, typed F1 against draft 1 / draft 2 0.186 / 0.245 (CRF 0.158 / 0.136)
batch-07, invented prerequisites against both drafts 0.327 (CRF 0.380, not clearly different)
ONNX FP32 export identical to PyTorch on 372 documents
ONNX dynamic INT8 fails the ≤ 1-point gate (−2.1 points on hand-labelled dev)

Details: reports/neural/run-3/, reports/calibration/, reports/batch-07-comparison/ and reports/onnx-int8/ in the repository. No test-split result exists for this model.

Limitations

  • Most training labels are model drafts, not hand labels. Only 22 records in the whole dataset are hand-labelled.
  • These are suggested checklist items for a person to review. They are not a judgement of whether a deployment is vulnerable.
  • Use PyTorch or FP32 ONNX. Dynamic INT8 changes most predictions.
Assets 3
Loading