Skip to content

Amada 3.1

Choose a tag to compare

@DevLiamDb DevLiamDb released this 08 Nov 05:03
· 8 commits to main since this release

Short fine-tune of 3.0 to tell near-neighbour labels apart, then recalibrated.

  • Added: option lists whose distractors are the gold label's nearest labels (embedding similarity) instead of random ones.
  • Added: CLINC150, real user requests (150 intents + out-of-scope). Banking77 is still never used for training.
  • Changed: 1 epoch (3,440 steps, about 3 h on a T4 GPU), 35k replay of v3 data against forgetting.
  • Changed: temperatures refitted on v3.1 validation + half of the bake-off sets; the other half stays unseen for the report.
  • Results, public datasets (3.0 -> 3.1): Banking77 0.605 -> 0.611, AG News 0.790 -> 0.800, emotion 0.447 -> 0.471,
    tweet_topic 0.722 -> 0.740, prompt injection 0.810 -> 0.845, SST-2 0.893 -> 0.899, When2Call 0.287 -> 0.324.
  • Results, short tests: yes/no 0.961, choice 0.915, score 0.760 (-1 point), 11 languages 0.984.
    ECE yes/no 0.004, choice 0.016 (3.0.1: 0.022), score 0.032 (3.0.1: 0.021). CPU about 117 ms per text.
  • Decision models (system-one-benchmark): SST-2 0.900, TREC 0.390, Banking77 0.603, MASSIVE 0.707 English / 0.575
    other languages. Jev 1.13 and Kev-0.8B are ahead; Laya, GLiNER2.5-Decide and Von are behind outside English.
  • Known issues: the near-neighbour lists barely helped Banking77 (goal 0.75 missed). TREC is weak. Messy writing
    costs 1.2 points in English and 2.2 points in other languages.

Smaller download (added 2026-11-08): amada-3.1-int4.zip (188 MB) is the same model with 4-bit weights. Same tests, same laptop: accuracy 0.897 vs 0.899 in English, 0.983 vs 0.984 in 11 languages, choice ECE 0.017 vs 0.016, peak RAM 1.15 GB vs 3.2 GB. Needs onnxruntime 1.20 or newer. Run: python play.py --version 3.1-int4.


Install: unzip into a Release/ folder next to play.py, then python play.py --version 3.1. Keep tokenizer/ next to amada_model.onnx.

License: CC BY-NC-ND 4.0 (non-commercial, no sharing of modified versions). Base model notices: NOTICE.md.