Skip to content

Amada 4.0

Latest
Latest

Choose a tag to compare

View all tags
@DevLiamDb DevLiamDb released this 10 Nov 16:04
· 8 commits to main since this release
v4.0
2472c1d

Messy real-life writing, "the text does not say" and ordered scales. Starts from the 3.1 weights: two short
fine-tunes (1 epoch each, T4 GPU), then recalibrated. Shown as "Amada 4" on the website.

  • Added: MASSIVE 1.1 (Amazon, CC BY 4.0, 60 voice-assistant intents, 12 languages) and 88milSMS (Université
    Paul-Valéry Montpellier, CC BY 4.0, real French text messages, questions written by GPT-6 Luna, half in French).
  • Added: half of the replayed data rewritten with keyboard typos, swapped letters, lost accents, SMS spelling and
    homophones, with the same answers (numbers and negations are never touched).
  • Added: answer-type questions (the TREC skill), near-identical support intents (the Banking77 skill, banking left
    out on purpose), scores written for one precise level, more long texts. TREC and Banking77 themselves stay out of training.
  • Note: after 4.0, MASSIVE scores are no longer zero-shot.
  • Added (second fine-tune): 3,731 short everyday texts in 12 languages written by GPT-6 Luna, each with yes / no
    questions whose answer is in the text or is not ("the text does not say"); 55 % of them rewritten with typos.
  • Added: an ordinal loss on score questions, so being one level off costs less than being two levels off.
  • Changed: temperatures refitted. Questions whose answer is "the text does not say" no longer pull the yes / no
    temperature (their yes / no target is 0.5).
  • Results, same inputs and code for 3.1 and 4.0 (10 November 2026):
    • "The text does not say" (yes / no / not stated, 939 held-out questions): 0.917 -> 0.963; messy writing 0.889 -> 0.954.
      These texts also helped fit the calibration: a diagnostic, not an independent test.
    • Messy writing, held-out half of the short tests: yes / no 0.953 -> 0.960 in English, 0.937 -> 0.967 in 11 languages.
    • Short tests, held-out half, English: yes / no 0.964 -> 0.966, choice 0.924 -> 0.913, score 0.752 -> 0.744
      (within one level: 0.955 -> 0.956). ECE yes / no 0.009, choice 0.016.
    • system-one-benchmark: TREC 0.390 -> 0.523, SST-2 0.900 -> 0.913, MASSIVE English 0.707 -> 0.780.
    • 11 languages (the non-English training languages): MASSIVE intents 0.428 -> 0.834 (no longer zero-shot), yes / no 0.839 -> 0.841, Belebele 0.568 -> 0.554.
    • Public datasets, zero-shot: SST-2 0.897, AG News 0.803, Banking77 0.586, emotion 0.447, tweet_topic 0.718,
      prompt injection 0.845, When2Call 0.344.
    • Speed (12 November 2026, laptop CPU, median): 41 ms for one line of text, 54 ms for a short yes / no note,
      161 ms for a reading passage, 390 ms for 60 intents. The same as 3.1.
  • Known issues: Banking77 (0.611 -> 0.586), emotion (0.471 -> 0.447) and tweet_topic (0.740 -> 0.718) are lower than
    in 3.1. Exact score level -0.8 point in English. Very long texts not improved.
  • Note: the fresh 3.1 run gives 0.650 on system-one-benchmark Banking77 instead of the 0.603 published on 8 November
    (same 300 rows, identical on CPU and GPU); the cause of the old number was not found.
  • Added (2026-11-12): amada-4.0-int4.zip, the same model with 4-bit weights (196 MB instead of 1.25 GB, needs
    onnxruntime 1.20 or newer). Same 43,799 benchmark items as 4.0, compared answer by answer: SST-2 0.890, AG News 0.793
    (-1.0), Banking77 0.574 (-1.2), emotion 0.464 (+1.7), tweet_topic 0.711, prompt injection 0.810 (-3.4, only 116 texts: within chance), When2Call 0.340;
    11 languages: yes / no 0.837, Belebele 0.559, MASSIVE 0.829. Short tests: English 0.892 vs 0.896, score 0.742 vs 0.759,
    11 languages 0.983 vs 0.985; held-out ECE yes / no 0.008, choice 0.014, score 0.031. About 340 MB of RAM after loading
    instead of 1.4 GB, about the same speed. Activations stay in FP32.

Install: unzip into a Release/ folder next to play.py, then python play.py --version 4.0 (or --version 4.0-int4 for the smaller file). Keep tokenizer/ next to amada_model.onnx.

License: MIT (commercial use and modified versions allowed, keep the notice). Base model and dataset notices: NOTICE.md.

Checksums (check the file with Get-FileHash <file> in PowerShell or sha256sum <file>):

SHA-256  72c3f9e3ed8cc9cbeda09cd004360eb7e8bb44b6d326a9d4cf027486207345b5  amada-4.0-int4.zip
SHA-256  cfa9786b169505ad643911e79ad26af65177ad5f9b7bcc6734433a03ef25e250  amada-4.0.zip
Assets 4
Loading