Caution
This is not a medical device and must never be used on real people.
The model imitates a synthetic simulator of knee physiotherapy assessment. It has no clinical validation of any kind, it is not a medical device under Regulation (EU) 2017/745, and the sensitivity/specificity values behind its training data are unvalidated approximations invented for this exercise.
Every number below measures fidelity to that simulator. None measures clinical validity. → NOTICE
The model is not the point. The methodology is.
Most published fine-tunes report a single accuracy number with nothing to compare it against. This one ships a measured baseline, a measured theoretical ceiling, both joint and conditional accuracy, and runtime guards that make a specific class of silent evaluation failure impossible.
Those guards exist because an earlier run of this exact project produced a confident,
well-formatted, completely invalid result — nine metrics reading +0.0% because the
harness had evaluated the untrained model twice. That story, and the second bug it
uncovered, is written up in docs/postmortem.md. It is the most
useful file here.
Trained on 3,200 synthetic examples for 2 epochs (400 steps, ~5h40m on a free Colab T4).
Training loss 0.0297 / validation loss 0.0298 at step 400, tracking each other
monotonically throughout — no overfitting.
Full metric table
| Metric | Base | Fine-tuned | Ceiling |
|---|---|---|---|
| Valid JSON | 97.0% | 100% | 100% |
| Schema valid | 96.5% | 100% | 100% |
| Leading hypothesis vs. generator target | 0.0% | 90.5% | 100% |
| Leading hypothesis vs. sampled ground truth | 0.0% | 60.0% | ~67% |
| Red-flag recall | 100% | 100% (15/15) | 100% |
| Red-flag precision | 9.2% | 100% | 100% |
| Referral action correct | 23.3% | 100% | 100% |
| Confidence MAE | 17.1% | 6.8% | 0% |
| Evidence grounded in input | 91.0% | 83.9% | 83.8% |
Definitions, denominators, and the derivation of both ceilings: docs/evaluation.md
How to read this table honestly — four caveats that matter more than the numbers
The 0.0% baseline is a vocabulary mismatch, not a reasoning failure. The base model
produces well-formed JSON with sensible clinical content — it simply invents free-text
condition names (sindrome del ginocchio funzionale) instead of using this project's
eleven enum labels. Exact-match scoring therefore yields zero. The fine-tune's genuine
contributions are the label vocabulary, the calibration, and the red-flag precision.
60.0% against a ~67% ceiling is the headline, not 90.5%. The generator samples signs from imperfect sensitivity/specificity distributions, so in roughly a third of cases the findings honestly point away from the sampled condition. No model can beat ~67% without memorising. Landing at 60% captures ~90% of the attainable signal — and staying below the ceiling is the evidence of generalisation.
The drop in evidence grounding is convergence, not regression. 83.9% sits exactly on
the 83.8% ceiling measured against the gold targets. The base model scores higher because
it copies input phrasing verbatim; the fine-tuned model adopted the targets' compressed
style (lachman: positivo), which the lexical matcher under-credits.
Red-flag metrics rest on 15 positive cases. Both read 100%, but at n=15 the 95% confidence interval on recall runs roughly 78–100%. Never quote "100%" without the n.
Open question — the control nobody has run yet
The base model already emits 97% valid JSON and fails only on the label vocabulary. Does a few-shot prompt listing the eleven enum labels close most of the gap with no training at all?
Unanswered. It is the correct control for this project's central claim, it costs ~30 minutes with the harness already in the repo, and either outcome is worth publishing. See issue #1.
flowchart LR
A["Synthetic generator<br/><i>priors · sens/spec · Naive Bayes</i>"] --> B["4,000 examples<br/>3200 / 400 / 400"]
B --> C["Baseline eval<br/><i>persisted to disk</i>"]
C --> D["LoRA r=16<br/><i>QLoRA 4-bit · T4</i>"]
D --> E{"Guards"}
E -->|"training completed<br/>+ LoRA B norm != 0"| F["Post-training eval"]
E -->|"otherwise"| G["RuntimeError"]
F --> H["Comparison<br/><i>vs baseline + ceilings</i>"]
style G fill:#cf5464,stroke:#cf5464,color:#fff
style E fill:#d99a2b,stroke:#d99a2b,color:#fff
style F fill:#17a398,stroke:#17a398,color:#fff
style H fill:#17a398,stroke:#17a398,color:#fff
The two guards are the load-bearing part. LoRA initialises its B matrices to zero by
construction, so a non-zero norm is physical proof that gradients were applied —
independent of cell ordering, kernel state, or any bookkeeping flag:
if lora_b_norm(model) <= 1e-8:
raise RuntimeError("LoRA B matrices still at zero init — this is the base model.")
Select Runtime → Change runtime type → T4 GPU, then run top to bottom once, in order. The notebook stops you if the order breaks. Budget ~30 min baseline eval, ~5h40m training, ~30 min final eval; checkpoints land on Drive every 25 steps.
Use the trained model directly
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("rogiskhan/knee-physio-qwen3-4b")
model = AutoModelForCausalLM.from_pretrained("rogiskhan/knee-physio-qwen3-4b", device_map="auto")
messages = [
{"role": "system", "content": SYSTEM_PROMPT}, # from src/genera_dataset.py
{"role": "user", "content": "VALUTAZIONE FISIOTERAPICA - ginocchio sx ..."},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
enc = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**enc, max_new_tokens=700, do_sample=False)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))add_special_tokens=False is not optional — apply_chat_template already emits them,
and adding them twice degrades output badly. That bug cost this project a baseline of 32%
valid JSON that was really 97%. See the postmortem.
Italian input only, knee only, adults only. Anything else is out of distribution.
Regenerate the dataset
python src/genera_dataset.py --n 4000 --seed 42 --out data/Deterministic under a fixed seed — CI runs it twice on every push and diffs the output. The corpus is not committed; the generator is.
| Path | What it is |
|---|---|
notebooks/ |
The pipeline, executed end-to-end. Outputs are the evidence — never stripped |
src/genera_dataset.py |
The synthetic generator. The actual science; everything else is engineering |
docs/postmortem.md |
The invalid run, the two bugs, and the guards that came out of it |
docs/evaluation.md |
Every metric, its denominator, and why each ceiling exists |
docs/model_card.md |
Intended purpose, limits, evaluation |
results/ |
Metrics, manifest, loss curve from the published run |
scripts/validate_repo.py |
Blocks publication on leaked secrets, placeholders, or missing guards |
- Conditional independence is assumed and clinically false. Naive Bayes overstates confidence when correlated tests agree — expect a leading hypothesis at 0.99 with alternatives at exactly 0.0. The model reproduces this flaw faithfully.
- No clinician has reviewed the prevalence and test-characteristic tables. This is the project's scientific bottleneck.
- Italian only, knee only, adults only.
- The test set comes from the same generator as the training set. Held-out inputs prove the model did not memorise; they say nothing about external validity.
- No bias or subgroup analysis has been performed.
Ranked by how much it would improve the project:
- Clinical review of the tables in
src/genera_dataset.py→ open an issue - An independent evaluation set written by a clinician rather than sampled
- The few-shot control described above
- Fixing conditional independence by grouping correlated tests
Never include real patient data — in an issue, a PR, a fixture, or a thread. Not even pseudonymised. Full guidelines: CONTRIBUTING.md.