Repository navigation
Search Quality
"Hit" means the right article is in the first 5 results. Run python tools/eval_retrieval.py --failures to reproduce the tuning and held-out numbers.
| Question set | Questions | Hit | How to read it |
|---|---|---|---|
Tuning (data/eval/gold.json) |
142 | 99% | The index was written against these, so this mostly proves nothing broke |
Held-out (heldout.json) |
41 | 98% | Read before the index was written, so optimistic |
First blind (blind.json) |
36 | 97% now, 50% when first measured | Used for tuning since, so only the 50% says anything |
Second blind (blind2.json) |
34 | 68% | The honest number: written before the last round of work and not tuned on |
On the second blind set, the right law came first for 71% of questions. Off-topic questions: 11 of 12 correctly declined.
- Common questions do well, because the legal index covers them.
- Rare questions and free-form Darija often miss, and some get no answer at all.
- The tuning and held-out answers are checked against the official text by the test suite. The blind sets' expected articles were written from memory and not all checked.
- A lawyer has not reviewed any of these sets (#4).
Multilingual MiniLM embeddings over all articles reached 35% alone and only 5% on Darija. Combined with the lexical search of that time they reached 66% against 58%. That gain did not justify about 300 MB of runtime, so they are not used. A better Arabic model could change this.
Add to data/legal_index.json or the words list in data/lexicon.json, then re-run the evaluation. Write a new blind set before tuning: a set you have tuned on stops being blind.
Moroccan Law AI Legal Q&A Assistant Documentation — Maintained by @Bosaj