Skip to content

Search Quality

Oussama Elhadji edited this page Nov 4, 2026 · 1 revision

Search quality

"Hit" means the right article is in the first 5 results. Run python tools/eval_retrieval.py --failures to reproduce the tuning and held-out numbers.

Question set Questions Hit How to read it
Tuning (data/eval/gold.json) 142 99% The index was written against these, so this mostly proves nothing broke
Held-out (heldout.json) 41 98% Read before the index was written, so optimistic
First blind (blind.json) 36 97% now, 50% when first measured Used for tuning since, so only the 50% says anything
Second blind (blind2.json) 34 68% The honest number: written before the last round of work and not tuned on

On the second blind set, the right law came first for 71% of questions. Off-topic questions: 11 of 12 correctly declined.

What this means

  • Common questions do well, because the legal index covers them.
  • Rare questions and free-form Darija often miss, and some get no answer at all.
  • The tuning and held-out answers are checked against the official text by the test suite. The blind sets' expected articles were written from memory and not all checked.
  • A lawyer has not reviewed any of these sets (#4).

Embeddings

Multilingual MiniLM embeddings over all articles reached 35% alone and only 5% on Darija. Combined with the lexical search of that time they reached 66% against 58%. That gain did not justify about 300 MB of runtime, so they are not used. A better Arabic model could change this.

How to improve it

Add to data/legal_index.json or the words list in data/lexicon.json, then re-run the evaluation. Write a new blind set before tuning: a set you have tuned on stops being blind.

Clone this wiki locally