Cases with an unresolvable principal score recall 0.0 and drag down mean_recall

[royalpinto007]

Desired outcome

A case that correctly retrieves nothing does not count as recall 0.0, and the dead is not None filter goes away.

Why it matters

Two connected things in app/evaluate.py.

run_case() decides recall consistently for the normal path:

else:
    recall = 1.0  # nothing was expected; not retrieving anything is perfect

But the early return for an unresolvable principal never sets recall at all:

if principal is None:
    # An unknown user retrieving nothing is correct behaviour, not an error in the eval.
    return CaseResult(case_id=case.id, user_id=case.user_id, correct_refusal=not case.should_answer, llm_judged=llm_judged)

so it takes the dataclass default of 0.0. The comment says this is correct behaviour, and passed agrees, yet the case is averaged into mean_recall as a total miss. Add one gold case for an unknown user, which is a natural ACL test to write, and the headline recall number drops even though nothing regressed. The docstring calls recall "the metric that quietly dies when you tighten security", so a false drop here is expensive: it looks exactly like the failure mode the harness exists to detect.

Second, mean_recall filters on a condition that cannot be false:

scored = [c.recall for c in self.cases if c.recall is not None]

CaseResult.recall is typed float with default 0.0 and is never assigned None anywhere in the file. The filter reads as if some cases are deliberately excluded from the average, which is misleading given the bug above, where a case genuinely should have been excluded or scored 1.0.

Steps

1. In the principal is None branch, pass recall=1.0 when case.expected_docs is empty and 0.0 otherwise, matching the logic further down. That keeps a genuinely missed case honest.
2. Either drop the is not None filter in mean_recall, or make recall: float | None = None a real "not scored" signal and set it deliberately. Whichever, the type and the filter should agree.
3. Add a test: a gold case for a user id that does not exist, with expected_docs=[], must not drag mean_recall below 1.0.

Claiming this

Comment below to claim it. A reply usually comes within a day.