LLMDetector returns span dicts when output_format="tokens"

[adaamko]

Problem

LLMDetector advertises the same two output formats as every other detector, but only implements one of them. All three prediction entry points validate output_format as one of ["tokens", "spans"]:

• predict — lettucedetect/detectors/llm.py:472-475
• predict_prompt — lettucedetect/detectors/llm.py:494-497
• predict_prompt_batch — lettucedetect/detectors/llm.py:518-521

but every path returns the span-dict list from _predict unchanged (llm.py:479-480, llm.py:499-500, llm.py:524-529). There is no token conversion anywhere in the file. A caller who asks for tokens gets spans.

Two knock-on effects make this worse than a cosmetic mismatch:

1. The facade default is output_format="tokens" (lettucedetect/models/inference.py:32), so HallucinationDetector(method="llm").predict(...) returns the wrong shape by default — code written against the transformer detector's token output breaks when switched to the LLM detector.
2. BaseDetector._filter_spans_by_confidence deliberately skips filtering for token output (lettucedetect/detectors/base.py:90-91). Since the objects are actually spans, passing output_format="tokens" together with min_confidence silently disables the confidence filter.

Current behavior

Runnable repro (needs OPENAI_API_KEY):

from lettucedetect.models.inference import HallucinationDetector

detector = HallucinationDetector(method="llm", model="gpt-4.1-mini")
preds = detector.predict(
    context=["The capital of France is Paris. The population of France is 67 million."],
    question="What is the population of France?",
    answer="The population of France is 69 million.",
    output_format="tokens",
)
print(preds)
# Actual:   [{'start': 0, 'end': 39, 'text': 'The population of France is 69 million.'}]
# Expected: one dict per answer token, see below.

What token output looks like elsewhere in the codebase — one dict per answer token (prompt tokens are excluded via the -100 label mask), produced by TransformerDetector._predict_single (lettucedetect/detectors/transformer.py:173-185):

[{"token": " The", "pred": 0, "prob": 0.02},
 {"token": " 69",  "pred": 1, "prob": 0.97}, ...]

RAGFactCheckerDetector has no model tokenizer either, yet still honors "tokens" by emitting the same three keys over whitespace tokens (lettucedetect/detectors/rag_fact_checker.py:86 dispatch, rag_fact_checker.py:131-151 conversion).

What to do

Pick one of the two fixes and state the choice in the PR — both are acceptable resolutions:

Option A — implement token output (matches RAGFactCheckerDetector precedent):

1. Add a _convert_to_tokens(answer, spans) helper to LLMDetector: whitespace-split the answer, mark each token whose character range overlaps a predicted span with pred=1, and set prob from the span's confidence when present (the LLM only produces confidences when include_reasoning=True, llm.py:85-87; pick a documented constant otherwise, as rag_fact_checker.py:147 does).
2. Route all three entry points through it when output_format == "tokens", before the confidence filter.

Option B — spans-only is a legitimate scope for this detector:

1. Narrow the validation in all three entry points to accept only "spans" and raise ValueError with a message saying LLMDetector is spans-only.
2. Update the three docstrings (llm.py:466, llm.py:489, llm.py:513) accordingly.

Either way:

3. Add regression tests. LLMDetector accepts an injected client (llm.py:66, llm.py:132), so a stub client returning canned {"hallucination_list": [...]} JSON needs no network; pass cache_file pointing into tmp_path so the default on-disk cache (llm.py:161-171) is not touched.

Acceptance

• output_format="tokens" either returns [{"token", "pred", "prob"}, ...] dicts (option A) or raises ValueError (option B); it never returns span dicts.
• min_confidence is never silently ignored: with option A it is (correctly) inapplicable to tokens; with option B the tokens+confidence combination can no longer occur.
• New tests live in tests/test_llm_detector_pytest.py (the test_*_pytest.py pattern is required by tests/pytest.ini) and cover all three entry points.
• python -m pytest tests/test_llm_detector_pytest.py -v passes.
• python tests/run_pytest.py stays green (it runs tests/test_inference_pytest.py); also run python -m pytest tests/ -v to confirm nothing else regressed.

Non-goals

• No changes to TransformerDetector or RAGFactCheckerDetector token formats.
• No new output formats and no changes to the BaseDetector interface.
• No prompt or schema changes to the LLM call itself.

Start here

git clone https://github.com/KRLabsOrg/LettuceDetect.git
cd LettuceDetect
pip install -e ".[dev]"
python tests/run_pytest.py   # should be green before you change anything

# The code to read first:
grep -n "output_format" lettucedetect/detectors/llm.py
sed -n '173,185p' lettucedetect/detectors/transformer.py      # target token format
sed -n '131,151p' lettucedetect/detectors/rag_fact_checker.py # precedent conversion