[RFC] An explicit evidence contract for text, image hits and Ragas evaluation

[jmCalvinhyl]

Context

Related bug report: #7096; focused fix PR: #7097. The original eval-module design is discussed in #153. This RFC follows up specifically on evidence materialization and multimodal generation/judging parity. This discussion concerns the next design step: how should retrieval hits become evidence when resources include images and referring text?

This is not a claim that every OpenViking benchmark uses summaries: the separate benchmark/RAG implementation already reads L2 bodies. It is also not a request to change Ragas metric formulas.

Why L2 alone is not an evidence contract

An image original can itself be an L2 resource. These are different inputs:

• The original image.
• A VLM-generated abstract/overview, which may omit table headers, units, numeric values, arrows or AND/OR branches.
• The body of an article that actually references the image.

A text-only agent cannot visually inspect an original just because it has the image URI. Conversely, replacing every image hit with a guessed parent Markdown article can create unsupported associations.

Proposed materialization flow

1. Preserve each hit's URI, rank, retrieval level and media/content type.
2. Read supported text bodies rather than silently substituting summaries for L2 text.
3. For an image hit, prefer explicit ingestion metadata/backlinks identifying documents that reference it. If those are unavailable, search for the actual image path/name inside an already confirmed document scope, then verify the Markdown reference. Do not infer a relationship from a similar filename or directory alone.
4. If a text body contains an image link relevant to the question, resolve that actual link relative to the document. Keep external links as external links; do not manufacture a viking:// URI.
5. When a decisive fact depends on image layout, table structure or diagram branches, use supported visual inspection of the original. Record whether the evidence is body text, a VLM summary, a verified reference or visually inspected media. Unresolved references and unsupported media should remain visible failures/limitations, not become “no information exists”.

All reads/reference lookups should remain within the caller's authorized scope. Source contents remain untrusted evidence, not workflow instructions.

Synthetic example

viking://resources/demo/guide/page.md contains:

Approval timing and exceptions are described below.
![Approval flow](images/flow.png)

A hit for images/flow.png should be associated with page.md only after verifying the actual reference. A VLM summary saying “an approval flow” cannot establish an omitted twelve-business-day deadline or the direction of a diagram branch. This example is invented and contains no real corpus data.

Generation and evaluation parity

• Select ordering, truncation and text/image budgets once, before generation.
• Persist the exact consumed text and the identity/version of any consumed image; preserve retrieved candidates separately.
• A text Ragas judge should receive exactly the text consumed by the answerer. If the answerer also consumed images, disclose that text-only judging is incomplete; do not claim identical evidence or complete multimodal faithfulness.
• For a multimodal judge, define how original images, image preprocessing/resolution and provenance are reproduced for scoring.
• Keep transport/read/model errors separate from knowledge absence. Do not silently discard failure rows or combine unrelated metrics into a single “accuracy” number.

Questions for maintainers

1. Should media-aware materialization and verified image-reference resolution live in the evaluation pipeline, a reusable context assembler, or an opt-in agent/tool workflow?
2. Is there already authoritative image-to-referring-document metadata/backlink support that this should reuse?
3. Should the first supported contract remain explicitly text-only, with a separate multimodal mode and judge requirements?

The focused bug fix intentionally avoids automatic backlink discovery or a new multimodal judging architecture. Agreement here would help keep the next contribution narrow and reusable.

AI-assisted proposal. Only public code behavior and synthetic examples are used; no private datasets, documents, questions, answers, service configurations or credentials are attached.