Span-level source attribution in Foundry IQ responses - map answer claims to retrieved chunks

[Jubin Soni (jubins)]

Technical Feedback

Foundry IQ returns a grounded answer and the sources it retrieved, but it does not link specific parts of the answer to specific sources or chunks.

This becomes a problem when an answer combines sources. A paragraph might use three documents, but there is no way to tell which sentence came from which document, or which claims have no supporting source.

This blocks three things:

1. UI: Users expect to click a sentence and see its source.
2. Evaluation: Groundedness can be measured for the response, but not for individual claims. A response that is 80% grounded and 20% fabricated can still look "mostly grounded."
3. Compliance: Finance and healthcare require auditability at the assertion level. Without span-level attribution, this has to be built outside the platform.

Also, there is also no stable ID connecting an answer span to the document and chunk that support it, making attribution difficult to persist or replay after reindexing.

What is missing is span-level attribution from each claim to its supporting document and chunk, or an explicit indication that no source supports the claim.

Desired Outcome

An optional attribution field on the response, off by default, roughly like this:

{
  "answer": "...",
  "attributions": [
    {
      "span": { "start": 0, "end": 87 },
      "supports": [
        {
          "document_id": "doc_a1b2",
          "chunk_id": "chunk_0034",
          "source_span": { "start": 1204, "end": 1389 },
          "score": 0.91
        }
      ]
    },
    {
      "span": { "start": 88, "end": 156 },
      "supports": []
    }
  ]
}

Current Workaround

Currently there are 3 not so perfect workarounds.

1. Split the answer into sentences, embed each one, match against the chunk embeddings, apply a threshold.

sentences = split_sentences(answer)
sent_vecs = embed(sentences)
chunk_vecs = embed([c.text for c in retrieved_chunks])
sims = cosine_similarity(sent_vecs, chunk_vecs)
 
for i, sent in enumerate(sentences):
    best = sims[i].argmax()
    attributed = retrieved_chunks[best] if sims[i][best] > 0.72 else None

2. Check each sentence against each candidate chunk. More accurate, but it's N×M calls and roughly doubles latency and cost on the request path. Batching helps a bit but it doesn't fix the math.

3. Ask the model to emit [1], [2], ... after each claim. It drops markers in longer answers, sometimes cites indices that don't exist, and gives you no character offsets, so you still have to parse and reconcile everything yourself. It also fights with structured output.

[James Ross (jimy-r)]

Until the platform emits it, the version that has held up for me inverts the workaround: make the generation carry provenance instead of reconstructing it afterwards.

1. Ask for claims, not prose. The answer is produced as structured output: a list of claims, each with the chunk id it rests on and a verbatim quote from that chunk, or an explicit empty supports list. The prose is rendered from the claims, so every sentence already knows its source.
2. Verify by containment rather than similarity. Each quote must appear as a substring of the cited chunk, whitespace-normalised. That is a deterministic check with no threshold to tune, and a claim that fails it is demoted to unsupported rather than silently kept.
3. Hash the chunk ids. An id derived from document id plus chunk text survives reindexing, which is the persist-and-replay problem you name.

Evaluation then falls out. Groundedness per response becomes verified claims over total claims, so an 80/20 answer reads as 80/20 and never as "mostly grounded". Compliance reads the same table.

The cost is an extra generation step and a stricter prompt contract. I maintain a reference architecture that documents this exact pattern: https://github.com/jimy-r/agent-workspace-architecture/blob/main/PATTERNS.md#16-a-claim-carries-its-provenance-or-it-is-a-guess