Proposal: Integrate VeriTrail for hallucination detection & provenance in SpecKit

[richgo]

Disclosure: I work for a bank, where is a lot of distrust for generated content. Given the early nature of both SpecKit and VeriTrail (which I hope will be open sourced) this may well be a consideration for later iterations of SpecKit.

Summary
It would be valuable to explore integration between SpecKit and Microsoft Research’s VeriTrail, a framework for hallucination detection and provenance tracing in multi-step AI workflows. https://www.microsoft.com/en-us/research/blog/veritrail-detecting-hallucination-and-tracing-provenance-in-multi-step-ai-workflows/

Motivation

• Multi-step AI workflows in spec generation risk compounding hallucinations; catching errors only at the final step is insufficient.
• VeriTrail’s DAG-based approach can trace generated specs back to their sources and highlight exactly where unsupported content was introduced.
• Improved trust and adoption: Developers would gain confidence in AI-assisted spec generation when outputs are verifiable and auditable.

Benefits

• Detect hallucinations earlier and with greater precision
• Provide provenance mapping from final spec outputs to intermediate steps and source content
• Enhance transparency, auditability, and debugging in SpecKit workflows

[tiagostutz]

I like the idea, but I'd propose support for something a bit more generic like linters (maybe called evaluators?), being VeriTrail one of those possible linters, but then something that can be registered/extended, so people could even use their private/paid evaluators if wanted and community could also build new ones.

At the end, I think we are looking for a way to register "linters" that would run after each phase (spec, plan, task) and that would trigger iteration of those in case of linting error. We just need a way to be able to register those linters. Btw, that linter capability is what made JS/TS grow as it grown, so this can be an inspiration here.

Thoughts?

[localden]

Thank you for the suggestion. We flagged this for future investigation, but it won't be immediately part of Spec Kit.