My RAG system retrieves the right document but answers wrong. Why? #23
Answered
by
martex-dev
martex-dev
asked this question in
Q&A
|
My RAG system retrieves the right document but answers wrong. Why? Notes and answer below. |
Answered by
martex-dev
Aug 25, 2026
Replies: 1 comment
|
Usually the chunk boundary is cutting through the middle of the fact. If you split on a fixed token count, the retrieved chunk can contain the subject of a sentence but not its object, and the model then fills the gap by guessing. Split on document structure such as headings, paragraphs, or list items, with a small overlap, so a complete fact lands inside one chunk. Upgrading the embedding model rarely recovers what bad chunking loses. |
0 replies
Answer selected by
martex-dev
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Usually the chunk boundary is cutting through the middle of the fact. If you split on a fixed token count, the retrieved chunk can contain the subject of a sentence but not its object, and the model then fills the gap by guessing. Split on document structure such as headings, paragraphs, or list items, with a small overlap, so a complete fact lands inside one chunk. Upgrading the embedding model rarely recovers what bad chunking loses.