-
Notifications
You must be signed in to change notification settings - Fork 297
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rewrite queries based on chat history and mentioned context items for better enhanced context fetching #4623
Conversation
|
dc45290
to
f6185b4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Nice that you managed to contain all the new logic in one file!
* The retrieval performance boost is not evaluated yet and thus | ||
* it is only available when `experimentNoodle` is set to `true`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how will we evaluate this? I guess at first we will just dogfood and see if it helps, but then if we gain confidence from that we will try measure somehow?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will be difficult to make sense of anything by dogfooding, we will need to numbers to gain confidence.
I am working with the AI team to get a multi-turn evaluation dataset to evaluate if it boosts the performance of symf, local embeddings & remote search.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might also want to restrict rewriting only certain queries. Need to figure that out.
closes: https://linear.app/sourcegraph/issue/CODY-2272/rewrite-query-including-content-of-mention-before-sending-to-context
This PR adds an experimental feature under the
experimentalNoodle
flag to rewrite the queries based on the chat history (last 3 interactions) and the mentioned context items before sending it for enhanced context fetching; assuming it would improve the performance of the context search.This is similar to the keyword expansion integrated with symf search right now, but it performs the rewrite based on the chat history as well as the mentioned context items, unlike the keyword expansion which treats the query in isolation.
The formal evaluation of this technique has not been done yet but it is the next step.
Until we have evaluated that the rewrite boosts the performance of the context retrieval, we will keep the rewrite step behind the experimental flag.
Test plan
A few examples are (the rewritten query manually pasted in the input box while debugging):