Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite queries based on chat history and mentioned context items for better enhanced context fetching #4623

Merged
merged 5 commits into from
Jun 20, 2024

Conversation

thenamankumar
Copy link
Member

closes: https://linear.app/sourcegraph/issue/CODY-2272/rewrite-query-including-content-of-mention-before-sending-to-context

This PR adds an experimental feature under the experimentalNoodle flag to rewrite the queries based on the chat history (last 3 interactions) and the mentioned context items before sending it for enhanced context fetching; assuming it would improve the performance of the context search.

This is similar to the keyword expansion integrated with symf search right now, but it performs the rewrite based on the chat history as well as the mentioned context items, unlike the keyword expansion which treats the query in isolation.

The formal evaluation of this technique has not been done yet but it is the next step.

Until we have evaluated that the rewrite boosts the performance of the context retrieval, we will keep the rewrite step behind the experimental flag.

Test plan

A few examples are (the rewritten query manually pasted in the input box while debugging):

image

image

Copy link

‼️ Hey @sourcegraph/cody-security, please review this PR carefully as it introduces the usage of an unsafe_ function or abuses PromptString.

Copy link
Member

@keegancsmith keegancsmith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Nice that you managed to contain all the new logic in one file!

vscode/src/local-context/rewrite-chat-query.ts Outdated Show resolved Hide resolved
lib/shared/src/chat/chat.ts Outdated Show resolved Hide resolved
Comment on lines +721 to +722
* The retrieval performance boost is not evaluated yet and thus
* it is only available when `experimentNoodle` is set to `true`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how will we evaluate this? I guess at first we will just dogfood and see if it helps, but then if we gain confidence from that we will try measure somehow?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be difficult to make sense of anything by dogfooding, we will need to numbers to gain confidence.

I am working with the AI team to get a multi-turn evaluation dataset to evaluate if it boosts the performance of symf, local embeddings & remote search.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might also want to restrict rewriting only certain queries. Need to figure that out.

vscode/src/editor/utils/editor-context.ts Outdated Show resolved Hide resolved
@thenamankumar thenamankumar merged commit a3cda04 into main Jun 20, 2024
18 of 19 checks passed
@thenamankumar thenamankumar deleted the naman/rewrite-chat-query branch June 20, 2024 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants