Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cody: Evaluate result ranking for repos without embeddings #1721

Closed
umpox opened this issue Nov 13, 2023 · 1 comment
Closed

Cody: Evaluate result ranking for repos without embeddings #1721

umpox opened this issue Nov 13, 2023 · 1 comment
Labels
car: reduced-chat-commands-latency perf Stale

Comments

@umpox
Copy link
Contributor

umpox commented Nov 13, 2023 •
edited
Loading

Description

For repositories without embeddings, the LLM reranker was disabled in #1722 as it added significant latency to all chat/commands/edits.

We should still investigate how we can still rerank results if we don't already have them ranked.

Some thoughts:

  • If, in a world with app-less embeddings, embeddings are so easily enabled for users, do we need to invest time in supporting cases where we don't have them (Probably)
  • Are there alternative, quicker, ways to rerank results?
    • Can we use code intelligence? E.g. # symbols in code snippet that are also included in selected code
    • Distance to current file, measuring directory relevance
    • Other heuristics?
@umpox umpox mentioned this issue Nov 13, 2023
Merged
umpox added a commit that referenced this issue Nov 15, 2023
@umpox
Chat/Commands: Remove LLM reranker for repos without embeddings (#1722)
c777fef 
Part of #1544
Part of #1464

## Description

The LLM reranker adds a significant amount of latency to reranking any
fetched context. This applies to all commands, chat and edits. The
latency is typically between 2-5 seconds.

This PR just disables the reranker for now.

I think this is worth doing as:
- The added latency from the reranker is not worth the possible benefit
right now. It still can sometimes rerank poorly and doesn't provide
enough value to justify waiting that long.
- Context doesn't have a constant value, in many cases context will
provide little value (e.g. simple fixups) in some cases, like chat,
context would provide a lot more value. We don't have a way of
determining that right now, so we're often adding lots of latency for no
good reason.

I created an issue here: #1721
to evaluate a different or improved approach. Possible details there on
how we could improve this.
 
## Test plan

On a repository without embeddings, run:
- Chat
- Commands
- Edits

Review results and context included.
@github-actions GitHub Actions
Copy link

This issue is marked as stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed automatically in 5 days.

@github-actions github-actions bot added the Stale label Jan 13, 2024
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jan 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
car: reduced-chat-commands-latency perf Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@umpox