Unhelpful completions

[sqs]

Whenever Cody gives you an unhelpful completion, post a new comment here with:

• Screenshot or description
• Version of your Cody for VS Code extension

[sqs]

Seeing this a fair bit. It seems the newline after the comment is being ignored, or else we need to add some heuristics. v0.7.1690038292

[vovakulikov]

@sqs just to confirm I understand, do you mean that instead of pushing a new line with completion we should use the empty line for completion?

So it would be like

// set tracer
<cursor>
public async prove....

transforms into ↓

// set tracer
if provided
public async prove....

[sqs]

The completion on the newline is not syntactic - it's trying to complete the comment set tracer if provided not write code.

[philipp-spiess]

Quick update here:

For anthropic, the main reason for this is that we do trim the whitespace in the prefix here, including the \n. I think if we detect to be inside a comment, we should include the \n in the prompt. That's where @vovakulikov's work will be useful!

For the StarCoder config, this issue does not happen because we include all whitespace in the prompt.

[sqs]

I am still seeing this occasionally

[vovakulikov]

Should be closed by this PR #727

[sqs]

I did not expect or want this to generate a multi-line completion. v0.7.1690038292

[vovakulikov]

Hmm, @philipp-spiess, do you see here something that we can use as a condition? I mean, usually cursor between ( and ) doesn't mean that we don't want to have a multiline completion (for example, this is true for our ultimate example with function bubbleSort(<cursor>)).

So the only thing I can think of is setTracer is a method here, so we probably want to disable multi-completion for methods or function calls, but as I mentioned recently, we probably have to parse prefixes into lexeme to understand this in most cases. But bringing a tree-sitter might be a big thing for this kind of problem, let me know if I'm missing here a simpler solution.

[philipp-spiess]

@vovakulikov Yeah that's a tough one. I was thinking previously at some point that ( might not be a good multi-line trigger and instead we might be able to check for function definitions? I think when I tried CoPilot, function foo<cursor> would also do a multi-line completion already. Perhaps it makes sense to look at other completion providers a bit?

I think the main issue for the linked snipped though is the indentation that is completely screwed. E.g if the indentation would be correct, panel = null would be matched with the next non-empty line here and truncated correctly I believe.

[sqs]

2 other conditions:

1. ( as a trigger should only complete to the matching )
2. The suggestion duplicates the function call (eg there are now 2 calls to setTracer)

[vovakulikov]

@sqs "( as a trigger should only complete the matching )" I think you're right, but at the same time this might introduce some regression. Editors usually complete call statements with closing parenthesis automatically.

function bubbleSort( → function bubbleSort(<cursor>)

And in this case, we probably should ignore closing parenthesis and generate the function body with sort implementation.

So I think the condition with "( as a trigger should only complete the matching )" is valid only when we already have a "body" block statement after ), if we don't we should ignore closing ")" and apply the full completion.

[vovakulikov]

Should be closed by this PR #726

[philipp-spiess]

I haven't caught a picture of this but there's one issue that I find super annoying that happened to me a lot of times today. It happens when you move the cursor around, making some changes to a larger chunk of code, when sometimes you're accidentally hitting an earlier cache for a very large and now unrelated mutli-line completion.

I think for multi-line completions to be served from cache, we should still include a few hundred ms of a delay, just to make sure the user is not immediately editing the text. 🤔

[vovakulikov]

@philipp-spiess I haven't touched cache logic yet, but should we clear the cache in case a user has skipped/typed something else there? What do we use as key in cache? is it just a prefix string?

[philipp-spiess]

but should we clear the cache in case a user has skipped/typed something else there?

I think that would defeat the purpose of the cache a bit. It's super useful e.g. when you type something new and then use backspace to get back to the previous completion.

What do we use as key in cache? is it just a prefix string?

Pretty much, yeah.

[philipp-spiess]

#392

[sqs]

Seems to be because the context after the cursor is not sent to the LLM

[philipp-spiess]

Cody recommending an exactly identical comment (from @Strum355)

[philipp-spiess]

Indentation of the completed text is sometimes wrong which causes a bunch of follow up issues like unsuccessful indentation based multi-line termination and unsuccessfully finding a match with the next non-empty line.

What's weird though is that my logs don't show eight spaces before the completed insertion?

"completion": "\n        return\n    }\n\n    const { codeResults, textResults } = await codebaseContext.getEmbeddings({\n        query: {\n            text,\n            code: {\n                numResults: NUM_CODE_RESULTS + NUM_CODE_RESULTS_EXTRA,\n                filePaths: [currentFilePath],\n            },\n            text: {\n                numResults: NUM_TEXT_RESULTS,\n            },\n        },\n    })\n\n    const embeddings = codeResults\n        .slice(0, NUM_CODE_RESULTS) // Take only the first NUM_CODE_RESULTS results\n        .map(({ code }) => ({\n            // Extract relevant info from code result\n            text,\n            identifier: code.identifier,\n            code: code.code,\n        }))\n\n    embeddingsPerFile.set(\n        currentFilePath,\n        {\n            embeddings,\n            lastChange: new Date(),\n        }\n    )\n\n    logCompletionEvent({\n        event: 'fetchEmbeddings',\n        filePath: currentFilePath,\n        numCodeResults: codeResults.length,\n       ",

"completion": "\n        return\n    }\n\n    const { codeResults, textResults } = await codebaseContext.getEmbeddingsForText({\n        text,\n        numCodeResults: NUM_CODE_RESULTS + NUM_CODE_RESULTS_EXTRA, \n        numTextResults: NUM_TEXT_RESULTS,\n    })\n\n    // Filter out embeddings from the current file.\n    const embeddings = codeResults\n        .filter(result => result.filePath !== currentFilePath)\n        .slice(0, NUM_CODE_RESULTS)\n\n    embeddingsPerFile.set(\n        currentFilePath, \n        {\n            embeddings,\n            lastChange: new Date(),\n        }\n    )\n\n    logCompletionEvent({\n        action: 'code_completions',\n        numCodeResults: codeResults.length,\n        numTextResults: textResults.length,\n    })\n}",

"completion": "\n      return\n    }\n\n    const { codeResults, textResults } = await codebaseContext.getEmbeddingsForText(\n      text,\n      {\n        numCodeResults: NUM_CODE_RESULTS + NUM_CODE_RESULTS_EXTRA,\n        numTextResults: NUM_TEXT_RESULTS,\n      }\n    )\n\n    // Filter out embeddings from the current file.\n    const embeddings = codeResults\n      .filter(({ filePath }) => filePath !== currentFilePath)\n      .slice(0, NUM_CODE_RESULTS)\n\n    const lastChange = new Date()\n\n    embeddingsPerFile.set(\n      currentFilePath, \n      {\n        embeddings,\n        lastChange,\n      }\n    )\n\n    logCompletionEvent({\n      type: 'embeddings',\n      numEmbeddings: embeddings.length,\n    })\n}",

[philipp-spiess]

I wonder what happens if we try to rewrite all indentation to use tabs and only convert from tabs to spaces before we insert back into the editor? I feel like that mix up here where the indentation for the first line is six spaces wouldn't really happen with tabs?

[philipp-spiess]

Hm I think the logs I’m looking at might be wrong because 1) I can't repro this anymore 2) the exact string returned in the logs I attached does not cause it to break in our test suite.

[philipp-spiess]

Quick update here: The most promising direction for fixing these indentation issues so far seems to be switching to a code completion specific LLM like StarCoder.

[philipp-spiess]

Another example from YAML:

Screen.Recording.2023-08-08.at.3.22.22.PM.mov

[deepak2431]

@philipp-spiess In some cases, I see like incomplete generated completions. Can we handle this case?

[vovakulikov]

@philipp-spiess, here is an idea - should we show the completion only and if only this completion doesn't break the syntax of the snippet (if it's not already broken)?

My idea is to try to find a root node above the cursor, like function declaration or method declaration or program, which is the file itself, and then parse if parsing fails this could be a signal that completions break something, and we should not show it. If the snippet had syntax problems before we show completion even if it doesn't solve the problem.

[philipp-spiess]

here is an idea - should we show the completion only and if only this completion doesn't break the syntax of the snippet (if it's not already broken)?

@vovakulikov Yeah this sound good but maybe for now instead of completely filtering them out we can just reorder the three multiline samples we have based on wether they have errors or not and log how many we would filter out? I’m worried that we might filter out too many results otherwise 😅

I think we also need to raise the token limit from 256 to something higher - but can only do this with the changes in RFC 815 so we don't regress latency.

[deepak2431]

@philipp-spiess This early completion new UX is awesome!

Although, there's this bracket issue quite often. Can we fix this? This might be a known issue, so I just wanted to add it here to keep track of.

[deepak2431]

The second completion is CORRECT as there is no extra bracket with it. It might be the ranking issue.

Yielding just the first completion would be good, but this realise me of the experience that on hover, I need to wait for a few seconds till the other completions appear, so it would be better to think other way around if we would adopt this.
I haven't tried the StarCoder, any way to configure it?

Also, have you looked over the completion cache as how it's updated if we change the function/class name to something else for which the completion has appeared.

[philipp-spiess]

Yielding just the first completion would be good, but this realise me of the experience that on hover, I need to wait for a few seconds till the other completions appear, so it would be better to think other way around if we would adopt this.

Is that happening for you right now? This looks like a regression. 🤔

I haven't tried the StarCoder, any way to configure it?

I believe you got access to that from Justin. It's hosted by fireworks right now, so perhaps you know it under that name? I've recently also updated that to use an even faster (albeit slightly more stupid) model of StarCoder and am testing this right now. We aren't there yet for a broader test though, hopefully very soon though!

Also, have you looked over the completion cache as how it's updated if we change the function/class name to something else for which the completion has appeared.

We're currently thinking that the completions cache was a bad idea all together, especially because of cases like you mentioned where there are subtle edits to the document etc. Do you have a concrete example of what you have in mind though? I'd love to take this into account. A very recent discussion to that is happening here if you're curious 🙂 #559

[deepak2431]

No, this is not happening right now, but as you said, to yield the first completion first and the rest on hover. In that case, it might happen because right now, we are just showing all of them at once, I think.

[deepak2431]

Oh, yeah! I got access to it through Justin mine just one feedback was that it was generating more than required completions, but in terms of latency it was for sure good. I will configure that again and use it for few days to see how's my experience of using it.

[deepak2431]

We're currently thinking that the completions cache was a bad idea all together, especially because of cases like you mentioned where there are subtle edits to the document etc. Do you have a concrete example of what you have in mind though? I'd love to take this into account. A very recent discussion to that is happening here if you're curious 🙂 #559

My first experience when the completions results appeared from the cache was good, I thought it's so fast, but when I saw this issue, then I thought this needed to be handled in a good way as how we are displaying the cache results, and which cache to expire when considering the DocumentEdit thing you mentioned.

You can check this case. First, it generated the completions as per my class name, Person, but after I updated it to GithubAPI, it showed me the same results from the cache. Thanks for sharing the issue, I will follow the discussion :)

REC-20230804194238.mp4

[deepak2431]

@philipp-spiess Hmm, I see.
Yeah, I have both enabled. This log was from the dev mode. Let me see if I can allow for the live mode and share something with you.

No, this happened with .md, .tsx, .py files, all of them. It would be great if this can be fixed :)

[deepak2431]

@philipp-spiess I can't see any logs appearing in live mode. While trying this noticed one thing the completions are being triggered sometimes on the press of the delete button too.

[philipp-spiess]

While trying this noticed one thing the completions are being triggered sometimes on the press of the delete button too.

Yeah that's a small behavior change we've made.

Hm do you think you'd find time for a 5min video chat? We can chat on Discord to find a time suitable for both. Really sorry that this doesn't work for you but I’m running out of ideas on how to investigate 😅

[deepak2431]

@philipp-spiess No problem. Sure, we can connect and look into it. Let me ping you on Discord.

[philipp-spiess]

Had a lovely call with @deepak2431 right now:

• The missing logs were because he still had cody.debug = true as a setting which for some reason overwrites the other two settings lol
• I believe the issue was actually rate limiting becaue @deepak2431 was only having an increased chat message limit, not an increase code completion limit. I increased his rate limit and he'll reach out if the issue persists (everything was working fine during our call).

I'm going to close this thread from the GitHub discussion to keep this on unhelpful completions but @deepak2431 please reach out again if you still have issues. Thanks again 🙇

[philipp-spiess]

When the whole line is deleted (cmd + x), the completion that was previously shown on that line will apear on the next line for ~1sec until a new completion request comes in:

Screen.Recording.2023-08-09.at.10.09.05.mov

(Other code completion extensions behave the same)

[philipp-spiess]

Via @cartermp in #616

Here's a screenshot demonstrating the issue:

I have this code:

        user_description = input("Please describe what you want: ")
        # Make a GPT call to generate title and description
        # This will depend on how you want to generate them
        title, description = generate_title_description(user_description)

And it wanted to suggest this:

def generate_title_description(title, description):
    return f"Title: {title}\nDescription: {description}"

That seems like it's not taking codebase context into consideration, as I'm actually after something like this:

def generate_title_description(user_description):
    # do something with user_description...

[philipp-spiess]

Looks like comparing against the next non-empty line alone is not enough 😬 cc @vovakulikov

[sqs]

I often find multi-line completions useful, but sometimes they are just wrong. Common characteristics of unhelpful multi-line completions (mentioned in case they help with heuristics to detect and avoid them):

• they end my current syntax block and start another one - eg \nsomething\n})\n\ntest('something else' ,,,
• they include const foo = { ... } (with no type--I would almost always give a type in this kind of situation)

[philipp-spiess]

This is fixed by #913

[philipp-spiess]

Cody suggests code that is already present

#811

[olafurpg]

VS Code, Cody suggests declaring a variable inside an object literal

[olafurpg]

I'm getting these very frequently right now

[beyang]

It would be nice if a completion (like []*SymbolQuery{...) triggered here. I think it doesn't, because the comma is present (I think the comma was an artifact of a non-Cody autocomplete)

[beyang]

In cases where I have an empty struct definition in Go, the idea completion would fill in all the fields until the end of the struct declaration scope, rather than suggesting things line-by-line.

This occurs successfully here:

But if I add a newline, it does line-by-line completion, which is slightly undesirable in this case (I would expect the behavior to be at least consistent):

[abeatrix]

After typing / in a Go file, sometimes I'd get:

Expected //

[abeatrix]

Suggest code from test file in a non-test file:

autocomplete.-.before.-.dot.go.mp4

In this case, I would not expect Cody to make any suggestions, or make suggestion that is relevant to the code above, instead of suggesting code that is copied from another file that is not relevant.

[deepak2431]

@philipp-spiess Not sure if there's any change to the model being used with completions, but I am getting a lot of unhelpful completions while working with Python codebases. Some of them are:

1. Wrong import names from the pip package
2. Unused suggestion as functions definition anywhere after variable
3. Class method showing wrong params
4. Python auto-completion breaks the indentation, which I need to fix manually

[abeatrix]

Cody'd keep making the same suggestions after accepting it:

issues_944.repeated.suggestions.mp4

Issue filed: #944

[olafurpg]

JetBrains, local build. Cody autocomplete suggested defining a function inside of another function. The resulting code would not parse

[steveyegge]

Same happened to me today, also local build:

[olafurpg]

I was trying to get Cody autocomplete to find the Intellij IDEA API to get the name of the current JetBrains product (IntelliJ/GoLand/PhpStorm/...) and it suggested the product name "Cody"

[valerybugakov]

Quality suggestions only!

[olafurpg]

This is a quality suggestion, the "Cody" completion has low quality.

[abeatrix]

This reminds me of a "passive aggressive" suggestion I once got from Cody 😆

[abeatrix]

reported by @deepak2431

Few issues I saw still with the "}" with completion if it's still there, and broken indentation with Python and TS Codebase

[abeatrix]

related to existing issue #299

[deepak2431]

Suggested the same code from the other resources in the same file. It looks like the cache improvements.

CC: @abeatrix

[deepak2431]

It is not taking into consideration the code below. This sort of issue is lesser with claude-infill, but yeah, it happens sometimes.

[olafurpg]

Unhelpful single-line completion in TypeScript (via WebStorm)

[sqs]

Human: Here is another code block with context:

[valerybugakov]

@sqs, is it the latest nighly version of the stable one?

[jdorfman]

Hello, we will be closing our GitHub Discussions and moving to our new community forum. You can log in using your GitHub account. Thank you!