fix: preamble leak for anthropic

[abeatrix]

Close #1291

Fix issues that cause the preamble to leak for anthropic by enclosing the code with backticks to make sure Claude instant knows where the code ends, instead of having the code snippets end with a new line or unfinished code/comment, which would confuse Claude and lead it to think the code was not completed so they should continue the completion from the end of the code snippet instead of the code inside the tags.

Before

After

Test plan

1. build Cody from this branch
2. Go to line 213 of vscode/webviews/Chat.tsx in your editor
3. Instead of seeing Human: as a suggestion, Cody should not suggest anything that starts with Human here (or anywhere)
4. You should still get suggestions from Cody in other appropriate places

[valerybugakov]

It looks logical and solves the issue we found, but I wonder if it affects the output unexpectedly (like when we found out that Claude doesn't like trailing spaces). Did you have a chance to run this change against our completions dataset?

[philipp-spiess]

Agree with Valery. I’m a bit concerned too about shipping this so shortly before the release. Maybe as a quick fix we can add a filter for completions that start with Human: from Anthropic and then we can land this after the release and test it a bit more?

Unrelated but instead of \n<--End of code--> did you ever try to wrap everything in an xml tag like so?

<code>${infillPrefix}${OPENING_CODE_TAG}${CLOSING_CODE_TAG}${infillSuffix}</code>`

[abeatrix]

Agree with Valery. I’m a bit concerned too about shipping this so shortly before the release. Maybe as a quick fix we can add a filter for completions that start with Human: from Anthropic and then we can land this after the release and test it a bit more?

Unrelated but instead of \n<--End of code--> did you ever try to wrap everything in an xml tag like so?

<code>${infillPrefix}${OPENING_CODE_TAG}${CLOSING_CODE_TAG}${infillSuffix}</code>`

I tried that at the beginning but couldn't get it to work, I'm planning to spend some more time later to see if we can improve the prompt later, but for now I am enclosing the prompt with backticks like we do in Chat. I run the test suite and it seems like adding the {infillBlock} between the XML tags works better, but since this is LLM I think the output is going to be slightly different anyway. Let me know if you think we should remove the ${infillBlock} from the Human prompt and just enclose the code with backticks for now. This at least would solve the issue where Claude doesn't know if the code snippet has ended or not 😄

[abeatrix]

as discussed with @philipp-spiess on slack, we will merge this fix and make additional changes in patches if required.