feat: add infill mode to anthropic claude instant #974
Conversation
There was a problem hiding this comment.
This is so cool! Pinging @valerybugakov too but this looks safe enough to merge and try out.
There are some interesting diffs in the autocomplete test suite that I linked in the Slack thread but without actually improving the test suite OR running an A/B experiment, it’s so so hard to judge these nuances.
StarCoder vs Claude with Infill will be the battle of the giants. I've updated the temperature like we discussed last night for StarCoder too so it has gotten a bit more predictable.
Huge thank you for this awesome collaboration. You definitely managed to do what I couldn't.
| if (claudeInstantInfill === true) { | ||
| return { provider: 'anthropic', model: 'claude-instant-infill' } | ||
| } |
There was a problem hiding this comment.
Looks like if we set any starcoder FF to 50% and claude-instant-infill to 100% we can make a 50-50 test here 👍
refHere are the failed tests philipp mentioned above: However, all the tests passed with the expected results when running them manually in the editor: Will look into the test suite next. See slack discussion for details: https://sourcegraph.slack.com/archives/C05AGQYD528/p1694095905077429 |
Forgot to add a changelog entry for #974 😅 ## Test plan <!-- Required. See https://docs.sourcegraph.com/dev/background-information/testing_principles. --> changelog update
| }, | ||
| { | ||
| speaker: 'human', | ||
| text: `Below is the code from file path ${this.options.fileName}. First, review the code outside of the ${OPENING_CODE_TAG} XML tags. Then complete the code inside the tags using the same style, patterns and logics of the surrounding code precisely without duplicating existing implementations: |
There was a problem hiding this comment.
Noice! This is a logical enhancement. We overlooked these low-hanging fruits because we hadn't been investing time into improving our prompt. It's great that we're addressing them now.
Close #811 #944 #1026
This PR contains changes related to adding "infill" mode to the Anthropic autocomplete provider for the Claude instant model:
claude-instant-1claude-instant-1) or 'infill' (claude-instant-infill).Quick Demo
See improved suggestions on the right, vs suggestions provided by the latest release build on the left:
autocompletion_.compare.suggestions.before.after.mp4
Summary
This PR adds an "infill" mode to the Anthropic autocomplete provider Claude Instant model to generate more seamless completions based on surrounding code, which fixes issues where:
Before
Before.mp4
After
After.mp4
Resolve the issue where Cody suggests code that already exists
RE: #811 #944
As Claude instant was not intended to be used as a code completions model, additional prompt engineering is required to guide Claude to do the "right" thing, which includes asking Cluade not to complete the code with code already implemented.
update_.don.t.suggest.existing.code.mp4
Test plan
"cody.autocomplete.advanced.model": "claude-instant-infill"Compare the quality of suggestions you get from using this build with the ones from the stable release build that uses the claude instant model with the current prompt.
More examples
claude instant with infill vs starcoder:
starcoder.vs.claude.instant.-.code.comparison.mp4
starcoder.vs.claude.instant.-.code.comparison.mp4
starcoder.vs.claude.instant.-.suffix.mock.mp4