Make it obvious in-editor when Cody has provided an autocomplete

[toolmantim]

Autocomplete is our # 1 onboarding task for now. But Cody’s autocomplete works behind the scenes, with no chance for a new user to understand what's really going on. Currently it's somewhat confusing, and a little anticlimatic.

We need to:

• Make it visible that Cody is working, and when it contributes an autocorrect
• Give directions on how to accept it
• Point out that it was accepted (with a little celebratory moment)
• Create awareness of next steps of onboarding, with a mention of the status bar icon on acceptance of the second autocomplete provided by Cody, and an additional notification in the chat.

Related PRDs:

• https://docs.google.com/document/d/1d3AsJeAB3AuUwAUTvvMzwKVQeVniX1Q06LPCBna5BbU/edit

Related Slack messages:

• https://sourcegraph.slack.com/archives/C0HMGV90V/p1691614048664469

References:

• Gitlen’s inline blame annotations:
  • lineAnnotationController
  • getInlineDecoration

[toolmantim]

Figma: https://www.figma.com/file/9112BsKsJc1BpO2j8XYwLL/%F0%9F%A4%96-Cody-VS-Code-%5BMain%5D?type=design&node-id=4618-156116&mode=design&t=apmJ61U9vPAT0aVV-4

See Figma for notes on logic, animation, and prototypes (select a frame and hit command-space to start).

For the sidebar notification, it uses the same design as #653 — but this time I've fleshed out the motion details too (which would apply to both)

Screen.Recording.2023-08-14.at.7.01.50.pm.mov

Screen.Recording.2023-08-14.at.6.57.56.pm.mov

[chenkc805]

Nice! Love the ghost text in the editor and celebration text afterwards.

@philipp-spiess could we launch this as a test? I'd love to see if this leads to a meaningful bump in autocomplete metrics

[philipp-spiess]

@chenkc805 Sure, I don't see a reason why we can't :)

[chenkc805]

Lets do it then! CC: @kelsey-brown for visibility

[chenkc805]

Here's the experiment requirements:

• experiment name: obvious_autocomplete_667
• # variants: 2 variants, A (control, no extra text) and B (treatment, with extra text). Split is 50/50
• bucketing moment: fire a bucketing event when we show an autocomplete suggestion. Only fire for new users who have never seen a Cody autocomplete suggestion before (to measure new user impact).
• Key metrics: Autocomplete acceptance rate, W1 retention rate, # autocomplete suggestions (to see if users use autocomplete more)

[toolmantim]

I imagine the autocomplete acceptance rate is highly dependent on the quality of the answers themselves (you get very different results based on the type of file you open and what line you're editing etc) — we might need to control for that (if possible).

[toolmantim]

I forgot to mention earlier, we can also provide a small bit of markdown content in the decoration's hoverMessage property (which will be shown by VS Code with any built-in errors), similar to what Gitlens does:

This would be a good opportunity to provide any longer words, and link to docs or the getting started guide’s "Autocomplete" section.

This seems like an easy add if it's just a an extra decoration property to specify. @philipp-spiess should I quickly mockup up what that content would be?

[chenkc805]

I imagine the autocomplete acceptance rate is highly dependent on the quality of the answers themselves (you get very different results based on the type of file you open and what line you're editing etc) — we might need to control for that (if possible).

It should be fine - at scale with lots of different users who are randomly enrolled, the quality should average out to be the same across both groups

This would be a good opportunity to provide any longer words, and link to docs or the getting started guide’s "Autocomplete" section.

Can we scope this into a separate PR? Feels a bit like scope creep here and this PR already has a decent amount of changes in it.

[philipp-spiess]

Thanks for all the input! Just to set expectations here but we have not scheduled eng time for this in the current sprint, so let's find out who is going to work on this and how to prioritize it against the other things in our next planning on Monday. We also need to validate if we can do some of the work here, it's not clear to me yet if we can trigger a code lense "view" during autocomplete events yet.

Plus is it currently impossible to tell if our completion is visible or e.g. a copilot one is visible. We guard for this in our logs by only counting acceptance rate for users that do not have copilot or other extensions enabled. Should we do this here too and only show "👈 Cody autocomplete` if none of the other extensions are enabled?

And, I suspect, we want to make sure to only ever show these once? But what if we show a completion only for a few MS?

@chenkc805

experiment name: obvious_autocomplete_667

I would suggest we pick a different name here and also prefix with cody. What about cody-vscode-autocomplete-notifications-667?

Only fire for new users who have never seen a Cody autocomplete suggestion before (to measure new user impact).

We don't have logging for this on the client but I think we can probably do this with SQL foo? :D

[chenkc805]

I would suggest we pick a different name here and also prefix with cody. What about cody-vscode-autocomplete-notifications-667?

Yeah works for me!

We don't have logging for this on the client but I think we can probably do this with SQL foo? :D

We can do this in SQL :)

[philipp-spiess]

We've talked about this a bit during our morning sync today with @beyang and he had an interesting point: If the goal is to show our users the value they're getting, it might be better to have a less intrusive way of doing this. He suggested a "stats" widget that we can use to show some nice statistics instead. I’m curious what you think about this @toolmantim? If we had that, would we still want to use the code lenses to show when a completions is inserted?

[toolmantim]

@philipp-spiess sorry I didn't follow up on that comment! This is about onboarding, and it's only shown once. Once I mentioned that to @beyang he was on board. The stats widget is an interesting idea… but probably for another use case.

For onboarding, there's also a chat pane view version of this, which I've extracted out of a previous PR in #1071. That's probably a much easier first step, and is close to done?

But the in-editor experience as designed above is what a lot of people internally were asking for (and what a few of the other AI assistants implement).

[philipp-spiess]

Fixed via #1071

[toolmantim]

@DanTup we’d love you to get the in-editor version of this working next! It’s designed above, but I’m expecting we may need to iterate/adjust as we start to get things working.

[DanTup]

@toolmantim has this already been started? (the video above looks like a working implementation but I'm unfamiliar with Figma so don't know if this is just mocked up.. it looks very real)

[toolmantim]

@DanTup nothing started, the above is just a mockup! 😊