Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cody completions: Add metrics for completion events #51350

Merged
merged 5 commits into from
May 2, 2023

Conversation

philipp-spiess
Copy link
Contributor

Closes #51347

This adds metrics for inline and multiline suggestions so we can know how often we trigger requests, how often they are presented to our users, and what the acceptance rate is (note: acceptance rate doesn't work for multiline suggestions yet as we only allow copy-pasting for this and don't have a distinct UI action for accepting)

New events added:

  • CodyVSCodeExtension:completion:started (type: 'inline' | 'multiline')
    • Measure when we start a request to the backend for new suggestions (and in turn hit the LLM APIs). I wanted to add this so we can calculate how often these are aborted.
  • CodyVSCodeExtension:completion:suggested (type: 'inline' | 'multiline')
    • Measures when we receive a result from the backend and are displaying those to the user (Note: VS Code does not always present inline suggestions to the user, even if this event fires. We'll have to investigate why that's the case and make sure we don't start a completion unless they will be shown).
  • CodyVSCodeExtension:completion:accepted (type: 'inline')
    • Measures how often completions are inserted.

Open question: Logging completion on/off (cc @malomarrec)

It's not clear yet how we can measure total number of people that have suggestions enabled (changing the setting for this is not a discrete event since it can happen outside of our control). We could include the state of the feature flag in other Cody related pings but I really think given the occurence of inline completions, these will swiftly become the most-frequently-used Cody feature (when enabled). This leads me to believe that we should calculate DAUs based on the presence of an inline suggestion event.

Test plan

@philipp-spiess philipp-spiess requested review from beyang, malomarrec and a team May 2, 2023 11:56
@philipp-spiess philipp-spiess self-assigned this May 2, 2023
@cla-bot cla-bot bot added the cla-signed label May 2, 2023
@github-actions github-actions bot added the team/code-exploration Issues owned by the Code Exploration team label May 2, 2023
@vdavid
Copy link
Contributor

vdavid commented May 2, 2023

(Disclaimer: Haven't looked into the code, just read the desc) Thought: I think we could still detect copy events on the multiline results if we wanted metrics on those as well.

@philipp-spiess
Copy link
Contributor Author

(Disclaimer: Haven't looked into the code, just read the desc) Thought: I think we could still detect copy events on the multiline results if we wanted metrics on those as well.

Yea I was thinking the same but I’m worried about over-counting here if users copy more than once. Perhaps we can make sure to only trigger it once for a set of suggestions though. I can look into that.

@philipp-spiess
Copy link
Contributor Author

@vdavid Hm it's not so easy as just adding a copy listener. It is possible to monkey-patch the default copy behavior though: https://stackoverflow.com/a/44812797 but I’m hesitant to do that just now. I'd like to merge this without tracking usage of multiline completion and look into the actual usage first. My hunch is that more people use inline completions over multiline completions and we should optimize for the right thing!

@philipp-spiess philipp-spiess merged commit 495f258 into main May 2, 2023
17 checks passed
@philipp-spiess philipp-spiess deleted the ps/cody-autocomplete-stats branch May 2, 2023 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed team/code-exploration Issues owned by the Code Exploration team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Metrics: Add VSCodeExtension:completion:suggested, VSCodeExtension:completion:accepted, Completion on/off
2 participants