Cody completions: Add metrics for completion events #51350
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #51347
This adds metrics for inline and multiline suggestions so we can know how often we trigger requests, how often they are presented to our users, and what the acceptance rate is (note: acceptance rate doesn't work for multiline suggestions yet as we only allow copy-pasting for this and don't have a distinct UI action for accepting)
New events added:
CodyVSCodeExtension:completion:started
(type: 'inline' | 'multiline'
)CodyVSCodeExtension:completion:suggested
(type: 'inline' | 'multiline'
)CodyVSCodeExtension:completion:accepted
(type: 'inline'
)Open question: Logging completion on/off (cc @malomarrec)
It's not clear yet how we can measure total number of people that have suggestions enabled (changing the setting for this is not a discrete event since it can happen outside of our control). We could include the state of the feature flag in other Cody related pings but I really think given the occurence of inline completions, these will swiftly become the most-frequently-used Cody feature (when enabled). This leads me to believe that we should calculate DAUs based on the presence of an inline suggestion event.
Test plan