Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add aichat Amplitude/Statsig analytics #58397

Merged
merged 12 commits into from
May 6, 2024
Merged

Conversation

fisher-alice
Copy link
Contributor

@fisher-alice fisher-alice commented May 3, 2024

This PR adds logging sent to both Amplitude and Statsig for the following aichat events:

  • When a user updates a model customization (selected model, temperature, system prompt, or retrievals) that results in a change in the model
  • When a user saves or publishes the model card information that results in a change
  • When a user clicks on the 'Clear' or 'Copy Chat' buttons.

I added these events according to both the tech spec and this Slack thread.
For now, we will observe the volume of events before deciding to sample. 342575c

Screencast video showing logging of events in dev console:

analytics-aichat.mp4

Links

jira

Testing story

I tested locally. Screencast video above.

Deployment strategy

Follow-up work

Privacy

Security

Caching

PR Checklist:

  • Tests provide adequate coverage
  • Privacy and Security impacts have been assessed
  • Code is well-commented
  • New features are translatable or updates will not break translations
  • Relevant documentation has been added or updated
  • User impact is well-understood and desirable
  • Pull Request is labeled appropriately
  • Follow-up work items (including potential tech debt) are tracked and linked

@@ -260,6 +260,11 @@ const EVENTS = {

// Teacher Homepage
TEACHER_HOMEPAGE_VISITED: 'Teacher Homepage Visited',

// Aichat
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@samantha-code - lmk if these are okay. I based constants/descriptions from the spec.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked with Sam via Slack and she approved.

@@ -2,3 +2,5 @@ import {ModelDescription} from './types';
import modelsJson from '@cdo/static/aichat/modelDescriptions.json';

export const modelDescriptions: ModelDescription[] = modelsJson;

export const ANALYTICS_SAMPLE_RATE = 0.1;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From spec, we want to consider sampling (1/100 or 1/1000).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked with Sam and she said let's hold off on sampling for now. Commit that removes sampling: 342575c

@fisher-alice fisher-alice changed the title Add preliminary aichat Amplitude/Statsig analytics Add aichat Amplitude/Statsig analytics May 3, 2024
@fisher-alice fisher-alice marked this pull request as ready for review May 3, 2024 20:11
@fisher-alice fisher-alice requested a review from a team May 3, 2024 20:11
Copy link
Contributor

@bencodeorg bencodeorg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

apps/src/aichat/redux/aichatRedux.ts Show resolved Hide resolved
apps/src/aichat/redux/aichatRedux.ts Outdated Show resolved Hide resolved
apps/src/aichat/redux/aichatRedux.ts Outdated Show resolved Hide resolved
apps/src/aichat/views/AichatView.tsx Show resolved Hide resolved
@fisher-alice fisher-alice merged commit 19c24d6 into staging May 6, 2024
2 checks passed
@fisher-alice fisher-alice deleted the alice/aichat-analytics branch May 6, 2024 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants