Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add server telemetry #1687

Merged
merged 31 commits into from Sep 8, 2022
Merged

Add server telemetry #1687

merged 31 commits into from Sep 8, 2022

Conversation

dvsrepo
Copy link
Member

@dvsrepo dvsrepo commented Aug 20, 2022

Closes #1629

  • Add metrics documentation (how to disable, which metrics will be collected,...)
  • Minimal event to register: /api/me, /api/datasets/.../bulk and API errors
  • Define metrics component (encapsulate communication with the analytics backend, environment settings,...)
  • Configure analytics backend
  • Document events and attributes

@frascuchon frascuchon linked an issue Aug 23, 2022 that may be closed by this pull request
4 tasks
@codecov
Copy link

codecov bot commented Aug 23, 2022

Codecov Report

Base: 95.80% // Head: 95.75% // Decreases project coverage by -0.04% ⚠️

Coverage data is based on head (9fdfc3e) compared to base (f947dfe).
Patch coverage: 91.76% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1687      +/-   ##
==========================================
- Coverage   95.80%   95.75%   -0.05%     
==========================================
  Files         144      145       +1     
  Lines        6603     6669      +66     
==========================================
+ Hits         6326     6386      +60     
- Misses        277      283       +6     
Flag Coverage Δ
pytest 95.75% <91.76%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/rubrix/server/commons/telemetry.py 88.88% <88.88%> (ø)
src/rubrix/server/apis/v0/handlers/text2text.py 98.30% <100.00%> (ø)
...rix/server/apis/v0/handlers/text_classification.py 100.00% <100.00%> (ø)
...ix/server/apis/v0/handlers/token_classification.py 100.00% <100.00%> (ø)
src/rubrix/server/apis/v0/handlers/users.py 100.00% <100.00%> (ø)
src/rubrix/server/errors/api_errors.py 100.00% <100.00%> (ø)
src/rubrix/server/services/storage/service.py 100.00% <100.00%> (ø)
.../rubrix/server/services/tasks/text2text/service.py 100.00% <100.00%> (ø)
...rver/services/tasks/text_classification/service.py 96.66% <100.00%> (ø)
...ver/services/tasks/token_classification/service.py 100.00% <100.00%> (ø)
... and 1 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@frascuchon frascuchon changed the title Add telemetry Add server telemetry Aug 25, 2022
@frascuchon
Copy link
Member

frascuchon commented Aug 25, 2022

Here are the 3 kinds of events to track. Please, @dvsrepo, take a look and let me know your thoughts:

'ServerErrorFound', {
    'code': 'rubrix.api.errors::EntityNotFoundError',
    'machine': 'x86_64',
    'platform': 'Linux-5.15.0-1017-azure-x86_64-with-glibc2.10',
    'python_version': '3.8.13',
    'rubrix_version': '0.17.0.dev0',
    'sys_version': '#20~20.04.1-Ubuntu SMP Fri Aug 5 12:16:53 UTC 2022',
    'system': 'Linux'
}


'LogRecordsRequested', {
    'machine': 'x86_64',
    'platform': 'macOS-10.16-x86_64-i386-64bit',
    'python_version': '3.8.13',
    'records': 1,
    'rubrix_version': '0.17.0.dev0',
    'sys_version': 'Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:22 PDT 2022; root:xnu-8020.121.3~4/RELEASE_X86_64',
    'system': 'Darwin',
    'task': 'TextClassification'
}


'UserInfoRequested', {
  'machine': 'x86_64',
  'platform': 'macOS-10.16-x86_64-i386-64bit',
  'python_version': '3.8.13',
  'rubrix_version': '0.17.0.dev0',
  'sys_version': 'Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:22 PDT 2022; root:xnu-8020.121.3~4/RELEASE_X86_64',
  'system': 'Darwin'
}

@frascuchon frascuchon force-pushed the feat/usage-telemetry branch 3 times, most recently from 969dd6a to 1abd1e2 Compare August 29, 2022 13:36
@frascuchon
Copy link
Member

@dvsrepo Please, take a look at the final docs.

Once it is reviewed, we can merge and start testing with collected data

@frascuchon frascuchon self-requested a review September 1, 2022 11:13
@dvsrepo
Copy link
Member Author

dvsrepo commented Sep 2, 2022

@dvsrepo Please, take a look at the final docs.

Once it is reviewed, we can merge and start testing with collected data

Looks great!

@frascuchon frascuchon merged commit d57afd1 into master Sep 8, 2022
@frascuchon frascuchon deleted the feat/usage-telemetry branch September 8, 2022 15:19
frascuchon pushed a commit that referenced this pull request Sep 27, 2022
frascuchon pushed a commit that referenced this pull request Sep 27, 2022
frascuchon pushed a commit that referenced this pull request Sep 27, 2022
frascuchon pushed a commit that referenced this pull request Sep 29, 2022
frascuchon pushed a commit that referenced this pull request Oct 3, 2022
frascuchon added a commit that referenced this pull request Oct 5, 2022
Some changes related to the usage telemetry:

- Passing system data as event context (will simplify later integrations)
- Show warning message on server start to clear preventing users
- Include the Telemetry section inside the Reference doc.

Refs: #1687
frascuchon added a commit that referenced this pull request Oct 5, 2022
Some changes related to the usage telemetry:

- Passing system data as event context (will simplify later integrations)
- Show warning message on server start to clear preventing users
- Include the Telemetry section inside the Reference doc.

Refs: #1687
frascuchon pushed a commit that referenced this pull request Oct 5, 2022
(cherry picked from commit d57afd1)

Some changes related to the usage telemetry:

- Passing system data as event context (will simplify later integrations)
- Show warning message on server start to clear preventing users
- Include the Telemetry section inside the Reference doc.
frascuchon pushed a commit that referenced this pull request Oct 5, 2022
(cherry picked from commit d57afd1)

Some changes related to the usage telemetry:

- Passing system data as event context (will simplify later integrations)
- Show warning message on server start to clear preventing users
- Include the Telemetry section inside the Reference doc.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Usage telemetry
2 participants