-
Notifications
You must be signed in to change notification settings - Fork 754
fix(sdk): manual report of usage data #3045
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Important
Looks good to me! 👍
Reviewed everything up to 4a31ef7 in 27 seconds. Click for details.
- Reviewed
63
lines of code in2
files - Skipped
0
files when reviewing. - Skipped posting
2
draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/traceloop-sdk/tests/test_manual.py:29
- Draft comment:
Test now calls report_usage with LLMUsage. Consider adding assertions to verify that the usage attributes (like prompt/completion tokens) are correctly set on the finished span. - Reason this comment was not posted:
Confidence changes required:33%
<= threshold50%
None
2. packages/traceloop-sdk/traceloop/sdk/tracing/manual.py:48
- Draft comment:
The new report_usage method sets usage tokens on the span. Consider adding a docstring and input validation (if needed) to ensure robustness, and verify that the corresponding SpanAttributes constants exist. - Reason this comment was not posted:
Confidence changes required:33%
<= threshold50%
None
Workflow ID: wflow_Kjvjsq7WiBMDKFEV
You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.
bf8c704
to
3fa7da8
Compare
3fa7da8
to
7ab074e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Important
Looks good to me! 👍
Reviewed 3fa7da8 in 1 minute and 15 seconds. Click for details.
- Reviewed
311
lines of code in4
files - Skipped
0
files when reviewing. - Skipped posting
5
draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/traceloop-sdk/tests/test_manual.py:7
- Draft comment:
Removal of the @pytest.mark.vcr decorator is intentional for a manual test setup. Just ensure that this change is deliberate and that offline testing with hardcoded responses covers all needed scenarios. - Reason this comment was not posted:
Comment did not seem useful. Confidence is useful =30%
<= threshold50%
The comment is asking the PR author to ensure that the change is deliberate and that offline testing covers all scenarios. This violates the rule against asking the author to ensure behavior is intended or tested. However, it does mention the removal of a specific decorator, which could be useful if it were rephrased to confirm the intention of the removal.
2. packages/traceloop-sdk/tests/test_manual.py:21
- Draft comment:
Consider using a single multi-line literal for the response string instead of concatenating two strings. This may improve readability. - Reason this comment was not posted:
Confidence changes required:50%
<= threshold50%
None
3. packages/traceloop-sdk/traceloop/sdk/tracing/manual.py:26
- Draft comment:
The 'pass' statement in the init method is redundant and can be removed. - Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.
4. packages/traceloop-sdk/traceloop/sdk/tracing/manual.py:77
- Draft comment:
The explicit call to span.end() in the finally block may be redundant since the context manager from start_as_current_span already ends the span automatically. - Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.
5. packages/traceloop-sdk/traceloop/sdk/tracing/manual.py:69
- Draft comment:
Avoid using 'type' as a parameter name since it shadows a built-in. Consider renaming it (e.g., to request_type). - Reason this comment was not posted:
Comment was on unchanged code.
Workflow ID: wflow_6t2TWOq0xbeajxSE
You can customize by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.
feat(instrumentation): ...
orfix(instrumentation): ...
.Important
Enhances SDK manual reporting by adding usage metrics reporting and updating tests accordingly.
report_usage()
method inLLMSpan
to report usage metrics likeprompt_tokens
,completion_tokens
,total_tokens
,cache_creation_input_tokens
, andcache_read_input_tokens
.test_manual_report()
intest_manual.py
to use mocked response data and validate new usage metrics.test_manual_report.yaml
andtest_resource_attributes.yaml
, indicating a change in test data handling.cache_creation_input_tokens
andcache_read_input_tokens
toLLMUsage
class.This description was created by
for 3fa7da8. You can customize this summary. It will automatically update as commits are pushed.