Skip to content

fix(sdk): manual report of usage data #3045

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 24, 2025
Merged

fix(sdk): manual report of usage data #3045

merged 1 commit into from
Jun 24, 2025

Conversation

nirga
Copy link
Member

@nirga nirga commented Jun 24, 2025

  • I have added tests that cover my changes.
  • If adding a new instrumentation or changing an existing one, I've added screenshots from some observability platform showing the change.
  • PR name follows conventional commits format: feat(instrumentation): ... or fix(instrumentation): ....
  • (If applicable) I have updated the documentation accordingly.

Important

Enhances SDK manual reporting by adding usage metrics reporting and updating tests accordingly.

  • Behavior:
    • Adds report_usage() method in LLMSpan to report usage metrics like prompt_tokens, completion_tokens, total_tokens, cache_creation_input_tokens, and cache_read_input_tokens.
    • Updates test_manual_report() in test_manual.py to use mocked response data and validate new usage metrics.
  • Testing:
    • Removes test_manual_report.yaml and test_resource_attributes.yaml, indicating a change in test data handling.
  • Models:
    • Adds cache_creation_input_tokens and cache_read_input_tokens to LLMUsage class.

This description was created by Ellipsis for 3fa7da8. You can customize this summary. It will automatically update as commits are pushed.

Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

Looks good to me! 👍

Reviewed everything up to 4a31ef7 in 27 seconds. Click for details.
  • Reviewed 63 lines of code in 2 files
  • Skipped 0 files when reviewing.
  • Skipped posting 2 draft comments. View those below.
  • Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/traceloop-sdk/tests/test_manual.py:29
  • Draft comment:
    Test now calls report_usage with LLMUsage. Consider adding assertions to verify that the usage attributes (like prompt/completion tokens) are correctly set on the finished span.
  • Reason this comment was not posted:
    Confidence changes required: 33% <= threshold 50% None
2. packages/traceloop-sdk/traceloop/sdk/tracing/manual.py:48
  • Draft comment:
    The new report_usage method sets usage tokens on the span. Consider adding a docstring and input validation (if needed) to ensure robustness, and verify that the corresponding SpanAttributes constants exist.
  • Reason this comment was not posted:
    Confidence changes required: 33% <= threshold 50% None

Workflow ID: wflow_Kjvjsq7WiBMDKFEV

You can customize Ellipsis by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.

@nirga nirga force-pushed the manual-report-tokens branch from bf8c704 to 3fa7da8 Compare June 24, 2025 06:05
@nirga nirga force-pushed the manual-report-tokens branch from 3fa7da8 to 7ab074e Compare June 24, 2025 06:05
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

Looks good to me! 👍

Reviewed 3fa7da8 in 1 minute and 15 seconds. Click for details.
  • Reviewed 311 lines of code in 4 files
  • Skipped 0 files when reviewing.
  • Skipped posting 5 draft comments. View those below.
  • Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/traceloop-sdk/tests/test_manual.py:7
  • Draft comment:
    Removal of the @pytest.mark.vcr decorator is intentional for a manual test setup. Just ensure that this change is deliberate and that offline testing with hardcoded responses covers all needed scenarios.
  • Reason this comment was not posted:
    Comment did not seem useful. Confidence is useful = 30% <= threshold 50% The comment is asking the PR author to ensure that the change is deliberate and that offline testing covers all scenarios. This violates the rule against asking the author to ensure behavior is intended or tested. However, it does mention the removal of a specific decorator, which could be useful if it were rephrased to confirm the intention of the removal.
2. packages/traceloop-sdk/tests/test_manual.py:21
  • Draft comment:
    Consider using a single multi-line literal for the response string instead of concatenating two strings. This may improve readability.
  • Reason this comment was not posted:
    Confidence changes required: 50% <= threshold 50% None
3. packages/traceloop-sdk/traceloop/sdk/tracing/manual.py:26
  • Draft comment:
    The 'pass' statement in the init method is redundant and can be removed.
  • Reason this comment was not posted:
    Comment was not on a location in the diff, so it can't be submitted as a review comment.
4. packages/traceloop-sdk/traceloop/sdk/tracing/manual.py:77
  • Draft comment:
    The explicit call to span.end() in the finally block may be redundant since the context manager from start_as_current_span already ends the span automatically.
  • Reason this comment was not posted:
    Comment was not on a location in the diff, so it can't be submitted as a review comment.
5. packages/traceloop-sdk/traceloop/sdk/tracing/manual.py:69
  • Draft comment:
    Avoid using 'type' as a parameter name since it shadows a built-in. Consider renaming it (e.g., to request_type).
  • Reason this comment was not posted:
    Comment was on unchanged code.

Workflow ID: wflow_6t2TWOq0xbeajxSE

You can customize Ellipsis by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.

@nirga nirga merged commit fa73d91 into main Jun 24, 2025
9 checks passed
@nirga nirga deleted the manual-report-tokens branch June 24, 2025 06:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants