Skip to content

feat(vertexai): implement emitting events in addition to current beha… #2942

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

LuizDMM
Copy link

@LuizDMM LuizDMM commented May 13, 2025

✅ PR Requirements

  • I have added tests that cover my changes.
  • If adding a new instrumentation or changing an existing one, I've added screenshots from some observability platform showing the change.
  • PR name follows conventional commits format: feat(instrumentation): ... or fix(instrumentation): ....
  • (If applicable) I have updated the documentation accordingly.

📌 Issue Requirements

  • Make sure to keep the current way of emitting events and the new (event-based) way with the official semantic
  • Add a config to each instrumentation called use_legacy_attributes which defaults to true, and if set to false emit events instead of the prompt / completion attributes.
  • Propagate a new initialization parameter of the SDK called use_legacy_attributes that sets this config in all instrumentations.

Important

Adds event emission to Vertex AI instrumentation with a configuration option to toggle between legacy and event-based attributes.

  • Behavior:
    • Adds event emission functionality to Vertex AI instrumentation, controlled by use_legacy_attributes config in config.py.
    • Introduces emit_prompt_events and emit_response_events in event_emitter.py to handle event logging.
    • Modifies VertexAIInstrumentor in __init__.py to support event logging based on configuration.
  • Utilities:
    • Adds should_emit_events and should_send_prompts in utils.py to control event emission and prompt logging.
    • Refactors span attribute setting into span_utils.py.
  • Tests:
    • Updates tests in disabled_test_bison.py and disabled_test_gemini.py to verify event emission and legacy behavior.
    • Adds fixtures in conftest.py to support testing with different configurations.

This description was created by Ellipsis for e023a64. You can customize this summary. It will automatically update as commits are pushed.

…vior

-  Add "use_legacy_attributes" to Config and the Instrumentor constructor, defaulting to True;
- emit events for user prompts and AI responses, following [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-events/\\\);
- implement comprehensive tests to verify the new functionality and ensure that, when use_legacy_attributes == True, the existing behavior remains unchanged;
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Changes requested ❌

Reviewed everything up to e023a64 in 2 minutes and 15 seconds. Click for details.
  • Reviewed 1154 lines of code in 10 files
  • Skipped 0 files when reviewing.
  • Skipped posting 5 draft comments. View those below.
  • Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/opentelemetry-instrumentation-vertexai/opentelemetry/instrumentation/vertexai/utils.py:10
  • Draft comment:
    The default logic in should_send_prompts uses 'os.getenv(...) or "true"'; confirm that this default of 'true' is intended, as it may always enable sending prompt contents.
  • Reason this comment was not posted:
    Confidence changes required: 50% <= threshold 50% None
2. packages/opentelemetry-instrumentation-vertexai/opentelemetry/instrumentation/vertexai/__init__.py:298
  • Draft comment:
    Typographical error: The docstring in the VertexAIInstrumentor class contains a typo ('VertextAI'). It should be corrected to 'VertexAI' to maintain consistency.
  • Reason this comment was not posted:
    Comment was on unchanged code.
3. packages/opentelemetry-instrumentation-vertexai/opentelemetry/instrumentation/vertexai/event_emitter.py:125
  • Draft comment:
    Typo in comment: The phrase '... if it is the same as the in the event name.' could be reworded to '... if it is the same as in the event name.' for clarity.
  • Reason this comment was not posted:
    Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 10% vs. threshold = 50% While the comment is technically correct about the grammatical error, it's an extremely minor issue in an internal comment. The meaning is still clear despite the typo. According to the rules, we should not make purely informative comments or comments that are unimportant. Documentation typos that don't impact understanding fall into this category. The typo could potentially cause slight confusion for future maintainers. Documentation quality is important for long-term maintainability. While documentation quality matters, this particular typo is so minor that it doesn't impact comprehension. The rules explicitly state not to make unimportant comments. Delete this comment as it addresses an unimportant documentation issue that doesn't impact code functionality or comprehension.
4. packages/opentelemetry-instrumentation-vertexai/opentelemetry/instrumentation/vertexai/event_emitter.py:51
  • Draft comment:
    Potential typo in finish_reason_map: the value for key 8 is set to 'spii'. Please verify if 'spii' is intended or if it should be corrected (e.g. to 'spi' or another appropriate term).
  • Reason this comment was not posted:
    Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 10% vs. threshold = 50% SPII typically stands for "Sensitive Personal Identifiable Information" which is a common term in data privacy. Given this is a finish reason in a Vertex AI context, it's likely this is the intended meaning. The comment suggests changing it without any evidence that 'spii' is incorrect. We should not suggest changes to API-defined constants without strong evidence. I could be wrong about SPII being the intended meaning. Maybe there's documentation from Vertex AI that specifies a different term. Even if I'm not 100% sure about SPII being the meaning, we shouldn't suggest changing API-defined constants without clear evidence of an error. The burden of proof is on showing it's wrong, not speculating about typos. Delete the comment. It suggests changing what appears to be an API-defined constant without any evidence that it's incorrect, and 'spii' is likely the intended value representing "Sensitive Personal Identifiable Information".
5. packages/opentelemetry-instrumentation-vertexai/opentelemetry/instrumentation/vertexai/utils.py:32
  • Draft comment:
    Typographical error: 'OpenLLMetry' appears in the logging message. Please correct it to 'OpenTelemetry' to ensure consistency in our telemetry naming.
  • Reason this comment was not posted:
    Comment was not on a location in the diff, so it can't be submitted as a review comment.

Workflow ID: wflow_zwjfmbz3uB3Zy3qH

You can customize Ellipsis by changing your verbosity settings, reacting with 👍 or 👎, replying to comments, or adding code review rules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant