feat(telemetry) Instrument traces with more attributes and make them available to OTEL users

[heaventourist]

Summary

Added more attributes to OTel trace

Details

See packages/core/src/telemetry/constants.ts for added attributes.

These new attributes will be used by Google's internal project to surface the details of the traces to all Googlers.

Related Issues

Closes #20389
Closes #9217

How to Validate

Enable telemetry, specify the outfile and run make start, then check the log file as specified.

Pre-Merge Checklist

• Updated relevant documentation and README (if needed)
• Added/updated tests (if needed)
• Noted breaking changes (if any)
• Validated on required platforms/methods:
  • MacOS
    • npm run
    • npx
    • Docker
    • Podman
    • Seatbelt
  • Windows
    • npm run
    • npx
    • Docker
  • Linux
    • npm run
    • npx
    • Docker

[gemini-code-assist]

Summary of Changes

Hello @heaventourist, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the Gemini CLI's telemetry capabilities by integrating comprehensive OpenTelemetry tracing. The changes introduce a structured approach to capturing detailed information about key operations such as LLM interactions, tool executions, and agent activities. This instrumentation provides deeper insights into the CLI's internal workings, which will be leveraged by internal Google projects for improved monitoring and debugging.

Highlights

• Enhanced OpenTelemetry Tracing: Implemented detailed OpenTelemetry tracing for various Gemini CLI operations, including LLM calls, tool executions, agent calls, and user/system prompts.
• New Semantic Conventions: Introduced new semantic conventions and attributes for GenAI tracing, enhancing observability into AI interactions and standardizing trace attribute names.
• Updated Documentation: Updated documentation to provide clear guidance on enabling and interpreting dev traces, including a comprehensive list of captured attributes.
• Comprehensive Test Coverage: Added and updated tests across core modules to ensure correct tracing behavior, attribute capture, and error handling for all instrumented operations.

Changelog

• docs/cli/telemetry.md
  • Updated the telemetry documentation to include a new section on "Traces (Dev Tracing)", detailing the OpenTelemetry GenAI semantic conventions and the specific attributes captured for various operations.
• docs/local-development.md
  • Modified the runInDevTraceSpan example to use the GeminiCliOperation enum for specifying operation types, improving clarity and consistency.
• packages/cli/src/ui/hooks/useGeminiStream.test.tsx
  • Added new tests to verify the correct tracing of SystemPrompt and UserPrompt operations within useGeminiStream, including mocking runInDevTraceSpan and asserting metadata capture.
• packages/cli/src/ui/hooks/useGeminiStream.ts
  • Integrated GeminiCliOperation to differentiate between UserPrompt and SystemPrompt when calling runInDevTraceSpan for submitQuery operations.
• packages/core/src/agents/subagent-tool.test.ts
  • Extended testing for SubagentTool to include verification of AgentCall tracing, ensuring agent-specific attributes are correctly set.
• packages/core/src/agents/subagent-tool.ts
  • Implemented AgentCall tracing for SubagentTool executions, capturing agent name and description as attributes.
• packages/core/src/core/coreToolScheduler.test.ts
  • Added tests to confirm that CoreToolScheduler correctly initiates ScheduleToolCalls traces and captures relevant input data.
• packages/core/src/core/coreToolScheduler.ts
  • Modified the schedule method to use GeminiCliOperation.ScheduleToolCalls for tracing, capturing the input request and output results.
• packages/core/src/core/loggingContentGenerator.test.ts
  • Enhanced tests for LoggingContentGenerator to thoroughly validate LLMCall tracing, including the capture of request model, prompt name, system instructions, tool definitions, input/output messages, and token usage.
• packages/core/src/core/loggingContentGenerator.ts
  • Updated generateContent, generateContentStream, and embedContent methods to instrument LLMCall traces, capturing detailed attributes such as model, prompt name, system instructions, tool definitions, input/output content, and token counts.
• packages/core/src/scheduler/scheduler.test.ts
  • Updated Scheduler tests to include assertions for ScheduleToolCalls tracing, verifying input capture.
• packages/core/src/scheduler/scheduler.ts
  • Enhanced the schedule method to trace ScheduleToolCalls operations, ensuring input requests and completed tool calls are recorded.
• packages/core/src/scheduler/scheduler_parallel.test.ts
  • Added tests for parallel scheduler execution to confirm ScheduleToolCalls tracing with correct input capture.
• packages/core/src/scheduler/tool-executor.test.ts
  • Introduced tests for ToolExecutor to validate ToolCall tracing, ensuring tool name, call ID, description, input, output, and errors are correctly recorded.
• packages/core/src/scheduler/tool-executor.ts
  • Implemented ToolCall tracing within the ToolExecutor, capturing comprehensive details about each tool execution, including its name, call ID, description, input, output, and any errors.
• packages/core/src/telemetry/constants.ts
  • Defined a new GeminiCliOperation enum and a set of GEN_AI_ prefixed constants for OpenTelemetry GenAI semantic conventions, standardizing trace attribute names.
• packages/core/src/telemetry/index.ts
  • Exported the newly defined telemetry constants to make them accessible throughout the module.
• packages/core/src/telemetry/trace.test.ts
  • Added a new test file to thoroughly test the runInDevTraceSpan utility, covering scenarios like conditional tracing, attribute propagation, error handling, and manual span ending.
• packages/core/src/telemetry/trace.ts
  • Refactored runInDevTraceSpan to use the GeminiCliOperation enum for span naming, added default agent and conversation ID attributes, and updated attribute keys to align with the new GEN_AI_ semantic conventions.
• packages/sdk/src/agent.integration.test.ts
  • Added beforeEach and afterEach hooks to manage the GEMINI_API_KEY environment variable for integration tests.
• packages/vscode-ide-companion/NOTICES.txt
  • Updated the entry for path-to-regexp to reflect that its repository and license text were not found.

Activity

• The pull request author has completed the pre-merge checklist, indicating that documentation has been updated, tests have been added/updated, and validation has been performed on MacOS and Linux.
• No breaking changes were noted by the author.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request enhances OpenTelemetry tracing instrumentation by adding GenAI-specific semantic attributes to trace spans for LLM calls, tool executions, and agent interactions. However, a critical security vulnerability exists: the implementation captures and transmits full user prompts, conversation history, LLM responses, system instructions, and tool definitions as span attributes. This poses a significant risk of sensitive data exposure (PII, secrets, proprietary code) to the telemetry backend, as these traces are intended for broad organizational sharing. While guarded by an environment variable, the default behavior when enabled is to log full payloads without redaction. Additionally, consider further utilizing the new tracing capabilities within the sub-agent tooling.

[heaventourist]

Original PR: #18972

[jerop]

/gemini review

[jerop]

@heaventourist please rebase and resolve conflict

also file an issue for this work, add add a link in description e.g. Closes #123

[jerop]

/gemini review

[gemini-code-assist]

Code Review

This pull request enhances OpenTelemetry tracing by adding more detailed attributes to spans, which is a great improvement for observability. The changes are consistently applied across the codebase. The identified issue in subagent-tool.ts regarding incorrect tracing utility usage has been retained as it is a valid concern not covered by the provided rules, and a suggestion to fix it is included.

[jerop]

please remove this sensitive word

[heaventourist]

Used another attribute to illustrate the use case.