feat(agent): Add opt-in flag to include tool specs in traces for evaluation #1113

Ratish1 · 2025-10-30T19:56:25Z

Description

This PR implements a feature to include the specifications of all available tools in the main invoke_agent trace span. This provides context for external evaluation frameworks to accurately assess the agent's tool selection.

I went with the opt-in approach because:

Tool schemas can be 50KB+ per trace with many tools
Follows industry best practice (LangChain, OTel recommendations)
OpenTelemetry: No official standard for gen_ai.agent.tools yet
LangChain: Uses opt-in tracing for tool metadata due to data size concerns
OpenAI/Anthropic: Both use name, description, and input schema (JSON Schema format)

Key Changes

A new include_tools_in_trace: bool = False parameter has been added to the Agent constructor.
When enabled, the agent serializes the specifications (name, description, inputSchema, outputSchema) of all tools from the ToolRegistry and attaches the result to the invoke_agent span under the gen_ai.agent.tools attribute.
Unit tests have been added to verify both the enabled and disabled states of this feature.

Related Issues

Closes #1083

Documentation PR

N/A

Type of Change

New feature

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli

I ran hatch run prepare
I ran unit tests:
test_agent_does_not_include_tools_in_trace_by_default -> passed
test_agent_includes_tools_in_trace_when_enabled -> passed

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

codecov · 2025-10-31T15:18:37Z

Codecov Report

❌ Patch coverage is 77.77778% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/strands/agent/agent.py	77.77%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

poshinchen

Thanks for the contribution.
After the discussion within the team, I believe we should:

Move the attributes to invoke_agent span > gen_ai.tool.definitions
(semantic conventions link).
Move the flag (include_tool_definitions) into StrandsTelemetry class instead of in the Agent class as this is specifically for the trace.

Ratish1 · 2025-11-04T07:44:32Z

Thanks for the contribution. After the discussion within the team, I believe we should:

Move the attributes to invoke_agent span > gen_ai.tool.definitions
(semantic conventions link).

Move the flag (include_tool_definitions) into StrandsTelemetry class instead of in the Agent class as this is specifically for the trace.

Thanks for the feedback, will change it right away @poshinchen .

Ratish1 · 2025-11-04T08:29:34Z

Oh no, I messed up something by force pushing since I was having some rebase problems locally and now it closed the PR automatically. Sorry, could you reopen it or should I create a new PR @poshinchen . Nvm i was able to reopen it.

poshinchen

Hi, sorry

After a careful read from the otel documentation. We are expecting customers to set multiple values in OTEL_SEMCONV_STABILITY_OPT_IN env var

As an example: OTEL_SEMCONV_STABILITY_OPT_IN="gen_ai_latest_experimental,gen_ai_tool_definitions" allows latest semantic conventions, and opt-in gen_ai.tool.definitions.

That being said, could you do the following:

In tracer.py, modify the parsing logic for use_latest_genai_conventions and include_tool_definitions (new).
pass all_tools_config to _start_agent_trace_span and move tool_details = [...] into that method.
I wonder if it's possible to pass them as array?

From the description:

It’s expected to be an array of objects where each object represents a tool definition. In case a serialized string is available to the instrumentation, the instrumentation SHOULD do the best effort to deserialize it to an array. When recorded on spans, it MAY be recorded as a JSON string if structured format is not supported and SHOULD be recorded in structured form otherwise.

Ratish1 · 2025-11-04T17:45:58Z

Hi, sorry

After a careful read from the otel documentation. We are expecting customers to set multiple values in OTEL_SEMCONV_STABILITY_OPT_IN env var

As an example: OTEL_SEMCONV_STABILITY_OPT_IN="gen_ai_latest_experimental,gen_ai_tool_definitions" allows latest semantic conventions, and opt-in gen_ai.tool.definitions.

That being said, could you do the following:

In tracer.py, modify the parsing logic for use_latest_genai_conventions and include_tool_definitions (new).

pass all_tools_config to _start_agent_trace_span and move tool_details = [...] into that method.

I wonder if it's possible to pass them as array?

From the description:

It’s expected to be an array of objects where each object represents a tool definition. In case a serialized string is available to the instrumentation, the instrumentation SHOULD do the best effort to deserialize it to an array. When recorded on spans, it MAY be recorded as a JSON string if structured format is not supported and SHOULD be recorded in structured form otherwise.

No problem , will change it according your feedback. Thanks. Also, regarding your question about passing the data as an array: I looked into it, and you're right that the spec prefers a structured format. However, the OpenTelemetry Python SDK's set_attribute function doesn't have guaranteed support for complex nested objects like a list of dictionaries. What are your thoughts on this, open to being wrong about this.

poshinchen · 2025-11-04T18:51:01Z

src/strands/agent/agent.py

-        self.trace_span = self._start_agent_trace_span(messages)
+        self.trace_span = self._start_agent_trace_span(
+            messages,
+            all_tools_config=self.tool_registry.get_all_tools_config() or {},


or {} should not be required (source). And, what about just tools_config?

Yes will change it.

poshinchen · 2025-11-04T18:58:19Z

src/strands/telemetry/config.py

        Args:
            tracer_provider: Optional pre-configured tracer provider.
                If None, a new one will be created and set as global.
+            include_tool_definitions: Whether to include tool definitions in traces.


:) should be removed

poshinchen · 2025-11-04T19:01:48Z

src/strands/agent/agent.py

+        if self.tracer.include_tool_definitions and all_tools_config:
+            try:
+                tool_details = [
+                    {
+                        "name": name,
+                        "description": spec.get("description"),
+                        "inputSchema": spec.get("inputSchema"),
+                        "outputSchema": spec.get("outputSchema"),
+                    }
+                    for name, spec in all_tools_config.items()
+                ]
+                serialized_tools = serialize(tool_details)
+                span.set_attribute("gen_ai.tool.definitions", serialized_tools)
+            except Exception:
+                # A failure in telemetry should not crash the agent
+                logger.exception("failed to attach tool metadata to agent span")


This can be moved into tracer.py?

Ok got it , I will move it to traceer.py

poshinchen · 2025-11-04T19:02:29Z

src/strands/agent/agent.py

            messages=messages,
            agent_name=self.name,
            model_id=model_id,
-            tools=self.tool_names,


Let's keep it as customers might still track the tools?

Yes makes sense, I shouldnt have removed this

poshinchen

left some comments

poshinchen · 2025-11-04T20:31:52Z

src/strands/agent/agent.py

-        self.trace_span = self._start_agent_trace_span(messages)
+        self.trace_span = self._start_agent_trace_span(
+            messages,
+            tools_config=self.tool_registry.get_all_tools_config(),


I think you don't need this line. Instead, at line 936, you can do:

span = self.tracer.start_agent_span( messages=messages, agent_name=self.name, model_id=model_id, tools=self.tool_names, system_prompt=self.system_prompt, custom_trace_attributes=self.trace_attributes, tools_config=self.tool_registry.get_all_tools_config(), )

poshinchen · 2025-11-04T20:34:53Z

src/strands/agent/agent.py

        self._append_message(assistant_msg)

-    def _start_agent_trace_span(self, messages: Messages) -> trace_api.Span:
+    def _start_agent_trace_span(self, messages: Messages, tools_config: Optional[dict] = None) -> trace_api.Span:


Same from above, then this doesn't need to be changed.

poshinchen · 2025-11-04T20:35:08Z

src/strands/agent/agent.py

+        if tools_config:
+            self.tracer.add_tool_definitions_to_span(span, tools_config)
+
+        return span


these are not needed too

poshinchen · 2025-11-04T20:36:45Z

src/strands/telemetry/tracer.py


        self._end_span(span, attributes, error)

+    def add_tool_definitions_to_span(self, span: Span, tools_config: dict) -> None:


make this as _construct_tool_definitions, then you can call it in start_agent_span with:

if self.include_tool_definitions: tool_definitions= self._construct_tool_definitions(tools_config) attributes["gen_ai.agent.tools"] = serialize(tool_definitions)

poshinchen · 2025-11-04T20:44:40Z

src/strands/telemetry/tracer.py

+                span.set_attribute("gen_ai.tool.definitions", serialized_tools)
+            except Exception:
+                # A failure in telemetry should not crash the agent
+                logger.exception("failed to attach tool metadata to agent span")


I prefer it to be warning instead of exception.

ok fixed it. thanks.

poshinchen

Left the comments. There's conflict too: This branch has conflicts that must be resolved: tests/strands/agent/test_agent.py

In addition, can you test it locally to see whether set_attribute works without serialization?

Otherwise I can test it on my end

Ratish1 · 2025-11-04T21:56:42Z

Left the comments. There's conflict too: This branch has conflicts that must be resolved: tests/strands/agent/test_agent.py

I fixed the resolve conflicts and made the refactoring changes according to your feedback. Thanks for your help.

In addition, can you test it locally to see whether set_attribute works without serialization?

Otherwise I can test it on my end

For serialization: I initially removed the serialize() call. While the runtime tests passed, it caused a mypy error because the project's AttributeValue type hint does not allow complex objects (list[dict]). To satisfy the static type checker, I had to re-add the serialize() call. So, it seems serialization is required to pass the project's type checks.

github-actions bot added the size/s label Oct 30, 2025

Ratish1 requested a deployment to manual-approval October 30, 2025 19:56 — with GitHub Actions Waiting

Ratish1 changed the title ~~feat(agent): Add opt-in flag to include tool specs in trace~~ feat(agent): Add opt-in flag to include tool specs in traces for evaluation Oct 30, 2025

dbschmigelski requested a review from poshinchen October 31, 2025 13:51

poshinchen reviewed Nov 3, 2025

View reviewed changes

Ratish1 closed this Nov 4, 2025

Ratish1 force-pushed the tool-details branch from 5d4054d to 417ebea Compare November 4, 2025 08:23

github-actions bot added size/xs and removed size/s labels Nov 4, 2025

Ratish1 requested a deployment to manual-approval November 4, 2025 08:23 — with GitHub Actions Waiting

Ratish1 reopened this Nov 4, 2025

github-actions bot added size/s and removed size/xs labels Nov 4, 2025

Ratish1 requested a deployment to manual-approval November 4, 2025 10:18 — with GitHub Actions Waiting

Ratish1 requested a review from poshinchen November 4, 2025 10:20

Ratish1 force-pushed the tool-details branch from 6c00982 to 33d3de5 Compare November 4, 2025 10:26

github-actions bot added size/s and removed size/s labels Nov 4, 2025

Ratish1 requested a deployment to manual-approval November 4, 2025 10:26 — with GitHub Actions Waiting

poshinchen requested changes Nov 4, 2025

View reviewed changes

Ratish1 force-pushed the tool-details branch from 33d3de5 to 1e70994 Compare November 4, 2025 18:45

Ratish1 requested a review from poshinchen November 4, 2025 18:45

github-actions bot added size/m and removed size/s labels Nov 4, 2025

Ratish1 requested a deployment to manual-approval November 4, 2025 18:45 — with GitHub Actions Waiting

poshinchen reviewed Nov 4, 2025

View reviewed changes

poshinchen requested changes Nov 4, 2025

View reviewed changes

Ratish1 force-pushed the tool-details branch from 1e70994 to 5d1d3d2 Compare November 4, 2025 19:56

github-actions bot added size/m and removed size/m labels Nov 4, 2025

Ratish1 requested a deployment to manual-approval November 4, 2025 19:56 — with GitHub Actions Waiting

Ratish1 requested a review from poshinchen November 4, 2025 19:57

poshinchen reviewed Nov 4, 2025

View reviewed changes

poshinchen requested changes Nov 4, 2025

View reviewed changes

Ratish1 force-pushed the tool-details branch from 5d1d3d2 to 87d6ccb Compare November 4, 2025 21:04

github-actions bot removed the size/m label Nov 4, 2025

Ratish1 requested a deployment to manual-approval November 4, 2025 21:04 — with GitHub Actions Waiting

github-actions bot added the size/m label Nov 4, 2025

feat(agent): Add opt-in flag to include tool specs

768a791

Ratish1 force-pushed the tool-details branch from 87d6ccb to 768a791 Compare November 4, 2025 21:56

github-actions bot added size/m and removed size/m labels Nov 4, 2025

Ratish1 requested a deployment to manual-approval November 4, 2025 21:56 — with GitHub Actions Waiting

Ratish1 requested a review from poshinchen November 4, 2025 21:57


		self._end_span(span, attributes, error)

		def add_tool_definitions_to_span(self, span: Span, tools_config: dict) -> None:

feat(agent): Add opt-in flag to include tool specs in traces for evaluation #1113

Are you sure you want to change the base?

feat(agent): Add opt-in flag to include tool specs in traces for evaluation #1113

Conversation

Ratish1 commented Oct 30, 2025

Description

Key Changes

Related Issues

Documentation PR

Type of Change

Testing

Checklist

Uh oh!

codecov bot commented Oct 31, 2025

Codecov Report

Uh oh!

poshinchen left a comment

Choose a reason for hiding this comment

Uh oh!

Ratish1 commented Nov 4, 2025

Uh oh!

Ratish1 commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

poshinchen left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ratish1 commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

poshinchen Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

poshinchen left a comment

Choose a reason for hiding this comment

Uh oh!

poshinchen Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

poshinchen Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

poshinchen left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ratish1 commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Ratish1 commented Nov 4, 2025 •

edited

Loading

poshinchen left a comment •

edited

Loading

Ratish1 commented Nov 4, 2025 •

edited

Loading

poshinchen Nov 4, 2025 •

edited

Loading

poshinchen Nov 4, 2025 •

edited

Loading

poshinchen Nov 4, 2025 •

edited

Loading

poshinchen left a comment •

edited

Loading