Port Agent365 integration tests using real distro pipeline#146
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Ports Agent365 integration coverage into this repository so tests exercise the real Microsoft OpenTelemetry distro pipeline via use_microsoft_opentelemetry().
Changes:
- Adds shared A365 integration fixtures and span-capturing exporter.
- Adds Azure OpenAI-backed integration tests for AgentFramework, OpenAI Agents, and LangChain.
- Updates AgentFramework sensitive-data handling and adds a LangChain GenAI span utility shim.
Reviewed changes
Copilot reviewed 17 out of 19 changed files in this pull request and generated 14 comments.
Show a summary per file
| File | Description |
|---|---|
src/microsoft/opentelemetry/_agent_framework/_trace_instrumentor.py |
Adjusts AgentFramework enablement to preserve existing observability settings. |
src/microsoft/opentelemetry/_genai/_langchain/_span_utils.py |
Adds local GenAI span utility shim for LangChain tracing. |
src/microsoft/opentelemetry/_genai/_langchain/_tracer.py |
Switches LangChain tracer to use the local shim. |
src/microsoft/opentelemetry/_genai/_langchain/_tracer_instrumentor.py |
Adds debug logging when LangChain is unavailable. |
src/microsoft/opentelemetry/_genai/_langchain/_utils.py |
Adds prompt fallback extraction for LangChain LLM input messages. |
tests/agent_framework/test_trace_instrumentor.py |
Updates and extends AgentFramework instrumentor tests. |
tests/a365/integration/conftest.py |
Adds shared A365 integration fixtures and capturing exporter. |
tests/a365/integration/__init__.py |
Package marker for integration tests. |
tests/a365/integration/agentframework/__init__.py |
Adds package docstring. |
tests/a365/integration/agentframework/test_agentframework_trace_processor.py |
Adds AgentFramework trace processor integration tests. |
tests/a365/integration/agentframework/test_message_format.py |
Adds AgentFramework message-format integration tests. |
tests/a365/integration/agentframework/test_observability_pipeline.py |
Adds AgentFramework end-to-end pipeline integration tests. |
tests/a365/integration/langchain/__init__.py |
Adds license header. |
tests/a365/integration/langchain/conftest.py |
Adds LangChain-specific OpenAI v2 uninstrumentation workaround. |
tests/a365/integration/langchain/test_message_format.py |
Adds LangChain message-format integration tests. |
tests/a365/integration/langchain/test_observability_pipeline.py |
Adds LangChain end-to-end pipeline integration tests. |
tests/a365/integration/openai/__init__.py |
Adds OpenAI integration test package marker. |
tests/a365/integration/openai/test_message_format.py |
Adds OpenAI Agents message-format integration tests. |
tests/a365/integration/openai/test_openai_trace_processor.py |
Adds OpenAI Agents trace processor integration tests. |
Comments suppressed due to low confidence (2)
tests/a365/integration/openai/test_openai_trace_processor.py:334
- The helper records how many LLM, agent, and tool spans were found but only asserts that some span exists. A regression that drops all tool or LLM spans would still pass this tool-call integration test.
# Ensure we found the expected span types
assert len(distro_exporter.spans) > 0, "No spans were captured"
print(f"✓ Captured {len(distro_exporter.spans)} spans total")
print(
f"✓ Found {llm_spans_found} LLM spans, {agent_spans_found} agent spans, and {tool_spans_found} tool spans"
)
tests/a365/integration/agentframework/test_agentframework_trace_processor.py:225
- The helper records how many LLM, agent, and tool spans were found but only asserts that some span exists. A regression that drops all tool or LLM spans would still pass this tool-call integration test.
# Ensure we found the expected span types
assert len(distro_exporter.spans) > 0, "No spans were captured"
print(f"✓ Captured {len(distro_exporter.spans)} spans total")
print(
f"✓ Found {llm_spans_found} LLM spans, {agent_spans_found} agent spans, and {tool_spans_found} tool spans"
)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add integration tests for AgentFramework, OpenAI, and LangChain that exercise the full distro pipeline via use_microsoft_opentelemetry(). Tests cover: - Trace processing (span attributes, tool calls, InvokeAgentScope) - Message format mapping (A365 versioned format verification) - Observability pipeline (span hierarchy, trace IDs, operation names) Also includes: - LangChain span_utils shim for missing opentelemetry.util.genai.span_utils - LangChain _utils.py fix for prompts key handling - LangChain conftest to work around openai-v2 instrumentor KeyError on Azure deployment-based routing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
af915ee to
799882a
Compare
Move _apply_error_attributes, _apply_llm_finish_attributes, and _maybe_emit_llm_event from the _span_utils.py compatibility shim directly into _tracer.py. Rename _apply_error_attributes to _set_error for clarity. Delete the now-unused _span_utils.py module. No behavioral changes — the replacement functions produce identical span attributes and events. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Disable redefined-outer-name in conftest (standard pytest pattern) - Disable too-many-statements on long test method - Remove unnecessary f-string prefix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Use Sequence[ReadableSpan] instead of list[ReadableSpan] in SpanCapturingExporter.export() - Cast attribute values to str in _get_span_attr helper - Use intermediate variable for Optional span_map.get() results - Cast attrs.get() to str for operation name comparison Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rads-1996
reviewed
May 15, 2026
Add tox 'integration' env that installs langchain and agent-framework extras, then runs tests/a365/integration/ with Azure OpenAI env vars passed through from GitHub Actions secrets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bypass tox passenv to ensure env vars reach pytest directly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
alexlu4250
approved these changes
May 15, 2026
…genai 0.3b0 Revert inline helpers back to importing from opentelemetry.util.genai.span_utils. Pin opentelemetry-util-genai==0.3b0 which provides span_utils module. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ntests # Conflicts: # pyproject.toml
fpfp100
approved these changes
May 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Port integration tests from the Agent365SDK repo into tests/a365/integration/ exercising the full distro pipeline via use_microsoft_opentelemetry().
Changes
Integration Tests (15 tests across 3 frameworks)
AgentFramework (tests/a365/integration/agentframework/)
OpenAI Agents (tests/a365/integration/openai/)
LangChain (tests/a365/integration/langchain/)
Shared Test Infrastructure
Bug Fixes
Workaround
Testing