Skip to content

Port Agent365 integration tests using real distro pipeline#146

Merged
nikhilNava merged 11 commits into
mainfrom
nikhil/portIntegrationtests
May 15, 2026
Merged

Port Agent365 integration tests using real distro pipeline#146
nikhilNava merged 11 commits into
mainfrom
nikhil/portIntegrationtests

Conversation

@nikhilNava
Copy link
Copy Markdown
Collaborator

Summary

Port integration tests from the Agent365SDK repo into tests/a365/integration/ exercising the full distro pipeline via use_microsoft_opentelemetry().

Changes

Integration Tests (15 tests across 3 frameworks)

AgentFramework (tests/a365/integration/agentframework/)

  • test_agentframework_trace_processor.py - basic trace + tool calls
  • test_message_format.py - A365 versioned message format verification
  • test_observability_pipeline.py - full pipeline with InvokeAgentScope

OpenAI Agents (tests/a365/integration/openai/)

  • test_openai_trace_processor.py - basic trace, tool calls, invoke_agent span attributes
  • test_message_format.py - simple chat + tool call message format

LangChain (tests/a365/integration/langchain/)

  • test_message_format.py - simple chat + tool call message format
  • test_observability_pipeline.py - full pipeline with InvokeAgentScope

Shared Test Infrastructure

  • tests/a365/integration/conftest.py - session-scoped distro_exporter fixture using use_microsoft_opentelemetry and SpanCapturingExporter

Bug Fixes

  • AF enable_sensitive_data preservation - _trace_instrumentor.py no longer overwrites enable_sensitive_data when AF SDK already instrumented
  • LangChain span_utils shim - local _span_utils.py for missing opentelemetry.util.genai.span_utils

Workaround

  • LangChain conftest uninstruments openai-v2 instrumentor to avoid KeyError on Azure deployment-based routing

Testing

  • All 15 integration tests pass against real Azure OpenAI endpoints
  • All 950 existing unit tests pass (no regressions)

Copilot AI review requested due to automatic review settings May 14, 2026 23:22
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Ports Agent365 integration coverage into this repository so tests exercise the real Microsoft OpenTelemetry distro pipeline via use_microsoft_opentelemetry().

Changes:

  • Adds shared A365 integration fixtures and span-capturing exporter.
  • Adds Azure OpenAI-backed integration tests for AgentFramework, OpenAI Agents, and LangChain.
  • Updates AgentFramework sensitive-data handling and adds a LangChain GenAI span utility shim.

Reviewed changes

Copilot reviewed 17 out of 19 changed files in this pull request and generated 14 comments.

Show a summary per file
File Description
src/microsoft/opentelemetry/_agent_framework/_trace_instrumentor.py Adjusts AgentFramework enablement to preserve existing observability settings.
src/microsoft/opentelemetry/_genai/_langchain/_span_utils.py Adds local GenAI span utility shim for LangChain tracing.
src/microsoft/opentelemetry/_genai/_langchain/_tracer.py Switches LangChain tracer to use the local shim.
src/microsoft/opentelemetry/_genai/_langchain/_tracer_instrumentor.py Adds debug logging when LangChain is unavailable.
src/microsoft/opentelemetry/_genai/_langchain/_utils.py Adds prompt fallback extraction for LangChain LLM input messages.
tests/agent_framework/test_trace_instrumentor.py Updates and extends AgentFramework instrumentor tests.
tests/a365/integration/conftest.py Adds shared A365 integration fixtures and capturing exporter.
tests/a365/integration/__init__.py Package marker for integration tests.
tests/a365/integration/agentframework/__init__.py Adds package docstring.
tests/a365/integration/agentframework/test_agentframework_trace_processor.py Adds AgentFramework trace processor integration tests.
tests/a365/integration/agentframework/test_message_format.py Adds AgentFramework message-format integration tests.
tests/a365/integration/agentframework/test_observability_pipeline.py Adds AgentFramework end-to-end pipeline integration tests.
tests/a365/integration/langchain/__init__.py Adds license header.
tests/a365/integration/langchain/conftest.py Adds LangChain-specific OpenAI v2 uninstrumentation workaround.
tests/a365/integration/langchain/test_message_format.py Adds LangChain message-format integration tests.
tests/a365/integration/langchain/test_observability_pipeline.py Adds LangChain end-to-end pipeline integration tests.
tests/a365/integration/openai/__init__.py Adds OpenAI integration test package marker.
tests/a365/integration/openai/test_message_format.py Adds OpenAI Agents message-format integration tests.
tests/a365/integration/openai/test_openai_trace_processor.py Adds OpenAI Agents trace processor integration tests.
Comments suppressed due to low confidence (2)

tests/a365/integration/openai/test_openai_trace_processor.py:334

  • The helper records how many LLM, agent, and tool spans were found but only asserts that some span exists. A regression that drops all tool or LLM spans would still pass this tool-call integration test.
        # Ensure we found the expected span types
        assert len(distro_exporter.spans) > 0, "No spans were captured"
        print(f"✓ Captured {len(distro_exporter.spans)} spans total")
        print(
            f"✓ Found {llm_spans_found} LLM spans, {agent_spans_found} agent spans, and {tool_spans_found} tool spans"
        )

tests/a365/integration/agentframework/test_agentframework_trace_processor.py:225

  • The helper records how many LLM, agent, and tool spans were found but only asserts that some span exists. A regression that drops all tool or LLM spans would still pass this tool-call integration test.
        # Ensure we found the expected span types
        assert len(distro_exporter.spans) > 0, "No spans were captured"
        print(f"✓ Captured {len(distro_exporter.spans)} spans total")
        print(
            f"✓ Found {llm_spans_found} LLM spans, {agent_spans_found} agent spans, and {tool_spans_found} tool spans"
        )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/microsoft/opentelemetry/_genai/_langchain/_span_utils.py Outdated
Comment thread tests/a365/integration/langchain/conftest.py
Comment thread tests/a365/integration/openai/test_message_format.py
Comment thread tests/a365/integration/langchain/test_message_format.py
Comment thread tests/a365/integration/conftest.py
Comment thread tests/a365/integration/openai/test_openai_trace_processor.py
Comment thread tests/a365/integration/agentframework/test_message_format.py
Add integration tests for AgentFramework, OpenAI, and LangChain that
exercise the full distro pipeline via use_microsoft_opentelemetry().

Tests cover:
- Trace processing (span attributes, tool calls, InvokeAgentScope)
- Message format mapping (A365 versioned format verification)
- Observability pipeline (span hierarchy, trace IDs, operation names)

Also includes:
- LangChain span_utils shim for missing opentelemetry.util.genai.span_utils
- LangChain _utils.py fix for prompts key handling
- LangChain conftest to work around openai-v2 instrumentor KeyError
  on Azure deployment-based routing

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@nikhilNava nikhilNava force-pushed the nikhil/portIntegrationtests branch from af915ee to 799882a Compare May 15, 2026 16:40
nikhilc-microsoft and others added 4 commits May 15, 2026 11:23
Move _apply_error_attributes, _apply_llm_finish_attributes, and
_maybe_emit_llm_event from the _span_utils.py compatibility shim
directly into _tracer.py. Rename _apply_error_attributes to _set_error
for clarity. Delete the now-unused _span_utils.py module.

No behavioral changes — the replacement functions produce identical span
attributes and events.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Disable redefined-outer-name in conftest (standard pytest pattern)
- Disable too-many-statements on long test method
- Remove unnecessary f-string prefix

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Use Sequence[ReadableSpan] instead of list[ReadableSpan] in SpanCapturingExporter.export()
- Cast attribute values to str in _get_span_attr helper
- Use intermediate variable for Optional span_map.get() results
- Cast attrs.get() to str for operation name comparison

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread src/microsoft/opentelemetry/_genai/_langchain/_tracer.py Outdated
nikhilc-microsoft and others added 4 commits May 15, 2026 12:36
Add tox 'integration' env that installs langchain and agent-framework
extras, then runs tests/a365/integration/ with Azure OpenAI env vars
passed through from GitHub Actions secrets.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bypass tox passenv to ensure env vars reach pytest directly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
nikhilc-microsoft and others added 2 commits May 15, 2026 13:49
…genai 0.3b0

Revert inline helpers back to importing from opentelemetry.util.genai.span_utils.
Pin opentelemetry-util-genai==0.3b0 which provides span_utils module.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Member

@rads-1996 rads-1996 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nikhilNava nikhilNava merged commit a107c4a into main May 15, 2026
9 checks passed
@nikhilNava nikhilNava deleted the nikhil/portIntegrationtests branch May 15, 2026 20:38
@rads-1996 rads-1996 mentioned this pull request May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants