Data-driven integration tests

[rodrigobr-msft]

Microsoft 365 Agents SDK for Python - Testing Framework

A comprehensive testing framework designed specifically for Microsoft 365 Agents SDK, providing essential utilities and abstractions to streamline integration testing, authentication, data-driven testing, and end-to-end agent validation

Why This Package Exists

Building and testing conversational agents presents unique challenges that standard testing frameworks don't address. This package eliminates these pain points by providing powerful abstractions specifically designed for agent testing scenarios, including support for data-driven testing with YAML/JSON configurations.

Key Benefits:

• Write tests once in YAML/JSON, run them everywhere
• Reduce boilerplate code with pre-built fixtures and clients
• Validate complex conversation flows with declarative assertions
• Maintain test suites that are easy to read and maintain
• Integrate seamlessly with pytest and CI/CD pipelines

Key Features

🔐 Authentication Utilities

Generate OAuth2 access tokens for testing secured agents with Microsoft Authentication Library (MSAL) integration.

Features:

• Client credentials flow support
• Environment variable configuration
• SDK config integration

Example:

from microsoft_agents.testing import generate_token, generate_token_from_config

# Generate token directly
token = generate_token(
    app_id="your-app-id",
    app_secret="your-secret",
    tenant_id="your-tenant"
)

# Or from SDK config
token = generate_token_from_config(sdk_config)

🧪 Integration Test Framework

Pre-built pytest fixtures and abstractions for agent integration testing.

Features:

• Pytest fixture integration
• Environment abstraction for different hosting configurations
• Sample management for test organization
• Application lifecycle management
• Automatic setup and teardown

Example:

from microsoft_agents.testing import Integration, AiohttpEnvironment, Sample

class MyAgentSample(Sample):
    async def init_app(self):
        self.app = create_my_agent_app(self.env)
    
    @classmethod
    async def get_config(cls):
        return {"service_url": "http://localhost:3978"}

class MyAgentTests(Integration):
    _sample_cls = MyAgentSample
    _environment_cls = AiohttpEnvironment
    
    @pytest.mark.asyncio
    async def test_conversation_flow(self, agent_client, sample):
        # Client and sample are automatically set up via fixtures
        response = await agent_client.send_activity("Hello")
        assert response is not None

🤖 Agent Communication Clients

High-level clients for sending and receiving activities from agents under test.

Features:

• Simple text message sending
• Full Activity object support
• Automatic token management
• Support for expectReplies delivery mode
• Response collection and management

AgentClient Example:

from microsoft_agents.testing import AgentClient
from microsoft_agents.activity import Activity, ActivityTypes

client = AgentClient(
    agent_url="http://localhost:3978",
    cid="conversation-id",
    client_id="your-client-id",
    tenant_id="your-tenant-id",
    client_secret="your-secret"
)

# Send simple text message
response = await client.send_activity("What's the weather?")

# Send full Activity object
activity = Activity(type=ActivityTypes.message, text="Hello")
response = await client.send_activity(activity)

# Send with expectReplies delivery mode
replies = await client.send_expect_replies("What can you do?")
for reply in replies:
    print(reply.text)

ResponseClient Example:

from microsoft_agents.testing import ResponseClient

# Create response client to collect agent responses
async with ResponseClient(host="localhost", port=9873) as response_client:
    # ... send activities with agent_client ...
    
    # Collect all responses
    responses = await response_client.pop()
    assert len(responses) > 0

📋 Data-Driven Testing

Write test scenarios in YAML or JSON files and execute them automatically. Perfect for creating reusable test suites, regression tests, and living documentation.

Features:

• Declarative test definition in YAML/JSON
• Parent/child file inheritance for shared defaults
• Multiple step types (input, assertion, sleep, breakpoint)
• Flexible assertions with selectors and quantifiers
• Automatic test discovery and generation
• Field-level assertion operators

Using the @DDT Decorator

The @DDT (data-driven tests) decorator automatically loads test files and generates pytest test methods:

from microsoft_agents.testing import Integration, AiohttpEnvironment, ddt

@ddt("tests/my_agent/test_cases", recursive=True)
class TestMyAgent(Integration):
    _sample_cls = MyAgentSample
    _environment_cls = AiohttpEnvironment
    _agent_url = "http://localhost:3978"
    _cid = "test-conversation"

This will:

1. Load all .yaml and .json files from tests/my_agent/test_cases (and subdirectories if recursive=True)
2. Create a pytest test method for each file (e.g., test_data_driven__greeting_test)
3. Execute the test flow defined in each file

✅ Advanced Assertions Framework

Powerful assertion system for validating agent responses with flexible matching criteria.

ModelAssertion

Create assertions for validating lists of activities:

from microsoft_agents.testing import ModelAssertion, Selector, AssertionQuantifier

# Create an assertion
assertion = ModelAssertion(
    assertion={"type": "message", "text": "Hello"},
    selector=Selector(selector={"type": "message"}),
    quantifier=AssertionQuantifier.ALL
)

# Test activities
activities = [...]  # List of Activity objects
passes, error = assertion.check(activities)

# Or use as callable (raises AssertionError on failure)
assertion(activities)

From configuration dictionary:

config = {
    "activity": {"type": "message", "text": "Hello"},
    "selector": {"activity": {"type": "message"}},
    "quantifier": "all"
}
assertion = ModelAssertion.from_config(config)

Selectors

Filter activities before validation:

from microsoft_agents.testing import Selector

# Select all message activities
selector = Selector(selector={"type": "message"})
messages = selector(activities)

# Select the first message activity
selector = Selector(selector={"type": "message"}, index=0)
first_message = selector.select_first(activities)

# Select the last message activity
selector = Selector(selector={"type": "message"}, index=-1)
last_message = selector(activities)[0]

# Select by multiple fields
selector = Selector(selector={
    "type": "message",
    "locale": "en-US",
    "channelId": "directline"
})

From configuration:

config = {
    "activity": {"type": "message"},
    "index": -1
}
selector = Selector.from_config(config)

Quantifiers

Control how many activities must match the assertion:

from microsoft_agents.testing import AssertionQuantifier

# ALL: Every selected activity must match (default)
quantifier = AssertionQuantifier.ALL

# ANY: At least one activity must match
quantifier = AssertionQuantifier.ANY

# ONE: Exactly one activity must match
quantifier = AssertionQuantifier.ONE

# NONE: No activities should match
quantifier = AssertionQuantifier.NONE

# From string
quantifier = AssertionQuantifier.from_config("all")

Field Assertions

Test individual fields with operators:

from microsoft_agents.testing import check_field, FieldAssertionType

# String contains
result = check_field("Hello world", ["CONTAINS", "world"])  # True

# Regex match
result = check_field("ID-12345", ["RE_MATCH", r"ID-\d+"])  # True

# Value in list
result = check_field(5, ["IN", [1, 3, 5, 7]])  # True

# Value not in list
result = check_field(2, ["NOT_IN", [1, 3, 5, 7]])  # True

# Numeric comparisons
result = check_field(10, ["GREATER_THAN", 5])  # True
result = check_field(3, ["LESS_THAN", 10])  # True

# String doesn't contain
result = check_field("Hello", ["NOT_CONTAINS", "world"])  # True

# Exact equality
result = check_field("test", "test")  # True
result = check_field(42, ["EQUALS", 42])  # True

# Inequality
result = check_field("foo", ["NOT_EQUALS", "bar"])  # True

Verbose checking with error details:

from microsoft_agents.testing import check_field_verbose

passes, error_data = check_field_verbose("Hello", ["CONTAINS", "world"])
if not passes:
    print(f"Field: {error_data.field_path}")
    print(f"Actual: {error_data.actual_value}")
    print(f"Expected: {error_data.assertion}")
    print(f"Type: {error_data.assertion_type}")

Activity Assertions

Check entire activities:

from microsoft_agents.testing import check_model, assert_model

activity = Activity(type="message", text="Hello", locale="en-US")

# Check without raising exception
assertion = {"type": "message", "text": ["CONTAINS", "Hello"]}
result = check_activity(activity, assertion)  # True

# Check with detailed error information
passes, error_data = check_activity_verbose(activity, assertion)

# Assert with exception on failure
assert_model(activity, assertion)  # Raises AssertionError if fails

Nested field checking:

assertion = {
    "type": "message",
    "channelData": {
        "user": {
            "id": ["RE_MATCH", r"user-\d+"]
        }
    }
}
assert_model(activity, assertion)

[Copilot]

Pull Request Overview

Copilot reviewed 63 out of 76 changed files in this pull request and generated 10 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 134 out of 153 changed files in this pull request and generated 12 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 134 out of 153 changed files in this pull request and generated 7 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 129 out of 146 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.