dev/ folder restructure and integration test fixes#465
Merged
Conversation
…into users/robrandao/integration-tests
Contributor
There was a problem hiding this comment.
Pull request overview
This PR substantially restructures the dev/ area by removing the old dev/testing layout (including prior hosting_dialogs choice-related tests) and introducing a new dev/microsoft-agents-testing package plus a new dev/integration integration-test harness and supporting utilities.
Changes:
- Removed legacy
dev/testingdocs/scripts and deleted multipledev/hosting_dialogsunit tests (including choice dialog coverage). - Added a new
dev/microsoft-agents-testingPython package (pytest plugin, CLI, scenarios, formatting, transport, presets) and accompanying tests/docs/samples. - Added a new
dev/integrationtest project with pytest configuration, scenarios, and telemetry/streaming tests; adjusted import paths in integration tests to use relative imports.
Reviewed changes
Copilot reviewed 44 out of 213 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| dev/testing/README.md | Removed legacy testing folder README. |
| dev/testing/python-sdk-tests/run_tests.ps1 | Removed legacy PowerShell test runner. |
| dev/testing/python-sdk-tests/README.md | Removed legacy python-sdk-tests README. |
| dev/testing/python-sdk-tests/tests/init.py | Package marker for removed test suite. |
| dev/testing/python-sdk-tests/tests/activity_handler/init.py | Package marker for removed test suite. |
| dev/testing/python-sdk-tests/tests/activity_handler/dialogs/init.py | Package marker for removed test suite. |
| dev/testing/python-sdk-tests/tests/activity_handler/dialogs/sample/init.py | Package marker for removed test suite. |
| dev/testing/python-sdk-tests/tests/integration/init.py | Package marker for removed test suite. |
| dev/testing/cross-sdk-tests/Program.cs | Removed placeholder content. |
| dev/microsoft-agents-testing/pytest.ini | Added pytest configuration for the new testing package. |
| dev/microsoft-agents-testing/pyproject.toml | Added packaging metadata/dependencies for microsoft-agents-testing. |
| dev/microsoft-agents-testing/payload.json | Added sample activity payload JSON. |
| dev/microsoft-agents-testing/env.TEMPLATE | Added env template for auth/logging configuration. |
| dev/microsoft-agents-testing/microsoft_agents/testing/init.py | Added top-level exports for testing framework components. |
| dev/microsoft-agents-testing/microsoft_agents/testing/constants.py | Added constants (default local endpoint). |
| dev/microsoft-agents-testing/microsoft_agents/testing/source_scenario.py | Added script-based scenario runner for launching agents via PowerShell. |
| dev/microsoft-agents-testing/microsoft_agents/testing/pytest_plugin.py | Added pytest plugin and fixtures driven by @pytest.mark.agent_test. |
| dev/microsoft-agents-testing/microsoft_agents/testing/utils/poll.py | Added async polling utility. |
| dev/microsoft-agents-testing/microsoft_agents/testing/utils/send.py | Added simple send utilities for quick agent interactions. |
| dev/microsoft-agents-testing/microsoft_agents/testing/utils/init.py | Exported utility functions from utils. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/init.py | Added core exports for client/scenario/transport/fluent layers. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/config.py | Added ScenarioConfig / ClientConfig dataclasses. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/scenario.py | Added abstract Scenario and ClientFactory protocol. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/external_scenario.py | Added ExternalScenario for testing externally-hosted agents. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/_aiohttp_client_factory.py | Added internal factory for aiohttp-based clients and token/header handling. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/utils.py | Added helpers for extracting activities and generating tokens from config. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/init.py | Added transport exports (sender, callback server, transcript). |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/sender.py | Added abstract Sender interface. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/callback_server.py | Added abstract callback server interface. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/aiohttp_sender.py | Added aiohttp sender implementation for posting activities. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/aiohttp_callback_server.py | Added aiohttp callback server for collecting agent responses. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/transcript/init.py | Added transcript package exports. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/transcript/transcript.py | Added hierarchical transcript recording with parent/child propagation. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/init.py | Added fluent API exports (Expect/Select/backend utilities). |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/select.py | Added chainable selection/filtering API. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/utils.py | Added model normalization/flattening helpers. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/init.py | Added fluent backend exports. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/model_predicate.py | Added predicate evaluation over model collections. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/quantifier.py | Added quantifiers (all/any/none/one/n). |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/init.py | Added exports for fluent backend types. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/readonly.py | Added readonly mixin. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/unset.py | Added Unset sentinel singleton. |
| dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/safe_object.py | Added SafeObject wrapper for safe nested access. |
| dev/microsoft-agents-testing/microsoft_agents/testing/formatting/init.py | Added formatting exports. |
| dev/microsoft-agents-testing/microsoft_agents/testing/formatting/utils.py | Added timestamp/sort helpers for transcript formatting. |
| dev/microsoft-agents-testing/microsoft_agents/testing/formatting/transcript_formatter.py | Added formatter protocol/base class. |
| dev/microsoft-agents-testing/microsoft_agents/testing/formatting/json_transcript_formatter.py | Added JSON exchange transcript formatter. |
| dev/microsoft-agents-testing/microsoft_agents/testing/formatting/activity_transcript_formatter.py | Added flat activity JSON transcript formatter. |
| dev/microsoft-agents-testing/microsoft_agents/testing/formatting/conversation_transcript_formatter.py | Added human-readable conversation transcript formatter. |
| dev/microsoft-agents-testing/microsoft_agents/testing/formatting/print.py | Added convenience printing functions for transcripts. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/init.py | Added CLI package entrypoint export. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/main.py | Added CLI root, scenario registration, and command loading. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/scenarios/init.py | Added built-in scenario registry for CLI. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/scenarios/basic_scenario.py | Added basic echo scenario for CLI use. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/scenarios/auth_scenario.py | Added auth-testing scenario for CLI use. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/init.py | Added CLI core exports. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/cli_config.py | Added .env configuration loader for CLI. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/output.py | Added output styling/utilities for CLI. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/utils.py | Added scenario resolution utilities for CLI. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/init.py | Added CLI command registry. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/init.py | Added CLI preset scaffolding command. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/environment/init.py | Added environment command group package. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/environment/env_group.py | Added agt env group. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/environment/_show.py | Added agt env show. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/environment/_help.py | Added agt env help. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/scenario/init.py | Added scenario command group package and module imports. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/scenario/scenario_group.py | Added agt scenario group. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/scenario/_chat.py | Added interactive chat command. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/scenario/_list.py | Added scenario listing command. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/scenario/_post.py | Added single-post command with transcript output. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/scenario/_run.py | Added “run scenario as server” command. |
| dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/scenario/_utils.py | Added CLI helpers for loading activities and printing histograms. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/basic/my_agent/pyproject.toml | Added “basic” preset agent project metadata. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/basic/my_agent/env.TEMPLATE | Added preset env template. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/basic/my_agent/src/main.py | Added minimal sample agent implementation for preset. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/basic/my_agent/src/start_server.py | Added aiohttp server bootstrap for preset agent. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/basic/e2e-tests/pyproject.toml | Added preset e2e test project metadata. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/basic/e2e-tests/pytest.ini | Added preset pytest config. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/basic/e2e-tests/env.TEMPLATE | Added preset env template. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/basic/e2e-tests/config.json | Added preset scenario registry config. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/basic/e2e-tests/tests/init.py | Added preset tests package init that loads registry config. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/basic/e2e-tests/tests/test_my_agent.py | Added preset example e2e test. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/localhost/e2e-tests/pyproject.toml | Added localhost e2e project metadata. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/localhost/e2e-tests/pytest.ini | Added localhost e2e pytest config. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/localhost/e2e-tests/env.TEMPLATE | Added localhost env template. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/localhost/e2e-tests/config.json | Added localhost scenario config. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/localhost/e2e-tests/tests/init.py | Added localhost tests init that loads registry config. |
| dev/microsoft-agents-testing/microsoft_agents/testing/presets/localhost/e2e-tests/tests/test_my_agent.py | Added localhost example e2e test. |
| dev/microsoft-agents-testing/tests/init.py | Added tests package marker. |
| dev/microsoft-agents-testing/tests/utils/init.py | Added tests utils package marker. |
| dev/microsoft-agents-testing/tests/utils/test_poll.py | Added unit tests for poll() utility. |
| dev/microsoft-agents-testing/tests/test_scenario_registry_plugin.py | Added pytest-plugin/registry integration tests. |
| dev/microsoft-agents-testing/tests/test_examples.py | Placeholder or package marker for examples tests. |
| dev/microsoft-agents-testing/tests/manual.py | Added manual script (non-test) for experimenting with scenarios/clients. |
| dev/microsoft-agents-testing/tests/core/init.py | Added tests core package marker. |
| dev/microsoft-agents-testing/tests/core/fluent/init.py | Added tests fluent package marker. |
| dev/microsoft-agents-testing/tests/core/fluent/backend/init.py | Added tests backend package marker. |
| dev/microsoft-agents-testing/tests/core/fluent/backend/types/init.py | Added tests backend types package marker. |
| dev/microsoft-agents-testing/tests/core/fluent/backend/types/test_unset.py | Added tests for Unset sentinel. |
| dev/microsoft-agents-testing/tests/core/fluent/backend/types/test_readonly.py | Added tests for Readonly mixin. |
| dev/microsoft-agents-testing/tests/core/transport/init.py | Added transport tests package marker (license header). |
| dev/microsoft-agents-testing/tests/core/transport/transcript/init.py | Added transcript tests package marker (license header). |
| dev/microsoft-agents-testing/docs/SAMPLES.md | Added sample documentation for the new testing framework. |
| dev/microsoft-agents-testing/docs/samples/init.py | Added docs samples package marker and overview docstring. |
| dev/microsoft-agents-testing/docs/samples/quickstart.py | Added quickstart sample script. |
| dev/microsoft-agents-testing/docs/samples/interactive.py | Added interactive sample script. |
| dev/microsoft-agents-testing/docs/samples/scenario_registry_demo.py | Added scenario registry demo script. |
| dev/microsoft-agents-testing/docs/samples/pytest_plugin_usage.py | Added pytest plugin usage sample. |
| dev/microsoft-agents-testing/docs/samples/test_motivation_assertions.py | Added “expected failure” demo tests for assertion output formatting. |
| dev/microsoft-agents-testing/docs/samples/multi_client.py | Added multi-client patterns sample. |
| dev/microsoft-agents-testing/docs/samples/transcript_formatting.py | Added transcript formatting sample (currently references additional APIs). |
| dev/integration/init.py | Added integration project package marker. |
| dev/integration/README.md | Added instructions for running integration tests. |
| dev/integration/pytest.ini | Added pytest configuration for integration tests. |
| dev/integration/pyproject.toml | Added integration test project metadata/dependencies. |
| dev/integration/env.TEMPLATE | Added env template for integration tests. |
| dev/integration/tests/init.py | Added integration tests package marker. |
| dev/integration/tests/test_telemetry.py | Updated telemetry tests to use relative imports and fixture module. |
| dev/integration/tests/test_quickstart.py | Updated scenario import path and relaxed assertions. |
| dev/integration/tests/test_expect_replies.py | Updated scenario import path to relative import. |
| dev/integration/tests/test_streaming_response.py | Added streaming-response integration tests. |
| dev/integration/tests/scenarios/init.py | Added local scenario registry/loader for integration tests. |
| dev/integration/tests/scenarios/quickstart.py | Added quickstart scenario init for integration tests. |
| dev/integration/tests/utils/telemetry_utils.py | Added helper functions for inspecting metrics. |
| dev/integration/tests/utils/telemetry_fixtures.py | Added telemetry fixtures with delta metric reader wrapper. |
| dev/integration/tests/activity_handler/init.py | Added package marker for activity_handler integration tests. |
| dev/integration/tests/activity_handler/dialogs/init.py | Added dialogs package marker. |
| dev/integration/tests/activity_handler/dialogs/sample/init.py | Added dialogs sample package marker. |
| dev/integration/tests/activity_handler/dialogs/sample/dialog_agent.py | Added ActivityHandler-based dialog agent used by dialog tests. |
| dev/integration/tests/activity_handler/dialogs/sample/booking_dialog.py | Added booking dialog sample used by dialog tests. |
| dev/integration/tests/activity_handler/dialogs/sample/user_profile.py | Added user profile dataclass model for dialog tests. |
| dev/integration/tests/activity_handler/dialogs/scenario.py | Updated dialog scenario imports to relative imports. |
| dev/integration/tests/activity_handler/dialogs/test_booking_dialog.py | Updated imports to use local sample modules. |
| dev/integration/tests/activity_handler/dialogs/test_user_profile_dialog.py | Updated import path for dialog scenario factory. |
| dev/hosting_dialogs/test_dialog.py | Deleted legacy dialogs unit test. |
| dev/hosting_dialogs/test_dialog_set.py | Deleted legacy dialogs unit test. |
| dev/hosting_dialogs/test_dialog_extensions.py | Deleted legacy dialogs unit test. |
| dev/hosting_dialogs/test_waterfall_dialog.py | Deleted legacy dialogs unit test. |
| dev/hosting_dialogs/test_waterfall_step_context.py | Deleted legacy dialogs unit test. |
| dev/hosting_dialogs/test_replace_dialog.py | Deleted legacy dialogs unit test. |
| dev/hosting_dialogs/test_prompt_validator_context.py | Deleted legacy dialogs unit test. |
| dev/hosting_dialogs/test_prompt_culture_models.py | Deleted legacy dialogs unit test. |
| dev/hosting_dialogs/test_date_time_prompt.py | Deleted legacy dialogs unit test. |
| dev/hosting_dialogs/memory/test_at_path_resolver.py | Deleted legacy memory path resolver test. |
| dev/hosting_dialogs/memory/scopes/test_settings.py | Deleted legacy settings test stub. |
| dev/hosting_dialogs/choices/test_channel.py | Deleted choice channel tests. |
| dev/hosting_dialogs/choices/test_choice.py | Deleted Choice model tests. |
| dev/hosting_dialogs/choices/test_choice_factory_options.py | Deleted ChoiceFactoryOptions tests. |
| dev/hosting_dialogs/choices/test_choice_tokenizer.py | Deleted tokenizer tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
kylerohn-msft
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR reorganizes the
devdirectory which includes an integration testing suite and themicrosoft-agents-testingpackage