Python: updated integration tests and guidance#4181
Merged
eavanvalkenburg merged 9 commits intomicrosoft:mainfrom Feb 24, 2026
Merged
Python: updated integration tests and guidance#4181eavanvalkenburg merged 9 commits intomicrosoft:mainfrom
eavanvalkenburg merged 9 commits intomicrosoft:mainfrom
Conversation
markwallace-microsoft
added
documentation
Member
Python Test Coverage Report •
Python Unit Test Overview
|
||||||||||||||||||||||||||||||||||||||||
eavanvalkenburg
force-pushed
the
int_tests_update
branch
from
6dd6e9a to
0ea5081
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request refines Python integration test execution by introducing marker-based test selection, parallel job execution in CI, and comprehensive documentation updates. The changes enable better separation between unit and integration tests, maximize CI parallelism through provider-specific jobs with change-based filtering, and provide clear guidance for contributors.
Changes:
- Added
@pytest.mark.integrationmarker to all integration tests across 20+ test files - Restructured CI workflow to split tests into 6 parallel jobs (unit, OpenAI, Azure OpenAI, misc providers, Functions, Azure AI) with path-based change detection
- Updated documentation (DEV_SETUP.md, CODING_STANDARD.md, python-testing skill) with integration test patterns and CI workflow details
Reviewed changes
Copilot reviewed 43 out of 43 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/python-merge-tests.yml | Restructured workflow with path filters and 6 parallel test jobs for unit and provider-specific integration tests |
| python/pyproject.toml | Added integration marker definition; removed mypy from check-packages (still in ci-mypy) |
| python/packages/core/pyproject.toml | Added pytest-xdist parallelization (-n auto --dist worksteal) to test task |
| python/packages/orchestrations/pyproject.toml | Added pytest-xdist parallelization to test task |
| python/packages/anthropic/pyproject.toml | Added pytest-xdist parallelization to test task |
| python/packages/ag-ui/pyproject.toml | Added pytest-xdist parallelization to test task |
| python/packages//tests/.py | Added @pytest.mark.integration and @pytest.mark.flaky markers to 70+ integration tests |
| python/packages/core/agent_framework/observability.py | Removed unused SpanAttributes import (code formatting) |
| python/packages/core/agent_framework/openai/_chat_client.py | Line length formatting improvements |
| python/DEV_SETUP.md | Added integration test marker usage examples and pytest-xdist information |
| python/CODING_STANDARD.md | Added comprehensive integration test marker requirements and CI workflow guidance |
| python/.github/skills/python-testing/SKILL.md | Added integration test patterns, CI workflow details, and provider addition guidance |
TaoChenOSU
reviewed
Feb 23, 2026
TaoChenOSU
reviewed
Feb 23, 2026
dmytrostruk
approved these changes
Feb 24, 2026
Only one --dist mode can be active at a time; the second value silently overrides the first. Keep --dist worksteal (dynamic load balancing) and remove the redundant --dist loadfile from all workflow files and pyproject.toml configs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both python-merge-tests.yml and python-integration-tests.yml share the same parallel job structure. Added sync reminders in workflow file comments, the python-testing SKILL.md, and CODING_STANDARD.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Integration test gating now uses two mechanisms: - `@pytest.mark.integration` for test selection via `-m` filtering - `skip_if_*_disabled` for credential/service availability checks The RUN_INTEGRATION_TESTS env var was redundant since the marker handles selection and the skip decorators already check for actual credentials. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
eavanvalkenburg
force-pushed
the
int_tests_update
branch
from
f857cf7 to
cf48785
Compare
moonbox3
reviewed
Feb 24, 2026
Add OPENAI_EMBEDDINGS_MODEL_ID and AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME to python-integration-tests.yml to match python-merge-tests.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
moonbox3
approved these changes
Feb 24, 2026
…docs Missed test_openai_embedding_client.py and vector-stores README in the earlier cleanup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
moonbox3
approved these changes
Feb 24, 2026
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
moonbox3
approved these changes
Feb 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Refined our integration tests runs for Python, split them further out to ensure we maximise throughput for non-block parallelism, ensured we only run integration tests in those paths, while in parallel running the unit tests. Added filters to only run what needs running.
Description
Contribution Checklist