-
Notifications
You must be signed in to change notification settings - Fork 571
ci: Split up Test AI workflow #5148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+313
−37
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| # Do not edit this YAML file. This file is generated automatically by executing | ||
| # python scripts/split_tox_gh_actions/split_tox_gh_actions.py | ||
| # The template responsible for it is in | ||
| # scripts/split_tox_gh_actions/templates/base.jinja | ||
| name: Test Agents | ||
| on: | ||
| push: | ||
| branches: | ||
| - master | ||
| - release/** | ||
| - major/** | ||
| pull_request: | ||
| # Cancel in progress workflows on pull_requests. | ||
| # https://docs.github.com/en/actions/using-jobs/using-concurrency#example-using-a-fallback-value | ||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} | ||
| cancel-in-progress: true | ||
| permissions: | ||
| contents: read | ||
| env: | ||
| BUILD_CACHE_KEY: ${{ github.sha }} | ||
| CACHED_BUILD_PATHS: | | ||
| ${{ github.workspace }}/dist-serverless | ||
| jobs: | ||
| test-agents: | ||
| name: Agents | ||
| timeout-minutes: 30 | ||
| runs-on: ${{ matrix.os }} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| python-version: ["3.10","3.11","3.12","3.13","3.14","3.14t"] | ||
| # python3.6 reached EOL and is no longer being supported on | ||
| # new versions of hosted runners on Github Actions | ||
| # ubuntu-20.04 is the last version that supported python3.6 | ||
| # see https://github.com/actions/setup-python/issues/544#issuecomment-1332535877 | ||
| os: [ubuntu-22.04] | ||
| # Use Docker container only for Python 3.6 | ||
| container: ${{ matrix.python-version == '3.6' && 'python:3.6' || null }} | ||
| steps: | ||
| - uses: actions/checkout@v6.0.0 | ||
| - uses: actions/setup-python@v6 | ||
| if: ${{ matrix.python-version != '3.6' }} | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
| allow-prereleases: true | ||
| - name: Setup Test Env | ||
| run: | | ||
| pip install "coverage[toml]" tox | ||
| - name: Erase coverage | ||
| run: | | ||
| coverage erase | ||
| - name: Test openai_agents | ||
| run: | | ||
| set -x # print commands that are executed | ||
| ./scripts/runtox.sh "py${{ matrix.python-version }}-openai_agents" | ||
| - name: Test pydantic_ai | ||
| run: | | ||
| set -x # print commands that are executed | ||
| ./scripts/runtox.sh "py${{ matrix.python-version }}-pydantic_ai" | ||
| - name: Generate coverage XML (Python 3.6) | ||
| if: ${{ !cancelled() && matrix.python-version == '3.6' }} | ||
| run: | | ||
| export COVERAGE_RCFILE=.coveragerc36 | ||
| coverage combine .coverage-sentry-* | ||
| coverage xml --ignore-errors | ||
| - name: Generate coverage XML | ||
| if: ${{ !cancelled() && matrix.python-version != '3.6' }} | ||
| run: | | ||
| coverage combine .coverage-sentry-* | ||
| coverage xml | ||
| - name: Upload coverage to Codecov | ||
| if: ${{ !cancelled() }} | ||
| uses: codecov/codecov-action@v5.5.1 | ||
| with: | ||
| token: ${{ secrets.CODECOV_TOKEN }} | ||
| files: coverage.xml | ||
| # make sure no plugins alter our coverage reports | ||
| plugins: noop | ||
| verbose: true | ||
| - name: Upload test results to Codecov | ||
| if: ${{ !cancelled() }} | ||
| uses: codecov/test-results-action@v1 | ||
| with: | ||
| token: ${{ secrets.CODECOV_TOKEN }} | ||
| files: .junitxml | ||
| verbose: true | ||
| check_required_tests: | ||
| name: All Agents tests passed | ||
| needs: test-agents | ||
| # Always run this, even if a dependent job failed | ||
| if: always() | ||
| runs-on: ubuntu-22.04 | ||
| steps: | ||
| - name: Check for failures | ||
| if: needs.test-agents.result != 'success' | ||
| run: | | ||
| echo "One of the dependent jobs has failed. You may need to re-run it." && exit 1 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| # Do not edit this YAML file. This file is generated automatically by executing | ||
| # python scripts/split_tox_gh_actions/split_tox_gh_actions.py | ||
| # The template responsible for it is in | ||
| # scripts/split_tox_gh_actions/templates/base.jinja | ||
| name: Test AI Workflow | ||
| on: | ||
| push: | ||
| branches: | ||
| - master | ||
| - release/** | ||
| - major/** | ||
| pull_request: | ||
| # Cancel in progress workflows on pull_requests. | ||
| # https://docs.github.com/en/actions/using-jobs/using-concurrency#example-using-a-fallback-value | ||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} | ||
| cancel-in-progress: true | ||
| permissions: | ||
| contents: read | ||
| env: | ||
| BUILD_CACHE_KEY: ${{ github.sha }} | ||
| CACHED_BUILD_PATHS: | | ||
| ${{ github.workspace }}/dist-serverless | ||
| jobs: | ||
| test-ai_workflow: | ||
| name: AI Workflow | ||
| timeout-minutes: 30 | ||
| runs-on: ${{ matrix.os }} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| python-version: ["3.9","3.10","3.11","3.12","3.13","3.14"] | ||
| # python3.6 reached EOL and is no longer being supported on | ||
| # new versions of hosted runners on Github Actions | ||
| # ubuntu-20.04 is the last version that supported python3.6 | ||
| # see https://github.com/actions/setup-python/issues/544#issuecomment-1332535877 | ||
| os: [ubuntu-22.04] | ||
| # Use Docker container only for Python 3.6 | ||
| container: ${{ matrix.python-version == '3.6' && 'python:3.6' || null }} | ||
| steps: | ||
| - uses: actions/checkout@v6.0.0 | ||
| - uses: actions/setup-python@v6 | ||
| if: ${{ matrix.python-version != '3.6' }} | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
| allow-prereleases: true | ||
| - name: Setup Test Env | ||
| run: | | ||
| pip install "coverage[toml]" tox | ||
| - name: Erase coverage | ||
| run: | | ||
| coverage erase | ||
| - name: Test langchain-base | ||
| run: | | ||
| set -x # print commands that are executed | ||
| ./scripts/runtox.sh "py${{ matrix.python-version }}-langchain-base" | ||
| - name: Test langchain-notiktoken | ||
| run: | | ||
| set -x # print commands that are executed | ||
| ./scripts/runtox.sh "py${{ matrix.python-version }}-langchain-notiktoken" | ||
| - name: Test langgraph | ||
| run: | | ||
| set -x # print commands that are executed | ||
| ./scripts/runtox.sh "py${{ matrix.python-version }}-langgraph" | ||
| - name: Generate coverage XML (Python 3.6) | ||
| if: ${{ !cancelled() && matrix.python-version == '3.6' }} | ||
| run: | | ||
| export COVERAGE_RCFILE=.coveragerc36 | ||
| coverage combine .coverage-sentry-* | ||
| coverage xml --ignore-errors | ||
| - name: Generate coverage XML | ||
| if: ${{ !cancelled() && matrix.python-version != '3.6' }} | ||
| run: | | ||
| coverage combine .coverage-sentry-* | ||
| coverage xml | ||
| - name: Upload coverage to Codecov | ||
| if: ${{ !cancelled() }} | ||
| uses: codecov/codecov-action@v5.5.1 | ||
| with: | ||
| token: ${{ secrets.CODECOV_TOKEN }} | ||
| files: coverage.xml | ||
| # make sure no plugins alter our coverage reports | ||
| plugins: noop | ||
| verbose: true | ||
| - name: Upload test results to Codecov | ||
| if: ${{ !cancelled() }} | ||
| uses: codecov/test-results-action@v1 | ||
| with: | ||
| token: ${{ secrets.CODECOV_TOKEN }} | ||
| files: .junitxml | ||
| verbose: true | ||
| check_required_tests: | ||
| name: All AI Workflow tests passed | ||
| needs: test-ai_workflow | ||
| # Always run this, even if a dependent job failed | ||
| if: always() | ||
| runs-on: ubuntu-22.04 | ||
| steps: | ||
| - name: Check for failures | ||
| if: needs.test-ai_workflow.result != 'success' | ||
| run: | | ||
| echo "One of the dependent jobs has failed. You may need to re-run it." && exit 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| # Do not edit this YAML file. This file is generated automatically by executing | ||
| # python scripts/split_tox_gh_actions/split_tox_gh_actions.py | ||
| # The template responsible for it is in | ||
| # scripts/split_tox_gh_actions/templates/base.jinja | ||
| name: Test MCP | ||
| on: | ||
| push: | ||
| branches: | ||
| - master | ||
| - release/** | ||
| - major/** | ||
| pull_request: | ||
| # Cancel in progress workflows on pull_requests. | ||
| # https://docs.github.com/en/actions/using-jobs/using-concurrency#example-using-a-fallback-value | ||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} | ||
| cancel-in-progress: true | ||
| permissions: | ||
| contents: read | ||
| env: | ||
| BUILD_CACHE_KEY: ${{ github.sha }} | ||
| CACHED_BUILD_PATHS: | | ||
| ${{ github.workspace }}/dist-serverless | ||
| jobs: | ||
| test-mcp: | ||
| name: MCP | ||
| timeout-minutes: 30 | ||
| runs-on: ${{ matrix.os }} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| python-version: ["3.10","3.12","3.13","3.14","3.14t"] | ||
| # python3.6 reached EOL and is no longer being supported on | ||
| # new versions of hosted runners on Github Actions | ||
| # ubuntu-20.04 is the last version that supported python3.6 | ||
| # see https://github.com/actions/setup-python/issues/544#issuecomment-1332535877 | ||
| os: [ubuntu-22.04] | ||
| # Use Docker container only for Python 3.6 | ||
| container: ${{ matrix.python-version == '3.6' && 'python:3.6' || null }} | ||
| steps: | ||
| - uses: actions/checkout@v6.0.0 | ||
| - uses: actions/setup-python@v6 | ||
| if: ${{ matrix.python-version != '3.6' }} | ||
| with: | ||
| python-version: ${{ matrix.python-version }} | ||
| allow-prereleases: true | ||
| - name: Setup Test Env | ||
| run: | | ||
| pip install "coverage[toml]" tox | ||
| - name: Erase coverage | ||
| run: | | ||
| coverage erase | ||
| - name: Test mcp | ||
| run: | | ||
| set -x # print commands that are executed | ||
| ./scripts/runtox.sh "py${{ matrix.python-version }}-mcp" | ||
| - name: Test fastmcp | ||
| run: | | ||
| set -x # print commands that are executed | ||
| ./scripts/runtox.sh "py${{ matrix.python-version }}-fastmcp" | ||
| - name: Generate coverage XML (Python 3.6) | ||
| if: ${{ !cancelled() && matrix.python-version == '3.6' }} | ||
| run: | | ||
| export COVERAGE_RCFILE=.coveragerc36 | ||
| coverage combine .coverage-sentry-* | ||
| coverage xml --ignore-errors | ||
| - name: Generate coverage XML | ||
| if: ${{ !cancelled() && matrix.python-version != '3.6' }} | ||
| run: | | ||
| coverage combine .coverage-sentry-* | ||
| coverage xml | ||
| - name: Upload coverage to Codecov | ||
| if: ${{ !cancelled() }} | ||
| uses: codecov/codecov-action@v5.5.1 | ||
| with: | ||
| token: ${{ secrets.CODECOV_TOKEN }} | ||
| files: coverage.xml | ||
| # make sure no plugins alter our coverage reports | ||
| plugins: noop | ||
| verbose: true | ||
| - name: Upload test results to Codecov | ||
| if: ${{ !cancelled() }} | ||
| uses: codecov/test-results-action@v1 | ||
| with: | ||
| token: ${{ secrets.CODECOV_TOKEN }} | ||
| files: .junitxml | ||
| verbose: true | ||
| check_required_tests: | ||
| name: All MCP tests passed | ||
| needs: test-mcp | ||
| # Always run this, even if a dependent job failed | ||
| if: always() | ||
| runs-on: ubuntu-22.04 | ||
| steps: | ||
| - name: Check for failures | ||
| if: needs.test-mcp.result != 'success' | ||
| run: | | ||
| echo "One of the dependent jobs has failed. You may need to re-run it." && exit 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bug: Python version matrix includes unsupported combinations
The workflow Python version matrices include versions unsupported by some frameworks in each group. The generator script unions all Python versions across grouped frameworks, causing tests to fail for nonexistent tox environments. For example,
pydantic_ailacks Python 3.11, 3.14, and 3.14t intox.ini, yet the Agents workflow will attemptpy3.11-pydantic_ai. Similarly,langgraphlacks 3.11, andmcplacks 3.14 and 3.14t in their respective workflows.Additional Locations (2)
.github/workflows/test-integrations-ai-workflow.yml#L31-L32.github/workflows/test-integrations-mcp.yml#L31-L32