[ci] Refactor CI of test cases in integration module#702
Open
weiqingy wants to merge 1 commit into
Open
Conversation
Tag the 10 tests under python/flink_agents/integrations/*/tests/ that
require live external services (Ollama, DashScope, OpenAI, Azure,
Anthropic, Chroma) or optional Python deps (mem0) with a new
@pytest.mark.integration marker. Register the marker in
python/pyproject.toml with strict_markers enabled. Update tools/ut.sh
to split unit and integration arms via the new marker:
- ut-python (run_e2e=false): adds -m "not integration" to the
existing -k "not e2e_tests" filter. Drops 70 tests from the
unit-test job (was 518, now 448), which previously skipped
silently in 5 of 6 matrix cells.
- it-python (run_e2e=true): runs two sequential pytest invocations
-- existing -k "e2e_tests_integration" (27 tests, unchanged) and
new -m "integration" (70 tests) -- with aggregated exit codes and
an exit-5 trap on the integration arm to defend against -m
selector typos that --strict-markers cannot catch.
No .github/workflows/ci.yml change required.
test_mcp.py is intentionally not tagged: it self-hosts an MCP server
via multiprocessing and runs deterministically without external deps.
Closes apache#161
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linked issue: #161
Purpose of change
Splits the Python unit-test job from the integration-test job using a new
@pytest.mark.integrationmarker.Today, 10 tests under
python/flink_agents/integrations/*/tests/need live external services (Ollama, DashScope, OpenAI, Azure, Anthropic, Chroma) or optional Python deps (mem0), but they live in the unit-test job and silently skip in 5 of 6 matrix cells — only the Python 3.10 cell auto-pulls Ollama. When Ollama regresses, exactly one cell reports failure, which looks like environmental noise.After this PR:
python/pyproject.toml— new[tool.pytest.ini_options]block registers theintegrationmarker and enablesstrict_markers = true.python/flink_agents/integrations/*/tests/— module-levelpytestmark = pytest.mark.integration(list form fortest_mem0_vector_store.pyto preserve its existingskipif).tools/ut.sh— bothif $run_e2e; then ... else ... fiblocks (uv branch + pip fallback) updated:ut-python(run_e2e=false): appends-m "not integration"to the existing-k "not e2e_tests"filter. Drops 70 tests from the unit-test job (was 518, now 448).it-python(run_e2e=true): runs two sequential pytest invocations — existing-k "e2e_tests_integration"(27 tests, unchanged) and new-m "integration"(70 tests) — with aggregated exit codes and an exit-5 trap on the integration arm to defend against-mselector typos that--strict-markerscannot catch.No
.github/workflows/ci.ymlchange required.integrations/mcp/tests/test_mcp.pyis intentionally not tagged: it self-hosts an MCP server viamultiprocessingand runs deterministically without external deps.Tests
Locally verified empirical contract (
pytest --collect-only -q):pytest -m "not integration" -k "not e2e_tests"ut-pythonarm (was 518)pytest -m "integration"it-pythonarm 2pytest -k "e2e_tests_integration"it-pythonarm 1, unchangedOverlap between
-m "integration"and-k "e2e_tests_integration"is 0 — disjoint by directory structure.Additional verification:
./tools/lint.sh -cpasses./tools/check-license.shpasses./tools/ut.sh -ppasses locally (448 passed, 104 deselected, ~28s)pytest --markersshows the registeredintegrationmarker with description@pytest.mark.intergrationtriggers a collection error (strict_markers = trueenforcement is active)API
No public API changes. This PR is CI plumbing plus a pytest marker added to existing test files.
Documentation
doc-neededdoc-not-neededdoc-included