Skip to content

[ci] Refactor CI of test cases in integration module#702

Open
weiqingy wants to merge 1 commit into
apache:mainfrom
weiqingy:161-impl
Open

[ci] Refactor CI of test cases in integration module#702
weiqingy wants to merge 1 commit into
apache:mainfrom
weiqingy:161-impl

Conversation

@weiqingy
Copy link
Copy Markdown
Collaborator

Linked issue: #161

Purpose of change

Splits the Python unit-test job from the integration-test job using a new @pytest.mark.integration marker.

Today, 10 tests under python/flink_agents/integrations/*/tests/ need live external services (Ollama, DashScope, OpenAI, Azure, Anthropic, Chroma) or optional Python deps (mem0), but they live in the unit-test job and silently skip in 5 of 6 matrix cells — only the Python 3.10 cell auto-pulls Ollama. When Ollama regresses, exactly one cell reports failure, which looks like environmental noise.

After this PR:

  • python/pyproject.toml — new [tool.pytest.ini_options] block registers the integration marker and enables strict_markers = true.
  • 10 test files under python/flink_agents/integrations/*/tests/ — module-level pytestmark = pytest.mark.integration (list form for test_mem0_vector_store.py to preserve its existing skipif).
  • tools/ut.sh — both if $run_e2e; then ... else ... fi blocks (uv branch + pip fallback) updated:
    • ut-python (run_e2e=false): appends -m "not integration" to the existing -k "not e2e_tests" filter. Drops 70 tests from the unit-test job (was 518, now 448).
    • it-python (run_e2e=true): runs two sequential pytest invocations — existing -k "e2e_tests_integration" (27 tests, unchanged) and new -m "integration" (70 tests) — with aggregated exit codes and an exit-5 trap on the integration arm to defend against -m selector typos that --strict-markers cannot catch.

No .github/workflows/ci.yml change required.

integrations/mcp/tests/test_mcp.py is intentionally not tagged: it self-hosts an MCP server via multiprocessing and runs deterministically without external deps.

Tests

Locally verified empirical contract (pytest --collect-only -q):

Selector Tests Notes
pytest -m "not integration" -k "not e2e_tests" 448 New ut-python arm (was 518)
pytest -m "integration" 70 New it-python arm 2
pytest -k "e2e_tests_integration" 27 it-python arm 1, unchanged
no selector 552 Unchanged

Overlap between -m "integration" and -k "e2e_tests_integration" is 0 — disjoint by directory structure.

Additional verification:

  • ./tools/lint.sh -c passes
  • ./tools/check-license.sh passes
  • ./tools/ut.sh -p passes locally (448 passed, 104 deselected, ~28s)
  • pytest --markers shows the registered integration marker with description
  • Typo'd @pytest.mark.intergration triggers a collection error (strict_markers = true enforcement is active)

API

No public API changes. This PR is CI plumbing plus a pytest marker added to existing test files.

Documentation

  • doc-needed
  • doc-not-needed
  • doc-included

Tag the 10 tests under python/flink_agents/integrations/*/tests/ that
require live external services (Ollama, DashScope, OpenAI, Azure,
Anthropic, Chroma) or optional Python deps (mem0) with a new
@pytest.mark.integration marker. Register the marker in
python/pyproject.toml with strict_markers enabled. Update tools/ut.sh
to split unit and integration arms via the new marker:

  - ut-python (run_e2e=false): adds -m "not integration" to the
    existing -k "not e2e_tests" filter. Drops 70 tests from the
    unit-test job (was 518, now 448), which previously skipped
    silently in 5 of 6 matrix cells.
  - it-python (run_e2e=true): runs two sequential pytest invocations
    -- existing -k "e2e_tests_integration" (27 tests, unchanged) and
    new -m "integration" (70 tests) -- with aggregated exit codes and
    an exit-5 trap on the integration arm to defend against -m
    selector typos that --strict-markers cannot catch.

No .github/workflows/ci.yml change required.

test_mcp.py is intentionally not tagged: it self-hosts an MCP server
via multiprocessing and runs deterministically without external deps.

Closes apache#161
@github-actions github-actions Bot added doc-not-needed Your PR changes do not impact docs fixVersion/0.3.0 The feature or bug should be implemented/fixed in the 0.3.0 version. priority/major Default priority of the PR or issue. labels May 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs fixVersion/0.3.0 The feature or bug should be implemented/fixed in the 0.3.0 version. priority/major Default priority of the PR or issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant