Skip to content

CI runs only 4 of 272 unit-test files #222

Description

@0xKT

Problem

make test-python -- the only pytest invocation in CI -- hardcodes four file paths:

test-python:
	uv run --extra dev pytest tests/test_commit_lint.py tests/test_large_file_check.py tests/test_cli_smoke.py tests/test_litellm_setup.py -q

So CI executes 4 of 272 top-level test files (~1.5%), or 59 of 4448 collectable tests (~1.3%). The other ~268 files never run on any PR and can red-rot unnoticed. The Windows node-discovery test fixed in #189 was one symptom: it had been broken and nobody saw it, because CI never ran that file.

A second, independent defect: a bare uv run pytest does not even get to the running stage -- collection aborts with 7 errors, because 7 channel-adapter tests import optional SDKs (telegram, slack_sdk, nio, lark_oapi, botpy, dingtalk_stream, wecom_aibot_sdk) at module import time. So "just run everything in CI" fails immediately until those are guarded.

Third: testpaths = ["tests"] with no default deselection means a bare pytest also collects tests/integration/ (real LLM, real VM, real subprocess), which breaks on any machine without that environment configured.

Why it matters

A test suite that does not run is decorative. 268 files' worth of regression protection is currently inert, and contributors get no signal from it.

Proposed fix

  1. Guard the 7 channel-adapter tests with pytest.importorskip so collection succeeds.
  2. Make the default pytest invocation safe: register integration / e2e markers, add --strict-markers / --strict-config, exclude tests/integration via norecursedirs, and default to -m "not integration and not e2e".
  3. Relocate the integration-flavoured and real-resource files that sit in the flat unit tree into tests/integration/.
  4. Replace the hardcoded file list in make test-python with a bare pytest over the whole unit tree, and turn the CI python job into a blocking unit matrix.
  5. Fix whatever the first full run reveals.

Step 5 is not hypothetical: the first full run surfaced 44 pre-existing failures, including a test that asserted the provider catalog has exactly four concrete backends when it actually has six, and three tests that only passed because the developer's machine had a populated ~/.raven.

Note for maintainers

After this lands, the branch-protection required status check must be updated from python to unit -- the job is renamed, so the old required check would wait on a job that no longer exists.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions