Problem
make test-python -- the only pytest invocation in CI -- hardcodes four file paths:
test-python:
uv run --extra dev pytest tests/test_commit_lint.py tests/test_large_file_check.py tests/test_cli_smoke.py tests/test_litellm_setup.py -q
So CI executes 4 of 272 top-level test files (~1.5%), or 59 of 4448 collectable tests (~1.3%). The other ~268 files never run on any PR and can red-rot unnoticed. The Windows node-discovery test fixed in #189 was one symptom: it had been broken and nobody saw it, because CI never ran that file.
A second, independent defect: a bare uv run pytest does not even get to the running stage -- collection aborts with 7 errors, because 7 channel-adapter tests import optional SDKs (telegram, slack_sdk, nio, lark_oapi, botpy, dingtalk_stream, wecom_aibot_sdk) at module import time. So "just run everything in CI" fails immediately until those are guarded.
Third: testpaths = ["tests"] with no default deselection means a bare pytest also collects tests/integration/ (real LLM, real VM, real subprocess), which breaks on any machine without that environment configured.
Why it matters
A test suite that does not run is decorative. 268 files' worth of regression protection is currently inert, and contributors get no signal from it.
Proposed fix
- Guard the 7 channel-adapter tests with
pytest.importorskip so collection succeeds.
- Make the default
pytest invocation safe: register integration / e2e markers, add --strict-markers / --strict-config, exclude tests/integration via norecursedirs, and default to -m "not integration and not e2e".
- Relocate the integration-flavoured and real-resource files that sit in the flat unit tree into
tests/integration/.
- Replace the hardcoded file list in
make test-python with a bare pytest over the whole unit tree, and turn the CI python job into a blocking unit matrix.
- Fix whatever the first full run reveals.
Step 5 is not hypothetical: the first full run surfaced 44 pre-existing failures, including a test that asserted the provider catalog has exactly four concrete backends when it actually has six, and three tests that only passed because the developer's machine had a populated ~/.raven.
Note for maintainers
After this lands, the branch-protection required status check must be updated from python to unit -- the job is renamed, so the old required check would wait on a job that no longer exists.
Problem
make test-python-- the only pytest invocation in CI -- hardcodes four file paths:test-python: uv run --extra dev pytest tests/test_commit_lint.py tests/test_large_file_check.py tests/test_cli_smoke.py tests/test_litellm_setup.py -qSo CI executes 4 of 272 top-level test files (~1.5%), or 59 of 4448 collectable tests (~1.3%). The other ~268 files never run on any PR and can red-rot unnoticed. The Windows node-discovery test fixed in #189 was one symptom: it had been broken and nobody saw it, because CI never ran that file.
A second, independent defect: a bare
uv run pytestdoes not even get to the running stage -- collection aborts with 7 errors, because 7 channel-adapter tests import optional SDKs (telegram,slack_sdk,nio,lark_oapi,botpy,dingtalk_stream,wecom_aibot_sdk) at module import time. So "just run everything in CI" fails immediately until those are guarded.Third:
testpaths = ["tests"]with no default deselection means a barepytestalso collectstests/integration/(real LLM, real VM, real subprocess), which breaks on any machine without that environment configured.Why it matters
A test suite that does not run is decorative. 268 files' worth of regression protection is currently inert, and contributors get no signal from it.
Proposed fix
pytest.importorskipso collection succeeds.pytestinvocation safe: registerintegration/e2emarkers, add--strict-markers/--strict-config, excludetests/integrationvianorecursedirs, and default to-m "not integration and not e2e".tests/integration/.make test-pythonwith a barepytestover the whole unit tree, and turn the CIpythonjob into a blocking unit matrix.Step 5 is not hypothetical: the first full run surfaced 44 pre-existing failures, including a test that asserted the provider catalog has exactly four concrete backends when it actually has six, and three tests that only passed because the developer's machine had a populated
~/.raven.Note for maintainers
After this lands, the branch-protection required status check must be updated from
pythontounit-- the job is renamed, so the old required check would wait on a job that no longer exists.