Day 2: Smoke tests + known-issues documentation#2
Merged
Merged
Conversation
Two xfail tests reproduce the current broken state: - test_cli_scan_dry_run_exits_cleanly: CLI calls Orchestrator(config) and orchestrator.run_pipeline(session) — neither matches the actual API - test_cli_scan_dry_run_produces_output: same root cause One always-passing sanity check: - test_cli_help_works: ensures the CLI module at least imports cleanly xfail(strict=False) — when day 7 fixes the API mismatch, these tests will XPASS without failing CI. When we un-xfail them in day 7, they will provide actual regression protection. Refs: STANDOFF.md day 2/30
- mock_llm_client: MagicMock with .call() / .acall() returning stub strings, for tests that exercise agents without real API keys - mock_nmap_result: realistic nmap output dict for recon-dependent tests - mock_nvd_response: NVD API 2.0 response shape for intel tests - session_with_recon: fix broken 'knowledge_base[...]' access — the actual PentestSession field is 'recon_data' (this fixture was silently broken) All fixtures are typed and documented for IDE autocomplete. Refs: STANDOFF.md day 2/30
Changes: - pytest.ini: add 'smoke' and 'network' markers, enable --strict-markers - ci.yml: add dedicated 'smoke' job with continue-on-error=true (smoke tests are xfail until day 7, so we don't want them blocking PRs yet) - ci.yml: exclude smoke tests from main integration run via '-m not smoke' Rationale: smoke tests reproduce known broken state; they should be visible in CI but not block merges. Once the API mismatch is fixed in day 7, we'll flip continue-on-error to false. Refs: STANDOFF.md day 2/30
Lists every broken API contract between CLI, Orchestrator, BaseAgent, agents, and tests, with: - concrete symptom (what error / what doesn't work) - which day of STANDOFF.md fixes it - progress tracker table for visual closure KI-8 (conftest.knowledge_base access) is fixed by this very PR — the fixture now uses .recon_data which is the actual PentestSession field. The other 7 issues will be checked off across days 3-7. This doc serves as a public 'before' snapshot — when day 7 is done, all rows turn green and the file gets archived. Refs: STANDOFF.md day 2/30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of 30-day STANDOFF plan. Day 2 of 30.
Reproduces and documents the current broken state — does NOT fix it yet.
Changes
tests/integration/test_cli_smoke.py(new) — 3 smoke teststests/conftest.py— mock fixtures + fix KI-8pytest.ini— smoke marker + strict markers.github/workflows/ci.yml— separate smoke jobdocs/architecture/known-issues.md(new) — 8 documented issuesLocal verification
pytest -m smoke → 1 passed, 1 xfailed, 1 xpassed
Refs: STANDOFF.md