Forbid ::warning:: advisory pattern; pip-audit + benchmark fail-fast#550
Merged
Conversation
…l-fast Ports the Bucket F regression guard from HomericIntelligence/Odysseus#282 and refactors 2 advisory-annotation site(s) to fail-fast: - .github/workflows/_required.yml:693 (pip-audit --strict) - .github/workflows/extras.yml:73 (make benchmark.native) Local pip-audit --strict --skip-editable: no findings. make benchmark.native exits non-zero only on real benchmark errors; non-blocking timing regressions are reported but do not gate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
Removing the ::warning:: suppressions in #550 surfaced two pre-existing issues that the advisory annotations had been hiding. ## pip-audit --strict CI failed with: ERROR:pip_audit._cli:projectkeystone: Dependency not found on PyPI and could not be audited: projectkeystone (0.1.0) No CVEs — the failure is that `pip install -e ".[dev]"` registers the local project in the venv, and pip-audit then tries to look it up on PyPI. Keystone is intentionally not published to PyPI. The third-party dependency tree itself is clean (matches the PR author's local `pip-audit --strict --skip-editable` run). Fix: add `--skip-editable` to the CI invocation with a comment explaining why. This is not an `--ignore-vuln` allowlist (no vulnerability is being suppressed); it's scoping pip-audit to the package set it can actually audit. ## make benchmark.native CI failed with: Error: Build directory not found make[1]: *** [Makefile:171: benchmark] Error 1 Real bug. `scripts/run_benchmarks.sh` defaulted `BUILD_DIR` to `$PROJECT_ROOT/build/release/bin`, but the Makefile actually produces binaries at `$(BUILD_DIR)/$(BUILD_SUBDIR)` -> `build/x86.release/`. The path the script looked at never existed; the prior `if ! make benchmark.native; then echo "::warning::..."; fi` had been swallowing this and benchmarks were never actually exercised in CI. Fix: point the script's default at `build/x86.release` and let callers override `BUILD_DIR` (full path) or `BUILD_SUBDIR` (suffix only). Follow-up tracked in #551 (single source of truth between Makefile and script; per-binary missing detection; doc reconciliation). Refs: #550, #551 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two pre-existing issues surfaced by removing the ::warning:: suppressions in #550 were only partially fixed by d10c8eb. Both checks were still red on run 25652460546 / 25652460547. Real root causes and fixes below. ## security/dependency-scan (Required) The previous fix tried `pip install -e ".[dev]" && pip-audit --strict --skip-editable`. In pip-audit 2.10.0 (newer than the pinned version), `--strict` treats a `--skip-editable` skip as a dependency-collection failure and errors out: ERROR:pip_audit._cli:projectkeystone: distribution marked as editable Process completed with exit code 1. `projectkeystone` is intentionally absent from PyPI — pyproject.toml exists only so Python dev/test tools (mypy, conan, pytest, ...) can be declared. There is no "install the project" requirement for an audit. Fix: drop the editable install entirely and pass the project path as a positional argument. `pip-audit --strict .` resolves pyproject.toml's production dependency closure (pydantic, nats-py, pydantic_core, annotated-types, typing_extensions, typing-inspection) directly from PyPI metadata without installing anything. Locally: $ pip-audit --strict . No known vulnerabilities found No `--ignore-vuln` allowlist needed; the closure is clean. No suppression patterns introduced — still fail-fast on findings. ## benchmarks (Extras / non-blocking) The previous fix pointed `BUILD_DIR` at `build/x86.release`, but CMakeLists.txt:88 sets `CMAKE_RUNTIME_OUTPUT_DIRECTORY` to `${CMAKE_BINARY_DIR}/bin`. CI log confirmed the linker produced `bin/message_pool_benchmarks`, `bin/hierarchy_benchmarks`, `bin/distributed_benchmarks` — but the script kept looking at `build/x86.release/hierarchy_benchmarks` (no bin/), so every binary was reported missing and `make benchmark` exited non-zero. Fix: introduce `BENCH_BIN_DIR` (default `$BUILD_DIR/bin`) and use it everywhere the script previously read `$BUILD_DIR/$bench`. Callers can still override `BUILD_DIR` (full path to the CMake binary dir), `BUILD_SUBDIR` (suffix only), or `BENCH_BIN_DIR` (full path to binaries) to point at non-standard layouts. Refs: #550, #551 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ees them After fixing the benchmark binary lookup path in 3145bc9, CI got further: all three benchmark executables (hierarchy, message_pool, distributed) ran to completion and wrote their per-suite JSON files. The post-run "Merging results..." heredoc then failed with: No result files found matching None/*_None.json make[1]: *** [Makefile:171: benchmark] Error 1 `python3 << EOF` runs as a subprocess of bash and only inherits *exported* environment variables. `RESULTS_DIR`, `TIMESTAMP`, `RESULTS_FILE`, and `COMPARE_BASELINE` were plain shell locals, so `os.environ.get('RESULTS_DIR')` returned None on both the merge heredoc and the regression-compare heredoc. Fix: export the four shell variables the heredocs reference. Refs: #550 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the Bucket F regression guard from HomericIntelligence/Odysseus#282 and refactors 2 advisory-annotation sites to fail-fast.
Refactor sites
.github/workflows/_required.ymlpip-audit --strictif ! pip-audit --strict; then echo "::warning::..."; fipip-audit --strict.github/workflows/extras.ymlmake benchmark.nativeif ! make benchmark.native; then echo "::warning::..."; fimake benchmark.nativeLint guard
forbid-advisory-warningspygrep hook to.pre-commit-config.yaml(exempts_required.ymlfor self-documentation).Reject advisory annotation patternstep to_required.ymlforbid-suppressionsjob.Local verification
pre-commit run forbid-or-true --all-files-> Passedpre-commit run forbid-continue-on-error --all-files-> Passedpre-commit run forbid-advisory-warnings --all-files-> Passedpip-audit --strict --skip-editableagainst.[dev]deps (pydantic>=2.0, nats-py>=2,<3, mypy, conan, pytest, pytest-asyncio, pytest-cov, pytest-timeout): no findingsBenchmark decision
make benchmark.nativecallsscripts/run_benchmarks.sh(usesset -e). The script exits non-zero on:--compare; CI invokes without--compare)So bare
make benchmark.nativecorrectly fail-fasts on real errors (broken builds, crashed binaries) while still emitting timing data on a happy run.extras.ymlitself is a non-required workflow ("Extras (non-blocking)"), so the policy ("every required CI tool must fail-fast") technically doesn't gate it — but the pygrep lint rule covers all workflow files, so the bare invocation is also the cleanest path.Reference: HomericIntelligence/Odysseus#282
Test plan
forbid-suppressionsjob passesdependency-scanjob runspip-audit --strictcleanlyExtrasworkflowbenchmarksjob runsmake benchmark.nativeto completion🤖 Generated with Claude Code