Use cargo-nextest for Rust tests

[lucarlig]

Summary

• install cargo-nextest 0.9.133 with cargo in Rust plugin CI and coverage jobs
• add a repository nextest config with ci and mutants profiles
• run per-plugin Rust unit tests with cargo nextest run and explicit package selection
• use the documented cargo llvm-cov nextest --no-report wrapper for the Rust coverage phase
• run Criterion benchmarks in nextest test mode via cargo nextest run --benches for CI benchmark verification
• install cargo-mutants 27.0.0 in PR CI with cargo and run it through nextest
• gate cargo-mutants on Rust code touched by the pull request diff using --in-diff
• expose local cargo-mutants Make targets for full mutation testing
• keep Python integration tests on pytest and preserve the existing PyO3 coverage report flow
• document the nextest local workflow and current doctest behavior

Closes #66

Validation

• cargo install cargo-nextest --version 0.9.133 --locked
• cargo nextest --version -> cargo-nextest 0.9.133 (65e806bd5 2026-04-14)
• cargo nextest show-config version -> required 0.9.133, evaluation result ok
• cargo install cargo-mutants --version 27.0.0 --locked
• cargo mutants --version -> cargo-mutants 27.0.0
• make plugin-mutants-list PLUGIN=retry_with_backoff -> listed retry_with_backoff mutants
• make -n plugin-mutants PLUGIN=retry_with_backoff -> cargo mutants -p "retry_with_backoff"
• cargo mutants -p rate_limiter --in-diff /tmp/cpex-mutants.diff --list -> no mutants to filter for this PR diff
• python3 -m unittest tests.test_plugin_catalog.PluginCatalogTests.test_ci_workflow_shared_paths_match_catalog_contract tests.test_plugin_catalog.PluginCatalogTests.test_ci_workflow_includes_parity_jobs_for_rust_plugin_checks tests.test_plugin_catalog.PluginCatalogTests.test_testing_docs_include_local_rust_coverage_command -> OK
• actionlint .github/workflows/ci-rust-python-package.yaml .github/workflows/ci-scaffold-generator.yaml
• git diff --check
• cargo fmt -- --check
• make plugins-validate -> 106 tests run, OK (skipped=2)
• cd plugins/rust/python-package/rate_limiter && make test-unit -> 57 tests run, 57 passed
• cd plugins/rust/python-package/rate_limiter && NEXTEST_PROFILE=ci make test-unit -> 57 tests run, 57 passed
• cd plugins/rust/python-package/rate_limiter && NEXTEST_PROFILE=ci make bench-no-run -> 67 tests run, 67 passed
• cd plugins/rust/python-package/url_reputation && NEXTEST_PROFILE=ci make bench-no-run -> 35 tests run, 35 passed
• cd plugins/rust/python-package/retry_with_backoff && NEXTEST_PROFILE=ci make coverage -> cargo llvm-cov nextest Rust phase passed; pytest/PyO3 coverage report generated; coverage-check passed

Notes

• nextest does not run Rust doctests; this repo currently has Rust docs but no Rust doctest code blocks, so no doctest step is added.
• no nextest retries, partitions, JUnit, archive/reuse, or extra timeout policy are added because the repo does not currently need them.
• Criterion benchmarks are run in test mode to catch compile/panic regressions without treating noisy CI runners as performance measurement environments.
• cargo-mutants runs in PR CI through nextest, scoped to the PR Rust diff so existing missed mutants on main do not fail unrelated changes. Full mutation runs remain available locally through the root Makefile targets.

[msureshkumar88]

Code Review

Overview

This PR migrates Rust tests from cargo test to cargo-nextest across all plugins, adds a mutation-testing CI job using cargo-mutants, and introduces .config/nextest.toml and .cargo/mutants.toml configs. The implementation is thorough and the validation section in the PR description is detailed. CI jobs are still running (most checks are pending), so the below focuses on code-level issues.

---

Issues Requiring Changes

1. cargo install from source is slow — use a binary installer

In every CI job (build-test matrix × 15 jobs, coverage, mutation-testing), cargo-nextest is compiled from source:

cargo install cargo-nextest --version 0.9.133 --locked

This adds significant compile time per job (nextest is a large binary). The nextest team provides pre-built binaries and an official install action. Using cargo-binstall is the conventional fast path:

- name: Install cargo-nextest
  run: |
    curl -LsSf https://get.nexte.st/0.9.133/linux | tar zxf - -C ${CARGO_HOME:-~/.cargo}/bin
    cargo nextest --version

Or via the official action:

- uses: taiki-e/install-action@v2
  with:
    tool: cargo-nextest@0.9.133

Same concern applies to cargo-mutants in the mutation-testing job.

2. Python::initialize() added to a Rust test needs justification

In plugins/rust/python-package/rate_limiter/src/plugin.rs:

Python::initialize();
Python::attach(|py| -> PyResult<()> {

Adding Python::initialize() inside a test function is unusual and not present in other plugins. If nextest is running tests in separate processes (which is its default), this may be masking a test isolation issue rather than fixing an underlying problem. Please:

• Add a comment explaining why this is needed specifically for nextest
• Verify other plugins don't have the same issue (pii_filter, retry_with_backoff, etc. use similar PyO3 patterns but weren't updated)

3. has_mutation_cargo_packages uses fragile string comparison

if [[ "${mutation_cargo_packages}" == "[]" ]]; then
  has_mutation_cargo_packages_output="false"

This is a string equality check against the literal []. While it works when Python serializes an empty list, it's fragile (whitespace, ordering). More robust:

if python3 -c "import json, sys; sys.exit(0 if json.loads('${mutation_cargo_packages}') else 1)" 2>/dev/null; then
  has_mutation_cargo_packages_output="true"
else
  has_mutation_cargo_packages_output="false"
fi

This also parallels the existing pattern for has_plugins.

4. The Python validation one-liner has grown past readability

The inline Python one-liner in the validate-and-detect step has grown to ~600 characters, and this PR extends it further with mutation_cargo_packages and mutation_jobs validation. The mutation_jobs validation in particular uses a non-obvious generator-throw pattern:

[(_ for _ in ()).throw(AssertionError()) for job in mutation_jobs if not (...)]

This is hard to audit for correctness. Consider extracting this to a small Python script in tools/ (similar to plugin_catalog.py). This is an existing debt, but extending a known problem further makes it the right time to address it.

---

Suggestions (Non-blocking)

• No doctest guard: Since nextest skips Rust doctests by design, if a future contributor adds doctest blocks they will silently not run in CI. A minimal cargo test --doc step in build-test (even just --no-run as a compile check) would catch this. A TODO comment or follow-up issue would also suffice.

• nextest-version pinning in .config/nextest.toml: nextest-version = "0.9.133" enforces a minimum version, which is good. This aligns with the locked cargo install version.

• retry_with_backoff/Makefile: Unlike the other plugins, CARGO_PACKAGE := is not added in this diff (only NEXTEST_PROFILE ?=). This is presumably because it was already defined earlier in that file — worth confirming the variable is set before the nextest run -p $(CARGO_PACKAGE) line to avoid a silent empty -p flag.

[lucarlig]

@msureshkumar88 Thanks for the detailed review — I’ve pushed follow-up changes to address the requested items.

Implemented:

• Replaced source installs for cargo-nextest with the official nexte.st binary installer on Linux (https://get.nexte.st/0.9.133/linux), with a cargo install fallback for non-Linux matrix jobs.
• Kept cargo-mutants install explicit and version-pinned.
• Removed fragile \"[]\" string comparison for has_mutation_cargo_packages; this is now derived from validated JSON payload data.
• Extracted the long inline CI selection validator into tools/validate_ci_selection.py and wired workflow to call it, improving readability and auditability.
• Added an explanatory comment next to Python::initialize() in the rate_limiter Rust test to document why it is needed with nextest process isolation.

Validation run locally:

• python3 tools/plugin_catalog.py ci-selection . all | python3 tools/validate_ci_selection.py
• cargo test -p rate_limiter await_async_tuple_parses_successful_result

If you want, I can also add a small follow-up for doctest guard (cargo test --doc) in CI.

[msureshkumar88]

Follow-up Review

Checking whether all four required changes from the previous review are addressed, and whether new issues were introduced.

---

Required Changes — Status

✅ Issue 3: has_mutation_cargo_packages fragile string check

Fully fixed. Logic moved to tools/validate_ci_selection.py which computes bool(mutation_cargo_packages) in Python — proper truthiness check, not string comparison against "[]".

✅ Issue 4: Python validation one-liner extraction

Fully fixed. tools/validate_ci_selection.py is clean, readable, and validates all fields including the new mutation_jobs structure with proper assertions. Much easier to audit.

✅ Issue 2: Python::initialize() justification

Comment added explaining the nextest process-isolation requirement. Other plugins verified: pii_filter already has Python::initialize() in its tests; retry_with_backoff, encoded_exfil_detection, url_reputation, and secrets_detection have no plugin.rs with PyO3 test patterns, so no updates needed there.

⚠️ Issue 1: Binary installer — partially addressed

cargo-nextest on Linux: Fixed — uses curl -LsSf https://get.nexte.st/0.9.133/linux | tar zxf - (fast binary).

cargo-nextest on macOS/Windows (build-test matrix): Still falls back to cargo install cargo-nextest --version 0.9.133 --locked (slow compile from source). The nexte.st service has macOS binaries at https://get.nexte.st/0.9.133/mac-arm and https://get.nexte.st/0.9.133/mac. Windows is harder without taiki-e/install-action (removed in #59). This is a remaining gap for the macOS jobs.

cargo-mutants (mutation-testing job, ubuntu-only): Still cargo install cargo-mutants --version 27.0.0 --locked — no official binary installer script exists for cargo-mutants, so this may be unavoidable. Not blocking, but worth noting.

---

New Issues Introduced

None critical. The implementation is clean. One observation:

cargo-mutants install time on ubuntu: Since the mutation-testing job only runs on ubuntu-latest, and cargo-mutants has GitHub release binaries, cargo-binstall could speed this up if build time becomes a concern. Not a blocker.

---

Summary

Three of four required issues are fully resolved. Issue 1 is partially resolved (Linux binary install works; macOS build-test jobs still compile nextest from source). CI checks show 24/45 passed with several still pending — the passing build-test and coverage jobs confirm the nextest integration works correctly. The validate_ci_selection.py extraction is a clear improvement.

If macOS compile time is acceptable, this is ready to merge pending CI completion.

[msureshkumar88]

🤖 Automated Verification Results for PR #68

Verification Status: ✅ PASSED (6/6 checks)

I've created and executed an automated verification script to confirm that this PR correctly implements cargo-nextest for Rust tests.

Verification Summary

Check	Status	Details
nextest Configuration	✅ PASS	.config/nextest.toml properly configured with v0.9.133, CI & mutants profiles
Plugin Makefiles	✅ PASS	All 6 plugins use cargo nextest run with profile variables
CI Workflow	✅ PASS	Installs nextest (binary on Linux, cargo install on macOS/Windows)
Mutation Testing	✅ PASS	.cargo/mutants.toml configured to use nextest
Coverage Integration	✅ PASS	Uses cargo llvm-cov nextest for coverage
Tooling Setup	✅ PASS	nextest installation verified

Key Findings

✅ Configuration Files

• nextest version 0.9.133 specified
• CI profile: fail-fast = false, failure-output = "immediate-final"
• Mutants profile: fail-fast = true, failure-output = "final"

✅ All Plugin Makefiles Updated

• encoded_exfil_detection, pii_filter, rate_limiter, retry_with_backoff, secrets_detection, url_reputation
• All use $(CARGO) nextest run --profile $(NEXTEST_PROFILE) -p $(CARGO_PACKAGE)
• NEXTEST_PROFILE variable properly defined

✅ CI Workflow Integration

• Fast binary installation on Linux: https://get.nexte.st/0.9.133/linux
• Fallback to cargo install on macOS/Windows
• NEXTEST_PROFILE: ci environment variable set
• Coverage uses cargo llvm-cov nextest

✅ Additional Improvements

• Extracted CI validation logic into tools/validate_ci_selection.py
• Added Python::initialize() to rate_limiter test for nextest process isolation
• Mutation testing integrated with nextest

Verification Artifacts

The following files have been created for verification:

1. verify_pr68_nextest.py - Automated verification script (can be rerun anytime)
2. pr68_verification_report.md - Detailed verification report with full analysis

Recommendation

✅ APPROVE - All verification checks pass. The PR correctly implements cargo-nextest for Rust test execution and is ready for merge.

---

This verification was performed by an automated script that checks configuration files, Makefiles, CI workflows, and tooling setup.

[lucarlig]

I pushed commit 8c1ae17 to branch fix/issue-66-nextest to address the remaining Issue 1 gap by using official macOS nextest binaries:

• Apple Silicon: https://get.nexte.st/0.9.133/mac-arm
• Intel macOS: https://get.nexte.st/0.9.133/mac

This is now wired in:

• ci-rust-python-package
• ci-scaffold-generator

Windows still uses the cargo install fallback.

[msureshkumar88]

Follow-up: Issue 1 Resolved

Commit 8c1ae17 fully addresses the macOS gap.

Verified in diff:

• Apple Silicon: curl -LsSf https://get.nexte.st/0.9.133/mac-arm | tar zxf -
• Intel macOS: curl -LsSf https://get.nexte.st/0.9.133/mac | tar zxf -
• Applied to all 3 install blocks in ci-rust-python-package.yaml and ci-scaffold-generator.yaml
• Windows retains cargo install fallback — acceptable, no official binary installer without taiki-e/install-action

All four required changes from the original review are now fully resolved. Approving.

[msureshkumar88]

All four required changes resolved:

• ✅ Issue 1: Binary installer — macOS now uses official get.nexte.st binaries (arm64 + x86_64); Windows fallback acceptable without taiki-e
• ✅ Issue 2: Python::initialize() comment added; other plugins verified
• ✅ Issue 3: has_mutation_cargo_packages string check replaced with proper Python truthiness
• ✅ Issue 4: Inline validation one-liner extracted to tools/validate_ci_selection.py

Approved.