fix(joblib): fail closed on inconclusive warning scans

[mldangelo-oai]

Summary

Keep Joblib scans failed closed whenever NumPy-wrapper validation is inconclusive, regardless of what the embedded pickle scanner reports.

The defect

JoblibScanner.scan() failed an inconclusive scan only when there were no WARNING/CRITICAL findings:

if scan_outcome == INCONCLUSIVE and not has_security_findings and not has_trusted_incomplete_tail:
    result.finish(success=False)
else:
    result.finish(success=not result.has_errors)

That conflates "we found something" with "we finished the analysis". When findings exist the scanner takes the else branch and reports success=True as long as no finding is CRITICAL — even though scan_outcome=inconclusive and analysis_incomplete=true.

Joblib was the outlier. Roughly a dozen scanners already use the unconditional form this PR adopts — lightgbm, catboost, coreml, rknn, r_serialized, paddle, xgboost, skops, flax_msgpack, llamafile, pytorch_zip, torchserve_mar all finish with scan_outcome != INCONCLUSIVE and not has_errors. This change brings joblib in line, keeping only the joblib-specific trusted_incomplete_tail exemption.

Correction to the original justification

This PR was originally filed as the fix for a recurring Windows CI failure of test_scan_fails_closed_on_protocol1_pickle_in_numpy_wrapper_tail. That claim was wrong and has been removed.

The Windows lane passes on #1780, #1783, #1788, #1790 and #1791 — every one of which contains main without this fix. The test is therefore intermittent, not deterministically failing, and this change is not what makes it pass. The Windows failures on #1792/#1793 were separately traced to stale July-30 runs that predated #1795.

The defect above is real and reproducible on POSIX independently of any platform flake, which is the basis for merging this. It is a correctness fix, not a CI fix.

Why it stays hidden on POSIX in normal runs: ScanResult.finish() computes success and self._merged_children_success, and the embedded pickle scanner usually returns success=False, which pins the parent to False. The joblib-level guard only becomes observable when the embedded result reports success.

Coverage

Two regressions, both failing on main and passing here:

• test_scan_fails_closed_on_invalid_numpy_wrapper_with_origin_warning — stubs the embedded scanner to isolate the guard.
• test_scan_fails_closed_when_embedded_pickle_reports_success — keeps the real pickle scanner, real payload parsing, and real metadata composition, forcing only the embedded verdict.

Demonstrated behaviour change with the embedded verdict forced to success:

	success	scan_outcome	trusted_incomplete_tail
main	True	inconclusive	absent
this PR	False	inconclusive	absent

Validation

• tests/scanners/test_joblib_scanner_codecs.py, test_joblib_scanner.py, test_pickle_scanner.py: 472 passed.
• Ruff lint + format clean; mypy clean on both changed files.

[mldangelo-oai]

@codex review

[Copilot]

Pull request overview

This PR fixes a fail-open behavior in the Joblib scanner: when Joblib NumPy-wrapper validation is inconclusive, the scan now fails closed even if the embedded pickle analysis reports warning-level findings (e.g., origin verification warnings). This aligns the scanner’s success semantics with the project’s fail-closed expectations for incomplete analysis and adds a deterministic regression for a Windows-observed failure mode.

Changes:

• Update JoblibScanner.scan() to always return success=False for scan_outcome=inconclusive results unless a trusted incomplete tail is explicitly present.
• Add a regression test covering an invalid NumPy wrapper combined with an embedded warning-level finding, ensuring the scan fails closed and is not cacheable.
• Document the fix in the root changelog under Bug Fixes.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
modelaudit/scanners/joblib_scanner.py	Makes inconclusive Joblib scans fail closed regardless of embedded warning findings (unless a trusted incomplete tail is present).
tests/scanners/test_joblib_scanner_codecs.py	Adds a deterministic regression reproducing the prior fail-open path (inconclusive wrapper + embedded WARNING finding).
CHANGELOG.md	Notes the Joblib fail-closed behavior change under Bug Fixes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

Workflow run and artifacts

Performance Benchmarks

Compared 13 shared benchmarks with a regression threshold of 15%.
Status: 0 regressions, 0 improved, 13 stable, 0 new, 0 missing.
Aggregate shared-benchmark median: 4.422s -> 4.360s (-1.4%).

Workload	Benchmark	Target	Size	Files	Baseline	Current	Change	Status
padded-multi-stream-upload	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_padded_multi_stream_upload	multi_stream_padded	4.1 KiB	1	359.2us	335.1us	-6.7%	stable
nested-payload-review	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_base64]	nested_base64	98 B	1	309.5us	290.9us	-6.0%	stable
mixed-model-repository	tests/benchmarks/test_scan_benchmarks.py::test_scan_release_candidate_repository	release-candidate	547.3 KiB	32	671.90ms	640.12ms	-4.7%	stable
direct-malicious-upload	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_direct_malicious_upload	malicious_reduce	52 B	1	224.3us	214.2us	-4.5%	stable
warm-cache-rescan	tests/benchmarks/test_scan_benchmarks.py::test_scan_warm_cached_repository_rescan	release-candidate	547.3 KiB	32	153.08ms	147.07ms	-3.9%	stable
nested-payload-review	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_raw]	nested_raw	78 B	1	283.6us	273.6us	-3.5%	stable
nested-payload-review	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_hex]	nested_hex	130 B	1	311.7us	302.5us	-3.0%	stable
clean-training-checkpoint	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_clean_training_checkpoint	safe_large	278.2 KiB	1	111.63ms	109.57ms	-1.8%	stable
rejected-basic-auth-candidates	tests/benchmarks/test_scan_benchmarks.py::test_rejected_basic_auth_candidates_scan_linearly	-	371.1 KiB	1	2.539s	2.516s	-0.9%	stable
single-checkpoint-preflight	tests/benchmarks/test_scan_benchmarks.py::test_scan_single_checkpoint_before_load	single_checkpoint.pkl	183.0 KiB	1	104.51ms	105.19ms	+0.6%	stable
chunked-upload-stream	tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_chunked_upload_stream	chunked_stream	278.2 KiB	1	114.23ms	113.88ms	-0.3%	stable
duplicate-heavy-registry	tests/benchmarks/test_scan_benchmarks.py::test_scan_duplicate_registry_snapshot	registry-snapshot	915.2 KiB	13	575.35ms	575.81ms	+0.1%	stable
suspicious-pickle-intake	tests/benchmarks/test_scan_benchmarks.py::test_scan_suspicious_pickle_intake	suspicious-intake	183.8 KiB	4	150.71ms	150.71ms	-0.0%	stable

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 713940829d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Cover fail-closed behavior through aggregation

For the malformed-wrapper-with-warning scenario, this regression stops at JoblibScanner.scan, so it does not verify the operational message, aggregate status/verdict, or which exit code wins when an inconclusive result coexists with a warning-level security finding. Exercise the result through scan_model_directory_or_file and assert the intended finding precedence and exit code as well as the existing cache behavior; otherwise an aggregate-level regression could ship while this test remains green.

AGENTS.md reference: AGENTS.md:L137-L137

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Security review completed. No security issues were found in this pull request.

Reviewed commit: 713940829d

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

tests/scanners/test_joblib_scanner_codecs.py:1426

• The test forces an embedded pickle scan to report success=True by directly mutating ScanResult.success and the private _merged_children_success field after finish(). This bypasses ScanResult.finish() invariants (e.g., restored-critical handling) and couples the test to private implementation details. Prefer the public trust_merged_child_failures() API and re-run finish(success=True) to recompute success consistently.

        if not result.success:
            result.success = True
            result._merged_children_success = True