Guard CodeQL kernel negatives with an endpoint-observation probe - #137
Merged
Merged
Conversation
Every CodeQL kernel run now evaluates a companion <Language>KernelEndpointProbe.ql alongside its kernel query in the same `database analyze` invocation. The probe mirrors the kernel's own endpoint recognizers (including the shared-extractor file-extension guards) and reports each dfb_source/dfb_sink endpoint the extracted database resolves. Normalization splits the probe's rows back out of the retained SARIF before reconciliation, and a run whose probe never observed both benchmark-controlled endpoints is `inconclusive` on the same terms as JoernEndpointRule::BothMustBeObserved — never a vacuous clean `not-reached`. The modeling matrix runs no probe: an absent declared endpoint is frequently the assertion a modeling negative makes. The probe query is bound into each kernel population's configuration_hash, so committed reports are superseded and the eleven kernel populations need a re-run under a new amendment before the next freeze. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #131.
What
sarif_anchor_outcomereturnsnot-reachedas soon as a SARIF document has zero results, and the kernel queries select flows only — nothing verified that thedfb_source/dfb_sinkendpoints were actually extracted into the database. A recoverable per-file parse error in an interpreted-language extractor (JS/TS/Python/Ruby) that drops the sink expression would therefore yield a vacuous clean true-negative with no diagnostic.This PR adds an endpoint-observation guard on the same terms as
JoernEndpointRule::BothMustBeObserved:<Language>KernelEndpointProbe.ql, rule iddataflowbench/<language>-kernel-endpoint-probe). Each probe mirrors its kernel's own endpoint recognizers exactly — thedfb_source()/dfb_sink(value)contract, the Java kernel's extra endpoint names, and the file-extension guards where C/C++, Kotlin, and the two ECMA populations share an extractor — and reports one row per endpoint the extracted database resolves.database analyzeinvocation evaluates the kernel query and the probe together, so the retained SARIF is the raw evidence for both (the probe output is retained as-is on disk).split_codeql_endpoint_probe), so kernel result sets and retained diagnostics are unchanged by the probe running alongside. A run whose probe never observed both benchmark-controlled endpoints is normalized toinconclusive(unobserved_codeql_endpoint_outcome), nevernot-reached; a document with no analysis run staysrunner-error. The gate sits before flow reconciliation, exactly where Joern applies its endpoint rule.AbsenceIsTheAssertionanalogue) — and the tool-native profile is untouched.configuration_hashnow binds the probe query, and each runner fails fast when the probe file is missing.Tests
Mirrors the existing anti-vacuous tests (
sarif_execution_errors_prevent_clean_negative_interpretation, the Joern endpoint tests):codeql_endpoint_probe_rows_are_split_from_kernel_findingsan_unobserved_codeql_endpoint_prevents_clean_negative_interpretationevery_codeql_kernel_evaluates_an_endpoint_probe_beside_its_query(all eleven populations: probe exists in the kernel query's pack, declares@kind problemand the expected rule id, and is recognized by the runner's split predicate)Verification
cargo fmt --check,cargo test(210 passed),cargo run -- validate(852 cases),cargo run -- validate-reports(82 reports) all pass.codeql query compileunder the pinned CLI 2.26.4 (build6b1e4dee94adb20f90a671f3fc9e04be32eecf65) against the committed pack locks.dataflowbench/javascript-kernel-endpoint-probeand messagesBenchmark source/sink endpoint observed., which is exactly what the splitter keys on.Committed reports need a re-run
The changed normalization semantics and configuration hashes supersede the committed CodeQL kernel evidence. No reports were hand-edited. The eleven CodeQL kernel populations (Java, JavaScript, TypeScript, Python, Kotlin, C#, Go, C, C++, Rust, Ruby) need a re-run under a new amendment before the next freeze, per docs/freeze.md and the amendment conventions in docs/adapters.md. Note the repo-wide amendment sequence currently tops out at A29 with A27–A29 still open (#123, #124, #125) — refetch main before claiming a number.
🤖 Generated with Claude Code