Skip to content

Guard CodeQL kernel negatives with an endpoint-observation probe - #137

Merged
DavidBakerEffendi merged 1 commit into
mainfrom
dave/issue-131-codeql-endpoint-probes
Sep 2, 2026
Merged

Guard CodeQL kernel negatives with an endpoint-observation probe#137
DavidBakerEffendi merged 1 commit into
mainfrom
dave/issue-131-codeql-endpoint-probes

Conversation

@DavidBakerEffendi

Copy link
Copy Markdown
Collaborator

Fixes #131.

What

sarif_anchor_outcome returns not-reached as soon as a SARIF document has zero results, and the kernel queries select flows only — nothing verified that the dfb_source/dfb_sink endpoints were actually extracted into the database. A recoverable per-file parse error in an interpreted-language extractor (JS/TS/Python/Ruby) that drops the sink expression would therefore yield a vacuous clean true-negative with no diagnostic.

This PR adds an endpoint-observation guard on the same terms as JoernEndpointRule::BothMustBeObserved:

  • Eleven companion probe queries, one per kernel population, living beside each kernel query in its own pack (<Language>KernelEndpointProbe.ql, rule id dataflowbench/<language>-kernel-endpoint-probe). Each probe mirrors its kernel's own endpoint recognizers exactly — the dfb_source()/dfb_sink(value) contract, the Java kernel's extra endpoint names, and the file-extension guards where C/C++, Kotlin, and the two ECMA populations share an extractor — and reports one row per endpoint the extracted database resolves.
  • One database analyze invocation evaluates the kernel query and the probe together, so the retained SARIF is the raw evidence for both (the probe output is retained as-is on disk).
  • Normalization splits the probe's rows back out before any finding is reconciled (split_codeql_endpoint_probe), so kernel result sets and retained diagnostics are unchanged by the probe running alongside. A run whose probe never observed both benchmark-controlled endpoints is normalized to inconclusive (unobserved_codeql_endpoint_outcome), never not-reached; a document with no analysis run stays runner-error. The gate sits before flow reconciliation, exactly where Joern applies its endpoint rule.
  • The modeling matrix runs no probe — an absent declared endpoint is frequently the assertion a modeling negative makes (the AbsenceIsTheAssertion analogue) — and the tool-native profile is untouched.
  • Each population's configuration_hash now binds the probe query, and each runner fails fast when the probe file is missing.

Tests

Mirrors the existing anti-vacuous tests (sarif_execution_errors_prevent_clean_negative_interpretation, the Joern endpoint tests):

  • codeql_endpoint_probe_rows_are_split_from_kernel_findings
  • an_unobserved_codeql_endpoint_prevents_clean_negative_interpretation
  • every_codeql_kernel_evaluates_an_endpoint_probe_beside_its_query (all eleven populations: probe exists in the kernel query's pack, declares @kind problem and the expected rule id, and is recognized by the runner's split predicate)

Verification

  • cargo fmt --check, cargo test (210 passed), cargo run -- validate (852 cases), cargo run -- validate-reports (82 reports) all pass.
  • All eleven probe queries compile cleanly with codeql query compile under the pinned CLI 2.26.4 (build 6b1e4dee94adb20f90a671f3fc9e04be32eecf65) against the committed pack locks.
  • End-to-end shape check on a scratch JavaScript database (no committed evidence touched): analyzing kernel + probe in one invocation yields probe rows with rule id dataflowbench/javascript-kernel-endpoint-probe and messages Benchmark source/sink endpoint observed., which is exactly what the splitter keys on.

Committed reports need a re-run

The changed normalization semantics and configuration hashes supersede the committed CodeQL kernel evidence. No reports were hand-edited. The eleven CodeQL kernel populations (Java, JavaScript, TypeScript, Python, Kotlin, C#, Go, C, C++, Rust, Ruby) need a re-run under a new amendment before the next freeze, per docs/freeze.md and the amendment conventions in docs/adapters.md. Note the repo-wide amendment sequence currently tops out at A29 with A27–A29 still open (#123, #124, #125) — refetch main before claiming a number.

🤖 Generated with Claude Code

Every CodeQL kernel run now evaluates a companion
<Language>KernelEndpointProbe.ql alongside its kernel query in the same
`database analyze` invocation. The probe mirrors the kernel's own
endpoint recognizers (including the shared-extractor file-extension
guards) and reports each dfb_source/dfb_sink endpoint the extracted
database resolves. Normalization splits the probe's rows back out of the
retained SARIF before reconciliation, and a run whose probe never
observed both benchmark-controlled endpoints is `inconclusive` on the
same terms as JoernEndpointRule::BothMustBeObserved — never a vacuous
clean `not-reached`. The modeling matrix runs no probe: an absent
declared endpoint is frequently the assertion a modeling negative makes.

The probe query is bound into each kernel population's
configuration_hash, so committed reports are superseded and the eleven
kernel populations need a re-run under a new amendment before the next
freeze.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CodeQL negatives lack an endpoint-observation probe (vacuous clean-TN risk)

1 participant