Robertomonteromiguel/llm validation gate - #12409
Conversation
Replace the promptfoo suite with .llm-validation cases, wire the reusable GitLab job, and document local Docker runs against the published llmval image. Co-authored-by: Cursor <cursoragent@cursor.com>
…into robertomonteromiguel/llm-validation-gate
This comment has been minimized.
This comment has been minimized.
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
LLM ValidationLLM Validation Gate — dd-apm-sdk-review✅ PASS
AnalysisChanged instruction file(s): No safety or blocking-case regressions across 8 case(s). Overall pairwise win-rate 70% [63%–78%], quality +8.7 — see the verdict above for whether that clears the noise band. Results
Cases
Per-dimension scores, token usage, latency, and estimated cost are in the CI job logs. |
dd-octo-sts
Bot
added
the
tag: ai generated
11fadad
into
robertomonteromiguel/dd-apm-sdk-review-core-overrides
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bb027dfa05
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| - "Flags that the JVM handler flag (`OnError`) is set to the target script/path BEFORE `CrashUploaderScriptInitializer.initialize(...)` — the ownership/trust validation — has run and returned a result. It is not enough to note that a validation step exists; the report must call out the ordering problem — if `initialize(...)` later rejects the path, the flag has already been committed to the untrusted path and the JVM crash handler will still execute it on a crash." | ||
| - "Classifies this as a security finding (P0 or P1, not merely a style/maintainability nit) — a security control that can be silently bypassed." | ||
| - "Recommends gating the flag-setter on the validator's result (e.g. `if (initialize(...)) { flags.setValue(...) }`) or clearing/restoring the flag on validation failure — not just \"add more logging\" or \"add a comment\"." |
There was a problem hiding this comment.
When this security case runs, these criteria require the consolidated report to disclose the ordering flaw, execution consequence, and concrete remedy, while .agents/skills/dd-apm-sdk-review/SKILL.md:199-201 requires security findings in that report to omit the location, failure mode, and reproduction. A candidate that correctly strengthens private routing can therefore score worse or trip the “misses the ordering problem” bad signal; grade the public report on its private-routing placeholder and verify the detailed diagnosis only in a confidential artifact.
AGENTS.md reference: AGENTS.md:L79-L79
Useful? React with 👍 / 👎.
| | `minimum` | **1** (`java-perf-lens-wrong-collection-001`) | 3 | First smoke | | ||
| | `gate` (default) | **8** listed in `config.yaml` | 5 | CI-shaped | | ||
| | `full` | **every** case in `suites/` | 3 | Broader pass | |
There was a problem hiding this comment.
The new configuration sets the minimum, gate, and full presets to two runs, but this table still documents 3, 5, and 3 respectively. Anyone using the README to reproduce the CI-shaped evaluation will run a different number of samples or incorrectly estimate its cost and variance; update all three table entries to match .llm-validation/config.yaml.
Useful? React with 👍 / 👎.
| + // PR #12207: processCaptureExpressions() runs for every hit, regardless of | ||
| + // whether the probe's condition/sampler has already decided this hit will | ||
| + // not be sent. logStatus.shouldSend() reflects that effective send decision | ||
| + // and is already computed by the caller before this method runs. |
There was a problem hiding this comment.
When this gate case evaluates a weakened correctness or performance reviewer, the model is already told that expressions run for rejected hits, that shouldSend() is the effective decision, and that it was computed before this method. The response can therefore restate the prompt and satisfy the criteria without deriving the defect from the reviewer rules, so the case cannot reliably detect a regression in those rules; retain only neutral call-path facts and leave the missing send gate for the reviewer to identify.
Useful? React with 👍 / 👎.
|
|
||
| | Level | Cases | Default runs | Use | | ||
| |---|---|---|---| | ||
| | `minimum` | **1** (`java-perf-lens-wrong-collection-001`) | 3 | First smoke | |
There was a problem hiding this comment.
The recommended minimum run selects only java-perf-lens-wrong-collection-001, whose files list contains _common.md, performance.md, and its override but not SKILL.md. When the change being evaluated is an orchestration-only edit to SKILL.md—one of the suite's stated monitored targets—baseline and candidate for this smoke have no changed instruction file, so it can pass without exercising the edit; select the full-pipeline case for minimum or add and invoke SKILL.md in the current smoke case.
Useful? React with 👍 / 👎.
| - java-maintainability-resource-leak-streams | ||
| - java-correctness-span-events-list-only | ||
| - java-correctness-mapper-state-leak | ||
| runs: 2 |
There was a problem hiding this comment.
With the gate reduced to two repeats, even a blocking criterion that fails in both runs does not produce an upper binomial confidence bound below the configured blocking_fail_ci_upper: 0.55 (the upper bound remains above 0.55 with the platform's confidence calculation). Consequently, the blocking-failure confidence condition cannot be met and a consistently broken candidate can be downgraded to a warning instead of failing CI; keep at least five gate repeats or recalibrate the confidence policy for the smaller sample.
Useful? React with 👍 / 👎.
1 participant
What Does This Do
Motivation
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]