feat(v1.8): real HTTP probe runner + release-gate strict findings#47
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0d96a061f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (profile.require_deterministic_replay && findingsCount > 0) { | ||
| reasons.push( | ||
| `profile "${profileKey}" requires deterministic replay; failing run because ${findingsCount} finding(s) were emitted`, | ||
| ); |
There was a problem hiding this comment.
This unconditional require_deterministic_replay failure now triggers on any finding even when scenarios were not executed by a real probe backend (e.g., projects without sut.base_url, or packs using non-HTTP probe kinds). In the current repo, default non-HTTP packs (packs/web-ui Playwright probes, packs/llm-agent llm_eval) can still produce synthetic/unsupported outcomes, so release-gate can fail due to runner limitations rather than an actual SUT regression. The strict finding gate should only apply when probe execution for the run is backed by a real implementation (or when synthetic/unsupported probe results are absent).
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR introduces a real HTTP probe runner in @aqa/runner, wires aqa run to use it when project.sut.base_url is configured, and restores strict release-gate semantics by failing deterministic-replay profiles when findings are emitted.
Changes:
- Add
makeHttpProbeRunner()(fetch-based execution, timeout, base URL resolution, response body parsing) and export it from@aqa/runner. - Use
project.sut.base_urlto enable real HTTP probes inaqa run. - Re-enable strict
require_deterministic_replaybehavior (findings → run failure) and update tests accordingly.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/runner/src/run.ts | Adds makeHttpProbeRunner() implementation. |
| packages/runner/src/index.ts | Exports the new HTTP probe runner API. |
| packages/runner/test/run.test.ts | Adds tests for HTTP runner URL resolution and kind rejection. |
| packages/kit/src/commands/run.ts | Wires probe runner from project.sut.base_url and re-enables strict release-gate failure on findings. |
| packages/kit/test/run-cmd.test.ts | Updates release-gate test to expect ok=false when findings are emitted. |
| ? (withCfg.headers as Record<string, string>) | ||
| : {}; | ||
| const body = | ||
| withCfg.body === undefined | ||
| ? undefined | ||
| : typeof withCfg.body === 'string' | ||
| ? withCfg.body | ||
| : JSON.stringify(withCfg.body); |
| const url = /^https?:\/\//i.test(rawUrl) | ||
| ? rawUrl | ||
| : `${base}${rawUrl.startsWith('/') ? '' : '/'}${rawUrl}`; |
| const probeRunner = project.sut.base_url | ||
| ? makeHttpProbeRunner({ baseUrl: project.sut.base_url }) | ||
| : undefined; |
| if (profile.require_deterministic_replay && findingsCount > 0) { | ||
| reasons.push( | ||
| `profile "${profileKey}" requires deterministic replay; failing run because ${findingsCount} finding(s) were emitted`, | ||
| ); | ||
| } |
Summary
makeHttpProbeRunnerin@aqa/runner(fetch-based HTTP probe execution with timeout, relative URL resolution from base URL, JSON/text body handling)aqa runto use real HTTP probes viaproject.sut.base_urlwhen availablerelease-gatebehavior: profiles withrequire_deterministic_replay=truenow fail when findings are emittedValidation