Skip to content

Add production-grade QA analysis and reporting - #1

Merged
declue merged 1 commit into
mainfrom
agent/qa-analysis-reporting
Jul 14, 2026
Merged

Add production-grade QA analysis and reporting#1
declue merged 1 commit into
mainfrom
agent/qa-analysis-reporting

Conversation

@declue

@declue declue commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

What changed

  • add standardized QA scoring, stable evidence-backed findings, portable static test-code analysis, slow-test and coverage-risk hotspots
  • add baseline regression, multi-run trend/flaky analysis, and Git changed-line coverage maps and gates
  • add HTML insights, coverage tree/treemap, risk and duration heatmaps, and safer Markdown/source rendering
  • harden runner artifact freshness, per-language result completeness, doctor capability checks, language attribution, and Maven/Gradle/.NET defaults
  • wire repository smoke TC and production gates into CI and documentation

Why

The tool previously normalized tests and coverage but did not provide a standard diagnostic model, repeated-run evidence, changed-code risk, or production-grade protection against stale artifacts and untrusted report roots. These changes make the output actionable for QA, CI, and AI-agent consumers across supported ecosystems.

Impact

Users get schema test-cli/report@2, richer static and runtime diagnostics, regression/flaky/change gates, and a self-contained report site with multiple coverage and quality views. Real executions now reject missing per-language test results and cannot silently reuse unchanged artifacts.

Validation

  • make lint test
  • go test -race -count=1 ./...
  • go mod verify
  • actionlint .github/workflows/ci.yml
  • shellcheck -S warning scripts/*.sh and bash -n scripts/*.sh
  • six-platform release build (linux/darwin/windows, amd64/arm64)
  • self-run: 100/100 tests, 82.8% line coverage, 82.6% changed-line coverage, quality 83/B, low risk

@declue
declue marked this pull request as ready for review July 14, 2026 12:43
@declue
declue merged commit 8c46fff into main Jul 14, 2026
3 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b0264ebe9c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

matches := 0
for _, file := range files {
candidate := normalizedChangePath(file.Path)
if strings.HasSuffix(candidate, "/"+wanted) || strings.HasSuffix(wanted, "/"+candidate) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid matching shorter coverage paths to changed files

When a changed file is packages/api/src/service.go and the report only has coverage for src/service.go, this reverse-suffix condition treats that unrelated shorter path as the changed file's coverage. In that case --fail-diff-coverage can pass using hits from another file instead of reporting missing evidence for the changed file; the import-prefix case only needs the coverage path to end with the repo-relative path, or else stronger disambiguation is needed.

Useful? React with 👍 / 👎.

Comment thread internal/cli/run.go
Comment on lines +155 to +156
if failQuality == 0 {
failQuality = cfg.FailQuality

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor an explicit zero quality threshold

When .test-cli.json sets failQuality (as this repo now does), passing --fail-quality 0 is parsed as the same zero default and this block restores cfg.FailQuality. Since gateExitCode treats 0 as disabled, users cannot temporarily disable the configured quality gate from the CLI; track whether the flag was provided instead of using zero as both “unset” and “disabled.”

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant