Skip to content

v2.1.0 — Test Quality Scoring, Language-Aware Recommendations, CI/CD Improvements

Choose a tag to compare

View all tags
@bri-stevenski bri-stevenski released this 21 May 18:42
· 175 commits to main since this release
v2.1.0
7b7cc8c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's new in v2.1.0

Test Quality Scoring (#104)

Every oracle generate run now includes a static-analysis quality score — no extra flags needed.

  • Coverage breadth: test function count + negative/error path detection + parametrize bonus
  • Assertion density: framework-aware assertion count per test (pytest `assert`, playwright `expect`, vitest `expect`, k6 `check`)
  • Flakiness risk: deductions for hardcoded `waitForTimeout`/`sleep` calls, `random.*`, timestamp-dependent assertions
  • Composite 0–100 score with letter grade A–F
  • Surfaced in CLI text output, `--json`, GitHub Action PR comment table, and SARIF `properties`

Language-Aware Framework Selection (#94)

The recommender now reads project metadata (detected from `package.json`, `tsconfig.json`, `requirements.txt`) and filters framework candidates to those matching the project's languages. TypeScript/JavaScript projects route API tests to Playwright; Python projects route to pytest. Falls back to the unfiltered order if no language match is detected.

Explicit Framework Hints (#98)

Prompts can now name a framework directly — `oracle generate "write vitest tests for …"` — and the classifier honours it, bypassing the automatic recommender. Useful when you want to override the default for a specific generation.

Commit-to-PR Workflow (#96)

New `oracle-commit-test.yml` workflow: triggered manually via `workflow_dispatch`, it checks out a PR branch, re-runs Oracle, commits the generated file, and posts a follow-up comment. The PR comment now includes a "Add to your regression suite" section linking directly to this workflow.

Environment Utilities (#99)

New CLI subcommands and core modules for first-run configuration:

  • `oracle env-setup` — interactive wizard
  • `oracle env-setup-legacy` — non-interactive flow for scripts
  • `agent/core/provider_ping.py` — no-cost API-key validation via `models.list`
  • `agent/core/env_writer.py` — idempotent `.env` merge (refuses to rewrite quoted/multiline values)

Other

  • #97 Spec: skills can bundle executable code via `cli:`/`entry:` fields
  • #100 Runnable example prompts for all four frameworks (Playwright, pytest, Vitest, k6)
  • #101 `scripts/extract_python_docstrings.py` — materialises module docstrings as harness knowledge
  • #95 Action: pass extracted JSON values via temp files (fixes empty PR comment fields)
  • #92/#93 Classifier: HTTP verb/path signals route to API tests; CI test-runner fix

Stats

  • 437 tests passing (up from 361 at v2.0.0)
  • All CI checks green: Architecture Enforcer, Security Reviewer, Harness, Docs Lint, Quality & Integrity

Upgrading

pip install --upgrade git+https://github.com/bri-stevenski/oracle-test-ai-agent@v2.1.0
# or
pipx install git+https://github.com/bri-stevenski/oracle-test-ai-agent@v2.1.0

GitHub Action users: bump your pin to @v2.1.0 or move the floating @v2 tag.

Assets 2
Loading