Skip to content

v3.9.0 — failure intelligence, AI provider choice, trend charts

Latest

Choose a tag to compare

@shossain786 shossain786 released this 21 Oct 14:17
· 30 commits to master since this release
4c88b30
  • Locator keyboard and pointer actions: press(Keys...), focus(), doubleClick(), rightClick() — send keys without clearing the value, focus without clicking, or perform native double/context clicks directly on a locator. Each action re-resolves on every poll and retries stale or not-yet-present elements until timeouts.explicit elapses. (#134, #188)

  • HTML report: flaky-test trend line — the dashboard gains a third "Flaky Test Trend" line chart (flaky tests per run) next to the pass-rate and duration charts. (#185)

  • Fix: Cucumber quarantine check no longer silently skipped — checkCucumberQuarantine threw when the framework context was not yet initialised and the error was swallowed, so quarantined scenarios ran instead of being skipped. It now guards on initialisation. (#186)

  • Fix: Gemini default model — ai.provider: gemini now defaults to the gemini-flash-latest alias; the previously pinned gemini-2.5-flash returned 404 for new API keys. (#171)

  • Dependency updates — selenium-java 4.51.0, cucumber-testng 8.0.4, cucumber-junit-platform-engine, jackson-databind, mockito-core 5.24.0, jakarta.mail 2.0.2.

  • HTML report: tab and failure fixes — the report always opens on the Dashboard (it no longer restores the last-open tab), failure details start collapsed, and the Failures tab no longer goes empty after you expand or collapse a class in Test Cases (both tabs shared one group id).

  • Failure Intelligence: high-level triage and suggested actions — each failure is classified as Product bug, Test bug, Environment, Flaky or Unknown (flaky keeps its cause, e.g. Flaky (Locator not found)), with a rule-based suggested action. Shown in the HTML report, as failureType / suggestedAction in the metrics JSON, and in the JUnit XML; opt-in AI analysis uses it as a baseline. No API key needed. (#138)

  • AI provider is configurable — ai.provider selects the backend for failure analysis and the run summary: anthropic (default, unchanged), gemini, or openai (any OpenAI-compatible /chat/completions endpoint — OpenAI, Azure, Groq, local Ollama via ai.baseUrl). ai.model now defaults per provider. Existing configs keep working with no change.

  • AI run summary (opt-in) — ai.runSummary: true adds one short paragraph to the HTML dashboard summarising the run's failures: which same-cause groups account for most of them, which look like product bugs versus test or environment problems, and what to check first. One API call per run, independent of ai.failureAnalysis; only grouped error messages and test names are sent (see the data note in the HTML report docs). Absent when disabled, when nothing failed, or on any error; the suite result is never affected. The text is exported as aiRunSummary in the metrics JSON.

  • HTML report: pass-rate and duration trend charts — the dashboard now shows two line charts under the donut and percentile row, built from the last 20 runs in the metrics history (the same flakiness.historyDir / -Dseleniumboot.history.dir location the Flakiness Radar reads). They appear once there are at least two runs of history and stay hidden otherwise. (#160)

  • Cucumber on the JUnit Platform: quarantine and suite-lifecycle parity — quarantined scenarios run under cucumber-junit-platform-engine are now reported as aborted instead of failed (TestAbortedException; TestNG still gets SkipException). SeleniumBootLauncherListener now also clears the API health cache, loads pre-conditions and starts test-management reporting at test-plan start, matching the TestNG path, and the suite-start work is idempotent. docs-site/docs/cucumber.md documents the JUnit Platform runner as the recommended option. (#155, #158)

  • BaseCucumberTest deprecated — @Deprecated(since = "3.9.0"), because AbstractTestNGCucumberTests is deprecated for removal upstream in Cucumber 8.x. Nothing changes at runtime and existing TestNG Cucumber suites keep working; subclasses now get a deprecation warning. Plan to remove it in 4.0.0, once Cucumber drops its TestNG runner, in favour of cucumber-junit-platform-engine (added as an optional dependency). CucumberStepLogger also moves off the deprecated getStepText(). (#155, #157)

  • JUnit Platform: suite-start hooks now fire — ExecutionHook.onSuiteStart() was never called on the JUnit 5 path (only TestNG), so hooks that open a run or start a clock silently did nothing. SeleniumBootLauncherListener now bootstraps the framework and fires onSuiteStart() once per test plan.

  • Failed tests with trace links in GitHub Actions job summary — when running inside GitHub Actions and tracing.enabled: true is on, failed tests are automatically summarized in a Markdown table (test name, error message, and path to the HTML trace file) appended to $GITHUB_STEP_SUMMARY. Triage failures and locate trace paths directly from the Actions job summary without downloading artifact bundles; no-op outside Actions or when all tests pass. (#137)

  • Bounded failure detail record — every failed test now exports one optional-field failureDetail object with the failing locator, top application frame, last successful step, page URL/title, heal events, console errors, retry/history status, and representative same-cause context. The HTML failure panel and opt-in AI prompt reuse this record; raw errors and stack traces remain available. Text/list/URL limits and the exact AI data disclosure are documented. (#142)

  • Richer auto-retrying locator assertions — LocatorAssert adds isFocused() and isEmpty(), plus Pattern overloads for hasText, containsText, and hasValue. Regex failures report the locator and last observed value, and stale elements are re-resolved until timeouts.explicit. (#136)

  • Locator: waitFor(State), allTextContents(), and element screenshot() — waitFor(State) (and waitFor()) waits for an element to reach a target lifecycle state (VISIBLE, HIDDEN, ATTACHED, DETACHED), retrying up to timeouts.explicit. allTextContents() returns the text content of all matched elements in a single pass without waiting if empty, re-resolving if an element goes stale during read. screenshot() returns the element's PNG bytes, and screenshot(Path) writes them to a file path; both terminal actions re-resolve on every poll to survive stale elements. (#135)

  • flakiness.historyDir / -Dseleniumboot.history.dir — metrics history can now live outside target/. Builds that clean target/ or discard the CI workspace between runs had no history, so the Flakiness Radar and the newly/consistently-failing and intermittent statuses stayed empty. Point it at a directory your CI restores and saves (a cache or artifact store); the system property wins over the config. Default unchanged: <reports dir>/metrics-history. (#141)

  • Retry- and history-aware test status — each test now gets a status from this run's retries and the earlier runs in target/metrics-history/: Passed on retry, Newly failing (passed in every earlier run in the window), Consistently failing (failed in the two or more earlier runs) or Intermittent (mixed earlier results). A "Test status" card sits next to the Flakiness Radar, the status shows in the failure detail, and newly failing tests are listed first in the Failures tab. The metrics JSON gains testStatus. Radar thresholds and failOnHighFlakiness are unchanged. (#141)

  • Failure grouping — the report's Failures tab now opens with a "N tests, same cause" panel: failed tests are grouped by a signature of category, exception type, normalised message (ids, numbers and quoted values stripped) and top application stack frame. The flat per-class list is unchanged below it. With ai.failureAnalysis on, one Claude call is made per group; the other tests in the group reuse its analysis. The metrics JSON gains failureSignature. (#140)

  • Failure classification — every failed test is tagged with a category (LOCATOR_NOT_FOUND, ELEMENT_NOT_INTERACTABLE, STALE_ELEMENT, TIMEOUT, ASSERTION_MISMATCH, NAVIGATION_ERROR, BROWSER_CRASH, or UNKNOWN) and the evidence for it, from the exception class chain and message alone. No AI, no API key, no network. Shown as a badge in the HTML report's failure panel, as failureCategory / failureEvidence in the metrics JSON, and as the type of <failure> in the JUnit XML. The raw error and stack trace are unchanged. (#139)