Flaky Unstable Test Cases

Flaky / Unstable Test Cases (modernized) — Issue #1954

Screen: gui/templates/results/flakyTests.html BFF: api/flakytests/index.php ASIDE: Reports → Flaky / Unstable Test Cases (new entry, $TLS_link_report_flaky_tests, present in all 19 server locales). Launcher: Metrics & Reports → Flaky / Unstable Test Cases (flaky_tests in api/resultsnav/index.php).

Purpose

TestLink 1.9.20 showed only the current status of a test case — the result of its last execution. It never showed the pattern of its execution history, so a test case that fails once in a while (a flaky one) stayed invisible: the plan looked green, because the last run was green.

This screen closes that gap. For every test-case version linked to a test plan it computes the verdict of the whole execution history and shows the executions behind it.

Screen features

• Teal Dashio header with the subtitle inconsistent execution history + the client-side TLi18n locale switcher (10 locales) in the footer.
• Toolbar: Refresh, Export CSV, the Build, Platform and Min. executions filters, Apply and Close.
• Report context panel: test project, test plan, build, platform, min. executions.
• Verdict banner: the number of flaky versions among the executed ones, an "all clean" variant and a "nothing was executed" variant.
• Eight KPI cards: Linked versions, Executed, Flaky, Stable, Unproven, Not run, Flaky rate %, Avg. executions.
• DataTable (13 columns, searchable, sortable): test case, external id, suite path, verdict, executions, passed, failed, blocked, flips, last status, last run, builds and the History action.
• Execution-history modal: date, status, build, platform, tester, duration and notes of every execution of the version, in chronological order. The modal moves the focus inside itself, traps Tab, makes the background inert and gives the focus back to the row button when it closes.
• CSV export of the current result set (UTF-8 BOM, ; separator, a formula-injection guard).
• State cards for 400 / 401 / 403 / 404 / 405, each keeping the machine code of the BFF visible.

The model

• Unit of the report — the test-case VERSION (tcversions.id, which is also the nodes_hierarchy node id of the version). A version linked on two platforms is one row, not two.
• Every execution counts — the whole history of the analysed scope, not only the last one.
• flaky — at least one passed AND at least one failed execution, over at least min_exec executions (default 2).
• stable — all the executions of the scope have the same status (a version with only blocked executions included).
• unproven — fewer executions than min_exec: the pattern cannot be judged yet.
• not run — linked to the plan but never executed in the scope.
• A blocked execution is an execution, never a failure — it never makes a version flaky.
• Flips — the number of status changes between two consecutive executions (window function LAG, ordered by execution_ts, id — the same order as the history modal).
• Last status / Last run — taken from the same (chronologically last) execution.

Filters

• Build — the executions of that build.
• Platform — narrows the universe as well: the platform is part of the assignment in TestLink (testplan_tcversions.platform_id), so only the versions linked on that platform are listed at all.
• Min. executions (1..100, default 2) — the threshold of the unproven verdict.

The API

Route	Answer
GET ?action=init&tproject_id=N[&tplan_id=M]	context + the builds / platforms of the project
GET ?action=report&tplan_id=M[&tproject_id=P][&build_id=B][&platform_id=PL][&min_exec=K]	the summary + the rows
GET ?action=history&tplan_id=M&tcversion_id=V[...]	the executions of one version

• Session based, read only (no write verb at all), bffSameOriginGuard + bffEnforceSession.
• The right testplan_metrics is checked on the owning project of the plan — the project the plan really belongs to, not the one asserted by the caller.
• Machine codes: 400 invalid_request, 401 not_authenticated / session_expired, 403 no_right, 404 tplan_not_found / project_mismatch / tcase_not_found, 405 method_not_allowed.

Bugs found and fixed while testing

1. The suite path was empty — the recursive walk started from the version node instead of the test-case node, so every row showed - as the suite.
2. The report was unreachable from the Reports launcher — the relative url of the cfg entry was resolved against the launcher page and answered 404.
3. The history modal rendered raw i18n keys (FLAKY.COLBUILD) — two keys were missing from the bundles.
4. A stale export — after a failed reload the previous result set was still exportable.
5. The modal had aria-modal without any focus management.
6. The Close button did nothing — window.close() is a no-op when the report is opened as a normal link.
7. Contrast — the teal header, the buttons and the verdict badges failed WCAG AA; Lighthouse now reports Accessibility 100 / 0 failed audits.

The Reports launcher (fixed in #1955)

The launcher rewrote the href of a report only for the keys listed in $legacy2modern; a modernized report missing from that map kept the relative cfg url, the browser resolved it against the launcher page and answered 404. The fix roots every modern gui/templates/... url and attaches the shared tplan_id/tproject_id context — 8 already-modernized reports (Release Quality Gates, Performance Test Reports, Automation Test Reports, CI/CD Result Publishing, Defect Intelligence, Test Strategy Report, Test Cases not run on any Platform, Metrics & Trends) are reachable again.

Screenshots