fix(test-insights): reshoot the three screenshots from the sandbox org - #12695
Merged
Conversation
The Detection, Mitigation and Prevention images were captured against a dashboard logged into an org with access to our private repositories, and they have been served on docs.mergify.com ever since: real internal test paths, a real pull request with its author, number and title, and a private repository name in the selector. These three replace them with captures of `mergify-sandbox`, an org that holds nothing private, so the same pages teach the same thing without publishing anything. The alternative was removing them and leaving the pages imageless, which is what I first proposed. Replacing is better: the pages are about reading a dashboard, and describing a donut in prose is not the same as showing one. ## What each shot shows, and that it is real Nothing here is mocked or posed. The sandbox repository runs a real pytest suite on a schedule through pytest-mergify, with two intentionally flaky tests, and Test Insights has been watching it for a month. - **Detection** — the tests-health donut (93.33% healthy, 6.67% flaky) and the CI Impact chart over its 141 failed executions. - **Mitigation** — quarantine scale and impact (2 tests saved 69 times), and the quarantined-tests list carrying both sources, auto and manual. - **Prevention** — caught-flaky, new-tests and CI-budget metrics, and the new-tests list broken down per pull request with each test's health. Two framing decisions worth stating. All three are cropped to the content pane rather than the whole window: a screenshot with the sidebar in frame goes stale every time a nav item is renamed, even when the documented feature has not moved, and that is how most of our screenshots rotted. Mitigation and Prevention are shot over a one-month window rather than the default week, because the week's numbers are zeros and an empty panel teaches nothing. ## What this does not do The git history keeps the old blobs, and I would not spend a history rewrite on them. What leaked is internal test paths, one pull request title and a private repository name — no credentials, nothing that grants access, no customer data. A rewrite means force-pushing main and invalidating every open pull request and clone, and it still would not delete anything: the blobs stay reachable by SHA until GitHub garbage-collects, forks keep their copies, and the built pages have been crawled. What is worth doing instead is purging the CDN cache for the three asset URLs once this lands. MRGFY-8780 Fixes Mergifyio/ci-bot#373 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012x4fo5P6sFzCsKGZ8DDxLE Change-Id: I98c28b4ca873d58a85b60c4df5e620145c2c10f1
There was a problem hiding this comment.
Copilot wasn't able to review any files in this pull request.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
Merge Protections🟢 All 6 merge protections satisfied — ready to merge. Show 6 satisfied protections🟢 🤖 Continuous Integration
🟢 👀 Review Requirements
🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
🟢 🔎 Reviews
🟢 📕 PR description
🟢 🚦 Auto-queueWhen all merge protections are satisfied, this pull request will be queued automatically. |
jd
marked this pull request as ready for review
September 5, 2026 21:10
sileht
approved these changes
Sep 7, 2026
AlexandreGaubert
approved these changes
Sep 7, 2026
Contributor
Merge Queue Status
This pull request spent 2 minutes 44 seconds in the queue, including 2 minutes 19 seconds running CI. Required conditions to merge
|
63 tasks
mergify
Bot
deleted the
devs/jd/jd/mrgfy-8780-docs-give-the-three-test-insights-pages-a-screenshot-again/reshoot-three-screenshots-sandbox-org--98c28b4c
branch
September 7, 2026 08:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Detection, Mitigation and Prevention images were captured against a
dashboard logged into an org with access to our private repositories, and they
have been served on docs.mergify.com ever since: real internal test paths, a
real pull request with its author, number and title, and a private repository
name in the selector. These three replace them with captures of
mergify-sandbox, an org that holds nothing private, so the same pages teachthe same thing without publishing anything.
The alternative was removing them and leaving the pages imageless, which is
what I first proposed. Replacing is better: the pages are about reading a
dashboard, and describing a donut in prose is not the same as showing one.
What each shot shows, and that it is real
Nothing here is mocked or posed. The sandbox repository runs a real pytest
suite on a schedule through pytest-mergify, with two intentionally flaky tests,
and Test Insights has been watching it for a month.
CI Impact chart over its 141 failed executions.
quarantined-tests list carrying both sources, auto and manual.
new-tests list broken down per pull request with each test's health.
Two framing decisions worth stating. All three are cropped to the content pane
rather than the whole window: a screenshot with the sidebar in frame goes stale
every time a nav item is renamed, even when the documented feature has not
moved, and that is how most of our screenshots rotted. Mitigation and
Prevention are shot over a one-month window rather than the default week,
because the week's numbers are zeros and an empty panel teaches nothing.
What this does not do
The git history keeps the old blobs, and I would not spend a history rewrite on
them. What leaked is internal test paths, one pull request title and a private
repository name — no credentials, nothing that grants access, no customer data.
A rewrite means force-pushing main and invalidating every open pull request and
clone, and it still would not delete anything: the blobs stay reachable by SHA
until GitHub garbage-collects, forks keep their copies, and the built pages have
been crawled. What is worth doing instead is purging the CDN cache for the three
asset URLs once this lands.
MRGFY-8780
Fixes Mergifyio/ci-bot#373
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_012x4fo5P6sFzCsKGZ8DDxLE