Add Quarantine recipes companion page covering 4 clustered gaps#59
Add Quarantine recipes companion page covering 4 clustered gaps#59samgutentag wants to merge 1 commit into
Conversation
…onal, cross-repo, fix verification Sourced from customer feedback mining — bundle of 4 adjacent quarantine clusters (quarantine-automation-behavior, conditional-quarantine-by-pr-or-branch, quarantine-history-and-events-clarity, cross-repo-quarantine-and-private-fork) all targeting the quarantining surface. New companion page keeps the main quarantining index focused on semantics while covering recipe-style content in one place.
samgutentag
added
the
needs review
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
|
Code verification (2026-05-21): 3 confirmed / 3 contradicted / 0 ambiguous / 1 unverifiable
Summary of items the author should resolve before publishing:
Source #1 — Propagation timing "up to a minute" (unverifiable)File: async setTestQuarantiningOverride({
runInfoId,
repoId,
userId,
quarantiningOverride,
annotation,
}: {
runInfoId: string;
repoId: string;
userId: string;
quarantiningOverride: TestQuarantiningOverride;
annotation?: string;
}) {
// ... (read previous override)
const [{ testCaseV2 }, quarantineAuditLog] = await this.prismaFlaky.$transaction([
this.prismaFlaky.runInfoMeta.upsert({
create: { runInfoId, quarantiningOverride, repoId },
update: { quarantiningOverride, repoId },
where: { runInfoId_repoId: { runInfoId, repoId } },
select: { testCaseV2: true },
}),
// ... (audit log)
]);Reasoning: The override write is a synchronous Prisma upsert against the primary database. The CLI's Source #2 — Recovery threshold setting and Settings path (contradicted)<div className="space-y-1.5">
<div className="flex items-center justify-between gap-4">
<div>
<Label>Resolution period</Label>
<p className="text-xs text-muted-foreground mt-0.5">
How many days without pass-on-retry behavior before the monitor
resolves. Shorter = faster resolution, longer = more confidence.
</p><div>
<h1 className="text-2xl font-bold">Monitors</h1>
<p className="text-muted-foreground mt-1">
Configure how tests are monitored and classified for {repoName}Reasoning: The user-visible label on the stepper is "Resolution period", not "recovery threshold". The underlying GraphQL field is Source #3 — Quarantined: Only filter on Test History (confirmed)const QUARANTINED_OPTIONS: { value: QuarantinedFilter; label: string }[] = [
{ value: "include", label: "Include" },
{ value: "exclude", label: "Exclude" },
{ value: "only", label: "Only" },
];Reasoning: The Test History view's Quarantined filter is a toggle group with three options: Include, Exclude, Only. Setting it to Only displays exactly the runs where Trunk overrode the exit code due to quarantine state. This matches the recipe in the docs page. The component is mounted via the Test History tab ( Source #4 — Action uploader `repo-url:` input (contradicted)File: gh-repo-url:
description: The URL of the GitHub repo
default: ${{ github.event.pull_request.head.repo.html_url }}File: prTitle: core.getInput("pr-title"),
ghRepoUrl: core.getInput("gh-repo-url"),
ghRepoHeadSha: core.getInput("gh-repo-head-sha"),
// ...
dryRun: parseBoolean(core.getInput("dry-run")),File: #[arg(
long,
env = constants::TRUNK_REPO_URL_ENV,
help = "Override the repository URL (normally from git config remote.origin.url)."
)]
pub repo_url: Option<String>,Reasoning: The CLI flag Source #5 — Events tab Quarantine vs Flake Detection scoping (confirmed with label nits)const FILTER_OPTIONS = [
{ value: AuditEventCategory.All, label: "All" },
{ value: AuditEventCategory.FlakeDetection, label: "Monitors" },
{ value: AuditEventCategory.Quarantine, label: "Quarantine" },
{ value: AuditEventCategory.Ticketing, label: "Ticketing" },
] as const;async function fetchQuarantineEvents({
repoId,
testCaseId,
pageSize,
}: { /* ... */ }): Promise<{ events: QuarantineAuditTrailEvent[]; totalCount: number }> {
try {
const prismaFlaky = await prisma();
const [rows, totalCount] = await Promise.all([
prismaFlaky.flaky_tests_audit_log.findMany({
where: { repoId, testCaseId },
orderBy: { createdAt: "desc" as const },
take: pageSize,
}),
// ...
]);
const events = rows.map(
({ id, createdAt, description, userId, annotation }) => ({
id,
kind: "QUARANTINE" as const,
// ...
}),
);Reasoning: The Quarantine filter sources from Source #6 — No bulk "Never Quarantine" API endpoint (confirmed)File: The complete list of public quarantine-related routes in // Line 2244
path: "/flaky-tests/list-quarantined-tests",
// Line 2991
router.post("/metrics/getQuarantineConfig", async (req, res) => { /* ... */ });Reasoning: Of the wrapped public-API routes in Source #7 — Broken tests do not auto-quarantine (contradicted in docstring, confirmed in execution)File: quarantined: z.boolean().openapi({
description: `Whether the test is quarantined.
This is \`true\` when quarantining is enabled for the repo and either of the following applies:
- The quarantine override is set to \`ALWAYS_QUARANTINE\` for this test
- Automatic quarantining is enabled for the repo, and this test's status is either \`flaky\` or \`broken\`
If this is \`true\`, the next test run will be marked as passed even if the test run conclusion is
failed.`,
}),export const getFlakyTestCaseIdsFromClickHouse = async (
client: ClickHouseClient,
repoId: string,
excludeTestIds: string[] = [],
lookbackDays: number = QUARANTINED_TESTS_LOOKBACK_DAYS,
): Promise<FlakyTestCaseWithUpdatedAt[]> => {
// ...
const result = await client.query({
query: `SELECT tcs.test_case_id, tcs.updated_at FROM ... WHERE tcs.repo_id = {repoId:String} AND tcs.status = {status:String}${excludeClause} ORDER BY tcs.updated_at DESC, tcs.test_case_id ASC`,
query_params:
excludeTestIds.length > 0
? { repoId, status: "FLAKY", excludeIds: excludeTestIds, lookbackDays }
: { repoId, status: "FLAKY", lookbackDays },
format: "JSONEachRow",
});/**
* Get all quarantined test ids ordered by the last status updated at.
*
* 1. get quarantine overrides
* 2. get auto-quarantined tests (flaky, non-never-quarantine) from ClickHouse
* 3. combine always-quarantine and auto-quarantined tests and sort
*/
async getAllQuarantinedTestIdsOrderedByLastStatusUpdatedAtDesc({
repoId,
autoQuarantineEnabled,
}: { /* ... */ }) {
// ...
if (autoQuarantineEnabled) {
const autoQuarantinedTestsFromClickHouse = await getFlakyTestCaseIdsFromClickHouse(
this.nativeClickHouseClientFlaky,
repoId,
quarantineOverridesIds,Reasoning: This is the most interesting finding. There are two sources telling different stories. The public-API description string ( The docs page's claim ("broken tests never auto-quarantine") matches the execution path, which is what customers actually experience. So the docs page is correct about product behavior. The public-api schema description is the outlier and should likely be tightened to drop the "or broken" clause, since it does not reflect the actual implementation. Worth raising with the flaky-tests team separately — this is a real product-truth discrepancy that the docs change happens to surface. For PR #59 itself, no change needed; the claim aligns with reality. |
samgutentag
added
the
needs eng review
1 participant
Summary
flaky-tests/quarantining/recipes.mdxdocs.jsonnav entry added under the existing Quarantining group, slotted ahead ofquarantine-service-availabilityquarantining/index.mdx("Recipes and operational patterns" pointing at the new page)Why
Sourced from customer feedback mining — bundle of 4 adjacent quarantine clusters (26 pairs / 8+ distinct customers total) all targeting the quarantining surface. The existing
quarantining/index.mdxis already long and doesn't address the recipe-style questions customers consistently ask. Splitting recipe content into a companion page keeps the index page focused on what quarantining is and how to enable it, and gives operational answers a single home.Items flagged for review
flaky-tests/quarantining/recipes.mdx(the actual on-disk structure isflaky-tests/quarantining/, notflaky-tests/management/quarantining/as the cluster files suggested). Verify the nav placement (currently abovequarantine-service-availability) is the right order.Settings > Repositories > repo > Flaky Tests > Pass-on-Retry Monitor) is current.--repo-url+--dry-run=true(CLI) andrepo-url:+dry-run: true(action). Verify both flag names against the current CLI and uploader action.Customer signal
Cluster bundle: 4 quarantine-adjacent clusters consolidated for shared IA.
quarantine-automation-behavior(16 pairs / 8 customers, verdict partial)conditional-quarantine-by-pr-or-branch(3 pairs / 1 customer, verdict missing)quarantine-history-and-events-clarity(4 pairs / 3 customers, verdict partial)cross-repo-quarantine-and-private-fork(3 pairs / 1 customer, verdict partial)Total: 26 pairs across 8+ distinct customers.
Representative permalinks (full lists in each cluster file at
findings/clusters/<id>.json):--repo-url+--dry-run=truepattern