Skip to content

chore(ci): remove the firestore flake probe workflow - #795

Open
tyler-reitz wants to merge 1 commit into
FirebaseExtended:mainfrom
tyler-reitz:chore/remove-flake-probe
Open

chore(ci): remove the firestore flake probe workflow#795
tyler-reitz wants to merge 1 commit into
FirebaseExtended:mainfrom
tyler-reitz:chore/remove-flake-probe

Conversation

@tyler-reitz

Copy link
Copy Markdown
Contributor

Why

.github/workflows/flake-probe.yaml was added in #780 to measure the firestore flake in #776. #776 is now closed, so the probe has done its job.

Verified while the file still exists, since afterwards neither claim can be checked without going back to this branch: it is workflow_dispatch only, and nothing outside the file references it. Removing it therefore changes no scheduled or PR-triggered behaviour, and it cannot orphan a required status check (main requires cla/google, Build, Test Node.js 22 (Ubuntu) and Test Node.js 24 (Ubuntu)).

On #783's removal conditions

They were written when the @grpc/grpc-js override was still the candidate fix, so the checklist asks for the probe to be re-run "if the override lands". It never landed: the override was measured at 5/60 against a 4/60 baseline, Fisher p = 1.0.

The conditions are met by a different route. The fix was #791's timeout ceilings, and the probe is what verified it: 0/60 flakes against an 11/60 baseline (p = 0.0006), with the desync rate statistically unchanged at 13/60 against 19/60. Five iterations show the rescue directly, passing at ~9.8s where the old 1000ms waitFor budget gave up. Full result on #776.

Closes #783

The probe was added in FirebaseExtended#780 as a temporary measurement tool for FirebaseExtended#776, which
is now closed. It has no callers: nothing outside the file referenced it, and
it was workflow_dispatch only, so removing it changes no scheduled or
PR-triggered behaviour.

FirebaseExtended#783's removal conditions were written when the @grpc/grpc-js override was
still the candidate fix. That override was measured and did not work (5/60 vs
4/60, Fisher p = 1.0), so it never landed. The fix was FirebaseExtended#791's timeout
ceilings instead, and the probe verified it: 0/60 flakes against an 11/60
baseline, with the desync rate statistically unchanged.

Closes FirebaseExtended#783

@armando-navarro armando-navarro left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The removal checks out: nothing in the tree or on any other branch references the workflow, and before the artifacts expire I re-derived the run 32061237569 column of your closing table, and the five rescue durations under it, from the raw logs, every number matching to the millisecond.

The override figures are swapped

The body says "the override was measured at 5/60 against a 4/60 baseline". What I found:

  • Your own table on #776 has it the other way: baseline (1.9.16) 5/60, override (1.14.4) 4/60.
  • The conclusion holds in either direction, p = 1.0 both ways, so nothing about the argument changes.
  • The commit message carries the same figures in a sentence whose subject is "That override", so they read override-first there too, and a squash carries that into history.

It may be worth rewording both, or catching the commit-message half in the squash editor at merge.

The raw evidence deletes itself on 2026-08-24

  • The probe uploads artifacts with a 7-day retention, so the 08-11 baseline set is already gone (it lapsed on the 18th) and run 32061237569's two sets expire on the 24th.
  • The runs' job logs keep the headline counts beyond that, but the per-iteration output behind the finer numbers (the rescue durations, the desync lines) lives only in the artifacts.
  • I pulled a copy of the proof run's artifacts while re-deriving the table, so shout if you ever want them.

Attaching them to #776 or a gist before the date would make the record self-supporting, since the workflow that could remake them is the one being deleted.

Optional

  • A dispatch-only workflow has to exist on the default branch to run at all (GitHub's docs), so restoring the probe someday means a PR to main, not a checkout. One line in the #783 close-out naming a commit that carries it (23da94e does) would save that future person the discovery.
  • The fixed-position repro signature you said would go upstream doesn't seem to have been filed on firebase-tools#8654 yet (its only comment is from another affected user), and now is the cheapest time, while the data behind it is fresh.

Approving. The swap is the only thing I'd want cleaned up, and if I've misread which arm was which, point me at it and I'll strike the whole item.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove the firestore flake probe workflow once #776 is resolved

2 participants