fix(experiment): make dataApiRevokeOnCreateDefault flag reads shape-agnostic#46289
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI (base), Organization UI (inherited) Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughAdds exported isInDataApiRevokeTreatment(flag) to interpret boolean or multivariate string variants of the PostHog flag, updates hooks/pages to use the helper and widened flag types, extends tests for multivariate variants, and widens telemetry event properties to accept boolean|string. ChangesData API revoke-on-create multivariate flag support
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…gnostic
Prep for migrating data-api-revoke-on-create-default from a boolean
rollout to a multivariate experiment with named variants (GROWTH-877).
The flag's current boolean shape can't distinguish in-experiment control
from non-cohort users — both return false — so the analysis dilutes any
control-arm signal across the ~95% of users who never entered the
experiment. The fix is reconfiguring the PostHog flag as multivariate
with control/test string variants, but doing that without coordinating
a frontend deploy would break the experiment in opposite ways depending
on which side ships first.
This PR removes that coordination requirement. Introduces an exported
`isInDataApiRevokeTreatment(flag)` helper that returns true iff the
caller is in the treatment arm, accepting both the current boolean
shape (true) and the future multivariate shape ('test'). Routes the
hook, both page surfaces (/new/[slug] and the Vercel deploy button),
and the convergence gate in the exposure hook through the helper. No
behavioral change on the boolean flag — the existing rollout keeps
working exactly as it does today. After the PostHog config flip lands,
the same code picks up the variant string automatically.
Also pre-fixes a latent bug that would have shipped on day one of the
multivariate switchover: the previous code computed the form's expected
default as `!flag`, which evaluates to false for *any* truthy string —
including 'control'. That would have set control-arm users to the
revoke checkbox by default. The helper-based derivation correctly
treats 'control' as not-treatment and leaves the legacy default.
Telemetry contract for project_creation_default_privileges_exposed and
project_creation_simple_version_submitted broadened to accept either
boolean or string for dataApiRevokeOnCreateDefaultEnabled. The payload
keeps sending the raw flag value (not the derived boolean) so the BQ
analysis can finally distinguish control from non-cohort once the
multivariate config lands.
GROWTH-877
seanoliver
force-pushed
the
sean/growth-877-shape-agnostic-flag-helper
branch
from
ac97469 to
81d066a
Compare
🎭 Playwright Test Results
Details
Flaky testsFeatures › assistant.spec.ts › AI Assistant › Can send a message to the assistant and receive a response Skipped testsFeatures › auth-users.spec.ts › should show web3 users as enabled when the matching web3 provider is enabled |
Braintrust eval report
|
3 participants
I have read the CONTRIBUTING.md file.
YES
What kind of change does this PR introduce?
Bug fix / experiment prep.
What is the current behavior?
The
data-api-revoke-on-create-defaultPostHog flag is configured as a boolean rollout, not a multivariate experiment. Confirmed via Hex on 2026-05-22: bucketing math reconciles (treatment share 2.56%, expected 2.5%), but in-experiment control users are indistinguishable from non-cohort users in the event data — both returnfalse. Any control-arm signal is diluted across the ~95% of users who never entered the experiment.The fix is reconfiguring the flag as multivariate with
control/teststring variants (GROWTH-877). But the boolean and string return shapes can't coexist cleanly — whichever side ships first breaks the experiment in the gap:'test'; strings are truthy, soif (flag)evaluates true for both arms.'test'while PostHog still returns booleantrue; nobody gets treatment.What is the new behavior?
Exported helper that accepts both shapes:
Routes the enabled-hook, both page surfaces (
/new/[slug]and the Vercel deploy button), and the convergence gate in the exposure hook through it. No behavioral change on the boolean flag today. After the PostHog config flip lands, the same code picks up the variant string automatically — no coordinated deploy needed.Pre-fixes a latent bug along the way: the previous code computed the form's expected default as
!flag, which evaluates false for any truthy string including'control'. That would have defaulted control-arm users to the revoke checkbox on day one of the migration. The helper-based derivation correctly treats'control'as not-treatment.Telemetry contract for
project_creation_default_privileges_exposedandproject_creation_simple_version_submittedwidened frombooleantoboolean | string. Payload keeps sending the raw flag value (not the derived boolean) so BQ analysis can distinguish control from non-cohort once multivariate lands.Testing
pnpm --filter studio exec vitest run hooks/misc/__tests__/useDataApiRevokeOnCreateDefault.test.ts— 24/24 pass (16 existing + 8 new covering both variant strings)'test'(treatment, expected default false) and'control'(not treatment, expected default true) — these document the latent bug fixAdditional context
Migration sequence after this lands is in GROWTH-877. BQ-side query updates to read the variant string are flagged as a followup on the ticket — they don't block this PR.
Summary by CodeRabbit
Refactor
Tests