Skip to content

Add generic e2e infrastructure for tier-gated and experimental features - #8314

Draft
hatton wants to merge 11 commits into
masterfrom
e2e-infrastructure
Draft

Add generic e2e infrastructure for tier-gated and experimental features#8314
hatton wants to merge 11 commits into
masterfrom
e2e-infrastructure

Conversation

@hatton

@hatton hatton commented Sep 4, 2026

Copy link
Copy Markdown
Member

Problem

An end-to-end test (the Playwright suite in src/BloomE2E) could not reach three kinds of Bloom behaviour. A feature behind a subscription tier stayed hidden, because the test collection had no subscription. A feature behind an experimental-feature token could only be turned on by writing the shared user.config, which would also turn it on for the developer's own Bloom. And anything that opens a native file or folder chooser hung the run, because no automation can drive that dialog. A branch that needed all three had built them for itself, along with helpers for the Canvas tool, page layout, page size, images and undo; this PR lands the generic parts on their own, with nothing specific to any one feature.

What the PR does

  • Bloom accepts --experimental-features <tokens> together with --e2e. Under --e2e the enabled features are exactly those tokens: they are never saved, and the developer's own saved experiments do not reach the run, so a test that needs a feature off gets it off by not naming it. Saving a feature from such a run leaves the saved setting as it was.
  • A test arms the next file or folder chooser with a path through the e2e/nextFileToChoose route. BloomOpenFileDialog and BloomFolderChooser answer with that path instead of showing a dialog, only under --e2e, once per arming. A missing file or one the filter would refuse throws. This is now the one such hook: the narrower one master had added for the Talking Book import goes through it, and under --e2e the file API no longer remembers a test's folder in the developer's machine-wide settings.
  • Under --e2e, Bloom no longer probes port 5173 for a Vite dev server, and refuses camera and microphone permission requests, which on a fresh browser profile would otherwise show a WebView2 prompt no test can dismiss.
  • The suite's launcher takes subscriptionCode and experimentalFeatures options, logs which Bloom.exe it runs and when that file was built, and refuses to launch a Bloom whose front-end bundle or Bloom.dll is older than its source, naming the newer file. A stale build would otherwise fail a test exactly the way a regression does. Tier-gated specs use the existing test code from SubscriptionTests.cs; the Test branding now stamps a butterfly on every page so such a book is visibly a test book.
  • New helpers cover the toolbox, the Canvas tool (opened by clicking the canvas, palette drags, the element menu and toolbar), Change Layout mode, page size and orientation, key-by-key typing, geometry, feature status, undo, and importing a video through the Sign Language tool with the chooser hook. The image helpers take an optional scope, so a test can act on a picture box other than the page's first.
  • Palette items and menu items carry test ids. A tier-gated menu item is marked as such, because it stays clickable and so has no disabled class, and it marks itself while it is still waiting to hear whether its feature is on offer, so the menu helper can wait for the answer before reporting what is enabled.

Devin review


This change is Reviewable

hatton and others added 2 commits September 4, 2026 16:04
An end-to-end test could not yet reach three kinds of Bloom behaviour: a
feature behind a subscription tier, a feature behind an experimental-feature
token, and anything that opens a native file or folder chooser. This adds the
generic pieces, with nothing specific to any one feature.

Bloom side:
- New --experimental-features <tokens> switch, accepted only with --e2e. The
  tokens count as enabled for that run and are never written to user.config,
  which every Bloom of one build shares.
- BloomOpenFileDialog and BloomFolderChooser answer with a path an e2e test
  armed through the new e2e/nextFileToChoose route, instead of showing a
  dialog. Honoured only under --e2e; the armed path is used once, and a
  missing file or one the filter refuses throws.
- Under --e2e, Bloom no longer probes port 5173 for a Vite dev server, and
  refuses camera and microphone permission requests, which on a fresh browser
  profile would otherwise put up a WebView2 prompt no test can dismiss.
- data-testid on the canvas tool's palette items and on LocalizableMenuItem,
  which also marks a tier-gated item with data-subscription-gated, because
  such an item stays clickable and so carries no disabled class.

BloomE2E side:
- launchBloom/bloomTest take subscriptionCode and experimentalFeatures
  options; kEnterpriseSubscriptionCode is the existing Test-727011-1339 code
  from SubscriptionTests.cs. The launcher picks the newest Bloom.exe.
- New helpers: toolbox (open a tool, list the shown tools), canvasElements
  (open the Canvas tool by clicking the canvas, drag palette items, element
  menu and toolbar), keys (type key by key), geometry, videos (import a video
  through the Sign Language tool with the chooser hook), and getFeatureStatus.
- AUTOMATION-DEBT.md updated: key events and the video import now have routes;
  two new entries for the anonymous canvas toolbar buttons and the palette
  drag that hangs with a real mouse.

The Test branding stamps a butterfly watermark on every page so a screenshot
made with the test subscription is recognisable.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Comment thread src/BloomE2E/helpers/canvasElements.ts
@hatton

hatton commented Sep 4, 2026

Copy link
Copy Markdown
Member Author

[Claude Fable 5.1 from Hatton's machine during preflight] Consulted Devin on 2026-09-04 22:15 UTC up to commit abbe372.

Devin found no bugs. It raised one Investigate flag, about the palette drag being dispatched rather than driven with the real mouse; that is a documented exception, and the thread records why: #8314 (comment). No informational items. CI (pr-automation) passed.

Devin review: https://app.devin.ai/review/BloomBooks/BloomDesktop/pull/8314

hatton and others added 3 commits September 5, 2026 10:09
The branch that motivated this infrastructure grew a few more helpers
that have nothing to do with tables. This brings over the generic parts,
with the table-specific wording and code left behind.

- helpers/origami.ts: drive the Edit tab's Change Layout mode: toggle it,
  count the sections, split one, read and choose the types an empty
  section offers.
- helpers/pageSize.ts: read and change the book's page size and
  orientation through the layout-choice API.
- helpers/images.ts: chooseImageFile, getImagePlacement and cropImage take
  an optional `within` locator, so a test can act on a picture box other
  than the page's first one.
- helpers/bookMaking.ts: reloadPageBeingEdited leaves the page and comes
  back, so the page rebuilds its editing surfaces.
- helpers/workspace.ts: canUndo and undo, which call the front end's
  workspaceBundle the way the WinForms shell does for Ctrl+Z, a key press
  the browser never receives.
- helpers/publish.ts: the preview wait looks for a page of the book, not
  the language chooser, which a single-language book never shows.
- PAPERCUTS.md: an e2e test cannot see a data-testid just added to the
  front end, because the launched Bloom reads output/browser.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Master added its own e2e hook for pre-answering the native file chooser
(e2e/nextChosenFile, stored in E2eTestingApi and consumed only by
FileIOApi.SelectFileUsingDialog). This branch already has one that covers
every chooser, file and folder alike, because it lives in
BloomOpenFileDialog and BloomFolderChooser themselves
(e2e/nextFileToChoose). Resolved by keeping the one hook: the endpoint
master added is gone, armFileChooser in helpers/talkingBook.ts posts to
e2e/nextFileToChoose, and FileIOApi keeps master's rule of not remembering
a test's folder in FilePathMemory, now as a guard on the normal path.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…lpers

canUndo compared the front end's answer with "", but workspaceRoot.ts
answers "yes" or "fail", so the helper said yes whenever it was asked and
undo's "nothing to undo" guard could never fire. It now looks for "yes",
as the C# shell does. undo's comment no longer promises a wait it did not
do; the caller waits for the state it expects.

Leaving Change Layout mode waited for an editable text box to appear, but a
page whose sections are all pictures or videos has none, so the helper hung
for 60 seconds on a page that was fine. It now marks the page element in a
JavaScript property before the click and waits for a page element without
the mark, which is what Bloom's rebuild produces. The property is not
markup, so it cannot be saved into the book.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Comment thread src/BloomExe/ExperimentalFeatures.cs Outdated
Comment thread src/BloomE2E/helpers/canvasElements.ts
Comment thread src/BloomE2E/fixtures/launchBloom.ts
Comment thread src/BloomExe/ExperimentalFeatures.cs Outdated
hatton and others added 2 commits September 5, 2026 10:38
…nd menus say when their feature status is pending

Under --e2e, TokensOfEnabledFeatures returned the developer's saved
experimental features together with the ones the command line named, so
a test that needed a feature off could not get it off, and a run
depended on whatever the developer had turned on. It now returns only
the command-line tokens under --e2e; the saved setting is untouched and
still applies to a normal Bloom. The tests cover both the saved feature
not counting and an empty command line enabling nothing.

A tier-gated menu item looked enabled until its feature status arrived,
so a test reading the canvas element menu the moment it opened could
report a gated command as available. LocalizableMenuItem now marks such
an item with data-feature-status-pending until the answer is in, and
getCanvasElementMenuItems waits for the mark to clear before reading.

The launcher's comment and the README described the port 5173 probe
that --e2e no longer makes. The launcher now logs which Bloom.exe it
runs and when it was built, because findBloomExe takes the newest build
in any configuration.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
SetValue asked IsFeatureEnabled whether a feature was already saved, but
under --e2e that answers from the command line and ignores the saved
setting, so a feature that was saved looked absent and was appended
again on every save. It now asks the saved setting itself. Found by
Devin's review of the previous commit.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Comment thread src/BloomExe/ExperimentalFeatures.cs
Comment thread src/BloomE2E/helpers/canvasElements.ts
Comment thread src/BloomE2E/fixtures/launchBloom.ts
@hatton

hatton commented Sep 5, 2026

Copy link
Copy Markdown
Member Author

[Claude Fable 5.1 from Hatton's machine during preflight]

Consulted Devin (review page) up to ded01e4627, the head of this preflight run. Over the run's seven reviewed commits it raised eight bugs and five Investigate flags, no Informational flags; every one has its own thread with the outcome recorded and is resolved:

  • Saved experiments leaked into --e2e runs: fixed in 7df51db (thread; a duplicate post of the same finding is here).
  • Tier-gated menu items could be read before their feature status arrived: fixed in 7df51db (thread).
  • SetValue could save a feature twice under --e2e: fixed in 7afde6f (thread).
  • A feature token was matched as a substring, so not-team-collections enabled Team Collections: fixed in f5a86bf (thread).
  • The new stale-build check counted tests and docs as source, missed .uitest.ts files, ignored the Markdown the build compiles, and scanned only the front-end tree: fixed across f5a86bf, 003e695 and ded01e4 (1, 2, 3, 4).
  • The launcher's comment still described the port 5173 probe: fixed in 7df51db (thread).
  • The stale-build refusal told the reader to run pnpm build, which AGENTS.md reserves for the developer: reworded in f5a86bf (thread).
  • The newest Bloom.exe in any configuration is the one tested: the launcher now logs which one and when it was built (thread).
  • The palette drag is dispatched rather than driven by the real mouse: a documented exception, not changed (thread).
  • Startup still runs the settings migration under --e2e: not an issue, it writes only what the developer's own Bloom writes at every start (thread).

Devin's report for each later commit repeats the earlier findings unchanged alongside the new ones. CI (pr-automation) passed for every commit. CodeRabbit has a config file in the repo but posted nothing.

hatton and others added 2 commits September 5, 2026 11:17
The launched Bloom serves its front end from output/browser, and nothing
rebuilds that bundle when a .tsx file changes or a merge brings in
someone else's; the same goes for Bloom.dll and the C# source. A run
against a stale build fails the way a real regression does, in a test
whose feature is not in the build yet, and nothing in the output says
so. Today a whole-suite run failed one of master's own tests that way,
against a bundle a day older than the commit the test needed.

The fixture now compares the newest build output with the newest source
file on each side before it launches Bloom, and stops the run naming the
newer file and what to do about it. The bundle side counts only the
files the build writes, because a running Bloom writes a history
database into a template book inside the bundle folder, and that would
make a stale bundle look fresh. The bundle check is skipped when
BLOOM_E2E_VITE_PORT is set, since a dev server then serves the working
tree. Checked once per worker process.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s and docs

IsFeatureEnabled asked whether the list of enabled tokens contained the
feature's name as a substring, so a token that merely contained it
("not-team-collections") enabled Team Collections. Both it and SetValue
now split the list and compare whole tokens. A test covers it.

The stale-build check counted every file under src/BloomBrowserUI and
src/BloomExe as source, so editing a spec, a story or a Markdown file
stopped the suite for a bundle that was current. Those files are not
part of what the build produces and are now left out. The refusal also
no longer tells an agent to run pnpm build, which AGENTS.md reserves for
the developer; it names the dev server route first and says the rebuild
is the developer's call. Both found by Devin.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Comment thread src/BloomExe/ExperimentalFeatures.cs
Comment thread src/BloomE2E/fixtures/launchBloom.ts Outdated
Comment thread src/BloomExe/ExperimentalFeatures.cs
Comment thread src/BloomE2E/fixtures/launchBloom.ts
The check skipped every Markdown file, but the build compiles the help
and info pages from Markdown, so an edit to one of those left a stale
bundle looking current. Markdown under help/ and infoPages/ now counts.
The check also missed .uitest.ts files and the folders that hold only
tests and their fixtures, so an edit there stopped the suite for a
bundle that was fine. Those are now left out. Found by Devin.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Comment thread src/BloomE2E/fixtures/launchBloom.ts
Comment thread src/BloomE2E/fixtures/launchBloom.ts
The build compiles Markdown from the templates' ReadMe files and from the
top of DistFiles as well as from the front-end tree, and the check only
scanned the front-end tree, so an edit to one of those left a stale
bundle looking current. The scan now covers all three. Found by Devin.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Comment thread src/BloomE2E/fixtures/launchBloom.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant