Skip to content

Fix E2E test quality issues: always-passing assertions, unawaited checks, and dead code#32801

Merged
richvdh merged 2 commits into
element-hq:developfrom
voidmatcha:fix/e2e-test-assertions
Mar 16, 2026
Merged

Fix E2E test quality issues: always-passing assertions, unawaited checks, and dead code#32801
richvdh merged 2 commits into
element-hq:developfrom
voidmatcha:fix/e2e-test-assertions

Conversation

@voidmatcha
Copy link
Copy Markdown
Contributor

@voidmatcha voidmatcha commented Mar 14, 2026

Fix five E2E test quality issues across four spec files.

These issues were found using e2e-reviewer, a static analysis skill for Playwright/Cypress E2E tests that detects common anti-patterns.


crypto/migration.spec.ts — Remove vacuous toBeGreaterThanOrEqual(0)

// Before
expect(initialProgress).toBeGreaterThanOrEqual(0);
expect(initialProgress).toBeLessThanOrEqual(500);

// After
expect(initialProgress).toBeLessThanOrEqual(500);

toBeGreaterThanOrEqual(0) can never fail for a progress bar value. The adjacent toBeLessThanOrEqual(500) already bounds the valid range — and since NaN <= 500 evaluates to false, it also catches the parseFloat() returning NaN case. The >= 0 check is entirely redundant.


devtools/devtools.spec.ts / crypto/utils.ts — Fix unawaited isVisible() calls

// Before — Promise result discarded, check never runs
await page.getByText("Saved").isVisible();

// After
await expect(page.getByText("Saved")).toBeVisible();

isVisible() returns a Promise<boolean> without retrying or throwing. Without await, the expression resolves to a Promise object (always truthy), making the check a silent no-op. Playwright docs state: "Returns whether the element is visible. The method returns immediately (does not wait)" — web-first assertions like toBeVisible() retry automatically and throw on failure, which is the correct behaviour here.


spotlight/spotlight.spec.ts — Remove 1-hour waitForTimeout from a skipped test

// test.skip("should find unknown public rooms on other homeservers", ...)
- await page.waitForTimeout(3_600_000);
  await page.waitForTimeout(500); // wait for the dialog to settle

This line is inside a test.skip() block (the test requires federation which is unavailable in local e2e runs), so it never executes in CI. It was introduced in the original Playwright migration (matrix-react-sdk#12033, commit d1562bef, 2023-12-18) and appears to be a debugging artifact: a 1-hour pause so the developer could inspect browser state, left in before committing.

A possible alternative reading is that it was intentional — to make CI hang obviously if someone removes test.skip() prematurely. Either way, the // TODO comment directly above the test already communicates the intent clearly. A 1-hour hang is a poor signal; an explicit throw new Error("Requires federation — do not remove test.skip") would be clearer if a guard is desired.


threads/threads.spec.ts — Replace toBeAttached() with toBeVisible()

// Before — passes even if messages are hidden (display:none / visibility:hidden)
await expect(locator.locator(".mx_EventTile").first().getByText("Hello Mr. Bot")).toBeAttached();
await expect(locator.locator(".mx_EventTile").last().getByText("Hello Mr. User")).toBeAttached();

// After
await expect(locator.locator(".mx_EventTile").first().getByText("Hello Mr. Bot")).toBeVisible();
await expect(locator.locator(".mx_EventTile").last().getByText("Hello Mr. User")).toBeVisible();

The test verifies that thread messages are rendered after opening a thread. toBeAttached() only checks DOM presence — an element hidden with display:none or visibility:hidden would still pass. toBeVisible() is the correct assertion here.

Checklist

Notes: none

…cks, and dead code

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@voidmatcha voidmatcha requested review from a team as code owners March 14, 2026 12:53
@github-actions github-actions Bot added the Z-Community-PR Issue is solved by a community member's PR label Mar 14, 2026
@florianduros florianduros added the T-Task Tasks for the team like planning label Mar 14, 2026
Copy link
Copy Markdown
Member

@florianduros florianduros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for the fixes!

const initialProgress = parseFloat(await progressBar.getAttribute("value"));
const initialMax = parseFloat(await progressBar.getAttribute("max"));
expect(initialMax).toBeGreaterThan(4000);
expect(initialProgress).toBeGreaterThanOrEqual(0);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm failing to follow why you're removing this. Asserting that the progress is not negative seems useful to me?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. My original reasoning was that NaN <= 500 evaluates to false, so toBeLessThanOrEqual(500) already catches the parseFloat() returning NaN case. But you're right that it doesn't guard against negative values — >= 0 is not redundant for that. Restored.

Comment on lines +457 to +458
await expect(locator.locator(".mx_EventTile").first().getByText("Hello Mr. Bot")).toBeVisible();
await expect(locator.locator(".mx_EventTile").last().getByText("Hello Mr. User")).toBeVisible();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not that I object to this change, but it seems a bit odd to change this but leave the instance of toBeAttached at line 444 unchanged.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, missed that one. Fixed to match the rest.

@richvdh richvdh self-requested a review March 16, 2026 15:05
Copy link
Copy Markdown
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@richvdh richvdh enabled auto-merge March 16, 2026 17:47
@richvdh richvdh added this pull request to the merge queue Mar 16, 2026
Merged via the queue into element-hq:develop with commit 68070b2 Mar 16, 2026
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

T-Task Tasks for the team like planning Z-Community-PR Issue is solved by a community member's PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants