Skip to content

Playwright - Fixing flaky test.#26807

Open
kshinde3110 wants to merge 22 commits intoopen-metadata:mainfrom
kshinde3110:fix-flaky-playwright
Open

Playwright - Fixing flaky test.#26807
kshinde3110 wants to merge 22 commits intoopen-metadata:mainfrom
kshinde3110:fix-flaky-playwright

Conversation

@kshinde3110
Copy link
Copy Markdown
Collaborator

@kshinde3110 kshinde3110 commented Mar 26, 2026

Describe your changes:

Fixes

I worked on ... because ...

Type of change:

  • Bug fix
  • Improvement
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

Summary by Gitar

  • Playwright config update:
    • Disabled CSS animations/transitions via reducedMotion: 'reduce' context option to reduce test flakiness
  • Test utility improvement:
    • Replaced single loader wait with waitForAllLoadersToDisappear() helper in user search flow

This will update automatically on new commits.

@kshinde3110 kshinde3110 requested a review from a team as a code owner March 26, 2026 20:44
@kshinde3110 kshinde3110 added the safe to test Add this label to run secure Github workflows on PRs label Mar 26, 2026
Comment thread openmetadata-ui/src/main/resources/ui/playwright/utils/importUtils.ts Outdated
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 26, 2026

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 63%
63.92% (59768/93492) 43.63% (31251/71621) 46.73% (9395/20103)

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 26, 2026

🔴 Playwright Results — 44 failure(s), 18 flaky

✅ 2756 passed · ❌ 44 failed · 🟡 18 flaky · ⏭️ 340 skipped

Shard Passed Failed Flaky Skipped
🔴 Shard 1 213 44 0 202
🟡 Shard 2 642 0 2 32
🟡 Shard 3 643 0 7 26
🟡 Shard 4 626 0 2 47
🟡 Shard 6 632 0 7 33

Genuine Failures (failed on all attempts)

Features/DataAssetRulesDisabled.spec.ts › Verify the ApiEndpoint entity item action after rules disabled (shard 1)
TimeoutError: page.goto: Timeout 60000ms exceeded.
Call log:
�[2m  - navigating to "http://localhost:8585/", waiting until "load"�[22m

Features/DataAssetRulesDisabled.spec.ts › Verify the Chart entity item action after rules disabled (shard 1)
TimeoutError: page.goto: Timeout 60000ms exceeded.
Call log:
�[2m  - navigating to "http://localhost:8585/", waiting until "load"�[22m

Features/DataAssetRulesDisabled.spec.ts › Verify the Pipeline Service entity item action after rules disabled (shard 1)
TimeoutError: page.goto: Timeout 60000ms exceeded.
Call log:
�[2m  - navigating to "http://localhost:8585/", waiting until "load"�[22m

Pages/AuditLogs.spec.ts › should apply both User and EntityType filters simultaneously (shard 1)
�[31mTest timeout of 60000ms exceeded while running "beforeEach" hook.�[39m
Pages/AuditLogs.spec.ts › should create audit log entry when glossary is updated (shard 1)
�[31mTest timeout of 60000ms exceeded.�[39m
Pages/AuditLogs.spec.ts › should create audit log entry when glossary is soft deleted (shard 1)
�[31mTest timeout of 60000ms exceeded.�[39m
Pages/AuditLogs.spec.ts › should create audit log entry when glossary is restored (shard 1)
�[31mTest timeout of 60000ms exceeded.�[39m
Pages/AuditLogs.spec.ts › should create audit log entry when glossary is hard deleted (shard 1)
�[31mTest timeout of 60000ms exceeded.�[39m
Pages/AuditLogs.spec.ts › should verify complete audit trail for entity lifecycle (shard 1)
�[31mTest timeout of 60000ms exceeded.�[39m
Pages/AuditLogs.spec.ts › should display audit log entry in UI after entity creation (shard 1)
�[31mTest timeout of 60000ms exceeded.�[39m
Pages/Bots.spec.ts › Bots Page should work properly (shard 1)
TimeoutError: page.waitForURL: Timeout 60000ms exceeded.
=========================== logs ===========================
waiting for navigation to "**/my-data" until "load"
============================================================
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m
Pages/Customproperties-part1.spec.ts › Integer (shard 1)
�[31m"beforeAll" hook timeout of 60000ms exceeded.�[39m

... and 14 more failures

🟡 18 flaky test(s) (passed on retry)
  • Features/BulkEditEntity.spec.ts › Glossary (shard 2, 1 retry)
  • Features/ChangeSummaryBadge.spec.ts › Automated badge should appear on entity description with Automated source (shard 2, 1 retry)
  • Features/RestoreEntityInheritedFields.spec.ts › Validate restore with Inherited domain and data products assigned (shard 3, 2 retries)
  • Features/RestoreEntityInheritedFields.spec.ts › Validate restore with Inherited domain and data products assigned (shard 3, 1 retry)
  • Features/RestoreEntityInheritedFields.spec.ts › Validate restore with Inherited domain and data products assigned (shard 3, 1 retry)
  • Features/RestoreEntityInheritedFields.spec.ts › Validate restore with Inherited domain and data products assigned (shard 3, 2 retries)
  • Features/RestoreEntityInheritedFields.spec.ts › Validate restore with Inherited domain and data products assigned (shard 3, 1 retry)
  • Features/RTL.spec.ts › Verify Following widget functionality (shard 3, 1 retry)
  • Flow/PersonaFlow.spec.ts › Set default persona for team should work properly (shard 3, 1 retry)
  • Pages/Customproperties-part2.spec.ts › entityReferenceList shows item count, scrollable list, no expand toggle (shard 4, 1 retry)
  • Pages/Domains.spec.ts › Rename domain with data products attached at domain and subdomain levels (shard 4, 1 retry)
  • Pages/Lineage/DataAssetLineage.spec.ts › verify create lineage for entity - Worksheet (shard 6, 1 retry)
  • Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)
  • Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab IS visible for supported type: searchIndex (shard 6, 1 retry)
  • Pages/Lineage/PlatformLineage.spec.ts › Verify domain platform view (shard 6, 1 retry)
  • Pages/ServiceEntity.spec.ts › Tier Add, Update and Remove (shard 6, 1 retry)
  • Pages/Tag.spec.ts › Add and Remove Assets for Data Consumer (shard 6, 1 retry)
  • Pages/Users.spec.ts › Permissions for table details page for Data Consumer (shard 6, 1 retry)

📦 Download artifacts

How to debug locally
# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

@@ -369,6 +359,7 @@ test.describe(
);
expect(postRes.status()).toBe(403);

await fetchIncidentId(apiContext);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why we need too fetch this here? also we are returning the incidentID but not storing and using it

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed this

Comment thread openmetadata-ui/src/main/resources/ui/playwright/utils/entity.ts
Comment thread openmetadata-ui/src/main/resources/ui/playwright/utils/importUtils.ts Outdated
Comment on lines +777 to +786
await expect(async () => {
const certificationResponse = page.waitForResponse(
'/api/v1/tags?parent=Certification*'
);
await page.keyboard.press('Enter', { delay: 100 });
await certificationResponse;

await expect(certRadioBtn).toBeVisible({ timeout: 5000 });
}).toPass({ timeout: 30000, intervals: [2000, 3000, 5000] });

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you provide some details for wrapping it in expect block?

Comment thread openmetadata-ui/src/main/resources/ui/playwright/utils/user.ts Outdated
Comment thread openmetadata-ui/src/main/resources/ui/playwright/utils/user.ts Outdated
Comment thread openmetadata-ui/src/main/resources/ui/playwright/utils/entity.ts Outdated
@github-actions
Copy link
Copy Markdown
Contributor

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@gitar-bot
Copy link
Copy Markdown

gitar-bot bot commented Apr 13, 2026

Code Review ⚠️ Changes requested 9 resolved / 12 findings

Flaky test fixes addressed multiple issues including retry logic restoration and dead code removal, but three findings remain: parseInt may access the wrong field in DataAssetsCoveragePieChartWidget, entityIconMapping lost EntityType keys during refactoring, and AuthProvider login polling silently fails after 5 seconds.

⚠️ Bug: entityIconMapping lost many EntityType keys during refactoring

The entityIconMapping was moved from inside getEntityIcon() to module scope and de-duplicated. However, many EntityType.* keys were removed (e.g., EntityType.DATABASE, EntityType.TABLE, EntityType.TOPIC, EntityType.DASHBOARD, EntityType.PIPELINE, EntityType.GLOSSARY, EntityType.GLOSSARY_TERM, EntityType.CHART, etc.) keeping only the SearchIndex.* variant. Similarly EntityType.TABLE_COLUMN (string 'tableColumn') was replaced by SearchIndex.COLUMN (string 'column') — these are different strings. Any caller of getEntityIcon passing an EntityType value that differs from its SearchIndex counterpart will get a missing icon. While many EntityType/SearchIndex pairs share the same string value, TABLE_COLUMN vs COLUMN is a confirmed mismatch, and MLMODEL_SERVICE casing may differ between enums.

Suggested fix
Restore the `EntityType.TABLE_COLUMN` entry alongside `SearchIndex.COLUMN`, and verify all removed EntityType keys have identical string values to their SearchIndex counterparts. For any mismatches, keep both entries:
```ts
[SearchIndex.COLUMN]: ColumnIcon,
[EntityType.TABLE_COLUMN]: ColumnIcon,
```
💡 Bug: parseInt on fullyQualifiedName field looks like wrong field access

In DataAssetsCoveragePieChartWidget, line 95 parses totalData[0].fullyQualifiedName as an integer, and line 94 parses coverageData[0].originEntityFQN as an integer. These field names typically hold string FQN values, not numeric counts. If the search aggregation API returns numeric values in these fields by convention, this code works but is fragile and confusing. If the API response format changes or different fields should be used, this would silently produce NaN.

💡 Edge Case: AuthProvider login polling silently fails after 5s timeout

The new polling mechanism in onLoginHandler retries up to 100 times (5 seconds) waiting for authenticatorRef.current to be available. If it never becomes available, the only feedback is setApplicationLoading(false) — the user sees the loading spinner disappear but gets no error message or redirect, leaving them on a broken login page with no indication of what happened.

Suggested fix
Show an error toast or redirect when max attempts are exhausted:
```ts
} else {
  setApplicationLoading(false);
  showErrorToast('Authentication service unavailable. Please refresh and try again.');
}
```
✅ 9 resolved
Bug: fetchIncidentId may return undefined, causing PATCH to wrong URL

📄 openmetadata-ui/src/main/resources/ui/playwright/e2e/Features/DataQuality/TestCaseIncidentPermissions.spec.ts:330-338
At line 330, fetchIncidentId is called but its return value is not checked. If the incident hasn't been created yet (async timing), incidentId remains undefined and the PATCH at line 332 will hit /api/v1/dataQuality/testCases/testCaseIncidentStatus/undefined — which likely returns 404, not the expected 403. This defeats the purpose of the permission test.

Notably, the other test at line 362-363 correctly guards with if (incidentId), but this test does not. The old code guaranteed incidentId was populated via polling in beforeAll; the new lazy approach needs a guard or assertion.

Quality: Commented-out line should be removed, not left as dead code

📄 openmetadata-ui/src/main/resources/ui/playwright/utils/importUtils.ts:202
Line 202 in importUtils.ts has a commented-out click action (// await page.click('[data-testid="tag-selector"]');). If the click is no longer needed, the line should be removed entirely rather than left as a comment.

Bug: visitUserProfilePage lost retry logic for search indexing lag

📄 openmetadata-ui/src/main/resources/ui/playwright/utils/user.ts:118-130
The original visitUserProfilePage used expect.poll() with a 60-second timeout and backoff intervals to handle cases where the search index hasn't caught up yet for newly created users. The new code (lines 118-130) makes a single search request and immediately asserts visibility with the default timeout (~5s). If the user was recently created and the search index hasn't been updated yet, the first search may return zero results and the test will fail immediately instead of retrying.

For a PR aimed at reducing flakiness, this removes a resilience mechanism that was specifically designed to handle eventual consistency in the search index.

Quality: Inconsistent indentation in waitForResponse callback

📄 openmetadata-ui/src/main/resources/ui/playwright/utils/user.ts:118-122 📄 openmetadata-ui/src/main/resources/ui/playwright/e2e/Pages/AuditLogs.spec.ts:312-322
The waitForResponse callback body at lines 119-121 in user.ts is indented at the top level (no indentation relative to the arrow function), breaking the surrounding code's indentation convention. Similarly, lines 312-322 in AuditLogs.spec.ts have the callback body indented at the wrong level.

Bug: Retry block with Enter key press may toggle popup open/close

📄 openmetadata-ui/src/main/resources/ui/playwright/utils/importUtils.ts:772-784
The expect().toPass() retry block at line 776-784 calls page.keyboard.press('Enter') on each retry attempt. If the first Enter press opens the certification dropdown but the visibility check fails (e.g., due to slow API response), the next retry's Enter press will close the already-open dropdown, making the radio button invisible again. This creates an alternating open/close pattern that will only succeed on odd-numbered attempts.

A more robust approach is to check whether the dropdown is already open before pressing Enter, or to re-click the active cell before each retry to reset state.

...and 4 more resolved from earlier reviews

🤖 Prompt for agents
Code Review: Flaky test fixes addressed multiple issues including retry logic restoration and dead code removal, but three findings remain: parseInt may access the wrong field in DataAssetsCoveragePieChartWidget, entityIconMapping lost EntityType keys during refactoring, and AuthProvider login polling silently fails after 5 seconds.

1. 💡 Bug: parseInt on fullyQualifiedName field looks like wrong field access

   In `DataAssetsCoveragePieChartWidget`, line 95 parses `totalData[0].fullyQualifiedName` as an integer, and line 94 parses `coverageData[0].originEntityFQN` as an integer. These field names typically hold string FQN values, not numeric counts. If the search aggregation API returns numeric values in these fields by convention, this code works but is fragile and confusing. If the API response format changes or different fields should be used, this would silently produce NaN.

2. ⚠️ Bug: entityIconMapping lost many EntityType keys during refactoring

   The `entityIconMapping` was moved from inside `getEntityIcon()` to module scope and de-duplicated. However, many `EntityType.*` keys were removed (e.g., `EntityType.DATABASE`, `EntityType.TABLE`, `EntityType.TOPIC`, `EntityType.DASHBOARD`, `EntityType.PIPELINE`, `EntityType.GLOSSARY`, `EntityType.GLOSSARY_TERM`, `EntityType.CHART`, etc.) keeping only the `SearchIndex.*` variant. Similarly `EntityType.TABLE_COLUMN` (string `'tableColumn'`) was replaced by `SearchIndex.COLUMN` (string `'column'`) — these are different strings. Any caller of `getEntityIcon` passing an EntityType value that differs from its SearchIndex counterpart will get a missing icon. While many EntityType/SearchIndex pairs share the same string value, `TABLE_COLUMN` vs `COLUMN` is a confirmed mismatch, and `MLMODEL_SERVICE` casing may differ between enums.

   Suggested fix:
   Restore the `EntityType.TABLE_COLUMN` entry alongside `SearchIndex.COLUMN`, and verify all removed EntityType keys have identical string values to their SearchIndex counterparts. For any mismatches, keep both entries:
   ```ts
   [SearchIndex.COLUMN]: ColumnIcon,
   [EntityType.TABLE_COLUMN]: ColumnIcon,
   ```

3. 💡 Edge Case: AuthProvider login polling silently fails after 5s timeout

   The new polling mechanism in `onLoginHandler` retries up to 100 times (5 seconds) waiting for `authenticatorRef.current` to be available. If it never becomes available, the only feedback is `setApplicationLoading(false)` — the user sees the loading spinner disappear but gets no error message or redirect, leaving them on a broken login page with no indication of what happened.

   Suggested fix:
   Show an error toast or redirect when max attempts are exhausted:
   ```ts
   } else {
     setApplicationLoading(false);
     showErrorToast('Authentication service unavailable. Please refresh and try again.');
   }
   ```

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

@sonarqubecloud
Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants