Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Add functional tests for Field stats flyout in Transforms & DFA creation wizard #150927

Merged
merged 9 commits into from
Feb 15, 2023

Conversation

qn895
Copy link
Member

@qn895 qn895 commented Feb 10, 2023

Summary

Part of #147170. This PR adds functional tests for the Field stats flyout to the Transforms creation wizards.

Checklist

Delete any items that are not applicable to this PR.

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk Probability Severity Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space. Low High Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. High Low Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled. Medium High Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

@qn895 qn895 added :ml test_ui_functional release_note:skip Skip the PR/issue when compiling release notes v8.7.0 labels Feb 10, 2023
@qn895 qn895 self-assigned this Feb 10, 2023
@qn895 qn895 requested a review from a team as a code owner February 10, 2023 17:05
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@qn895 qn895 changed the title [ML] Add functional tests for Field stats flyout in Transforms creation wizard [ML] Add functional tests for Field stats flyout in Transforms & DFA creation wizard Feb 10, 2023
@qn895 qn895 mentioned this pull request Feb 13, 2023
14 tasks
Copy link
Contributor

@alvarezmelissa87 alvarezmelissa87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ⚡

@qn895
Copy link
Member Author

qn895 commented Feb 14, 2023

Started flaky test suite runner...

@pheyos
Copy link
Member

pheyos commented Feb 14, 2023

Also, the flaky test runner has some failures, looking into these...

@qn895
Copy link
Member Author

qn895 commented Feb 14, 2023

Started flaky test suite runner...

Copy link
Member

@pheyos pheyos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM
Let's hope that the initial flakiness has been fixed by the changes 🤞

@qn895 qn895 force-pushed the ml-functional-test-field-stats-flyout branch from 10d9917 to 69d124f Compare February 14, 2023 20:45
@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #30 / Actions and Triggers app Connectors Opsgenie connector page should disable the run button when the message field is not filled

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
ml 3.5MB 3.5MB +35.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @qn895

@pheyos
Copy link
Member

pheyos commented Feb 15, 2023

The flaky test runner failed and from the log it looks like steps are done in a wrong order:

debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlFieldStatsFlyout"]') with timeout=120000
debg TestSubjects.exists(mlFieldStatsFlyoutContent airline-title)
debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlFieldStatsFlyoutContent airline-title"]') with timeout=120000
debg TestSubjects.exists(mlFieldStatsFlyoutContent airline-topValues)
debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlFieldStatsFlyoutContent airline-topValues"]') with timeout=120000
debg TestSubjects.exists(mlFieldStatsFlyout)
debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlFieldStatsFlyout"]') with timeout=2500
debg TestSubjects.click(mlFieldStatsFlyout > euiFlyoutCloseButton)
debg Find.clickByCssSelector('[data-test-subj="mlFieldStatsFlyout"] [data-test-subj="euiFlyoutCloseButton"]') with timeout=10000
debg Find.findByCssSelector('[data-test-subj="mlFieldStatsFlyout"] [data-test-subj="euiFlyoutCloseButton"]') with timeout=10000
debg TestSubjects.missingOrFail(mlFieldStatsFlyout)
debg Find.waitForDeletedByCssSelector('[data-test-subj="mlFieldStatsFlyout"]') with timeout=2500
debg TestSubjects.exists(mlFieldStatsFlyoutContent airline-topValues)
debg Find.existsByDisplayedByCssSelector('[data-test-subj="mlFieldStatsFlyoutContent airline-topValues"]') with timeout=120000
debg --- retry.tryForTime error: [data-test-subj="mlFieldStatsFlyoutContent airline-topValues"] is not displayed

So the test actually closes the flyout and then tries to validate the top values, which obviously can't work. The code for this bit clearly defines a different order: first check the top values then close the flyout:

await this.assertFieldStatContentByType(parentComboBoxSelector, fieldName, fieldType);

if (Array.isArray(expectedTopValuesContent)) {
    await this.assertTopValuesContent(fieldName, fieldType, expectedTopValuesContent);
}
await this.ensureFieldStatsFlyoutClosed();

So I'm not exactly sure what's going wrong here. Also, it doesn't reproduce on my local and the PR CI run also didn't show that error. Maybe the flaky test runner had an issue or was running from a different commit level? Might be worth to give it another run.

@qn895
Copy link
Member Author

qn895 commented Feb 15, 2023

Ran the Flaky test suite runner again and it was successful after 50 tries ✅

@qn895 qn895 merged commit 7906c76 into elastic:main Feb 15, 2023
@qn895 qn895 deleted the ml-functional-test-field-stats-flyout branch February 15, 2023 15:47
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Feb 15, 2023
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.7

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Feb 15, 2023
…& DFA creation wizard (#150927) (#151332)

# Backport

This will backport the following commits from `main` to `8.7`:
- [[ML] Add functional tests for Field stats flyout in Transforms & DFA
creation wizard
(#150927)](#150927)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Quynh Nguyen
(Quinn)","email":"43350163+qn895@users.noreply.github.com"},"sourceCommit":{"committedDate":"2023-02-15T15:47:47Z","message":"[ML]
Add functional tests for Field stats flyout in Transforms & DFA creation
wizard
(#150927)","sha":"7906c76749fa89f56df626e1e4012e01bfe14753","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":[":ml","test_ui_functional","release_note:skip","v8.7.0","v8.8.0"],"number":150927,"url":"#150927
Add functional tests for Field stats flyout in Transforms & DFA creation
wizard
(#150927)","sha":"7906c76749fa89f56df626e1e4012e01bfe14753"}},"sourceBranch":"main","suggestedTargetBranches":["8.7"],"targetPullRequestStates":[{"branch":"8.7","label":"v8.7.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"#150927
Add functional tests for Field stats flyout in Transforms & DFA creation
wizard (#150927)","sha":"7906c76749fa89f56df626e1e4012e01bfe14753"}}]}]
BACKPORT-->

Co-authored-by: Quynh Nguyen (Quinn) <43350163+qn895@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:ml release_note:skip Skip the PR/issue when compiling release notes test_ui_functional v8.7.0 v8.8.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants