Unskip enrichments tests #171983

nkhristinin · 2023-11-27T13:06:36Z

Fix enrichments cypress tests

Related PR with removing ILM as it not supported in serverless. Here the error in the CI related to that

nkhristinin · 2023-12-21T16:17:24Z

@elasticmachine merge upstream

nkhristinin · 2023-12-22T10:25:53Z

/ci

nkhristinin · 2023-12-22T10:27:14Z

@elasticmachine merge upstream

nkhristinin · 2023-12-22T12:35:41Z

flaky test runs: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4671#018c9151-11be-406a-ae63-8fd048ed0513

nkhristinin · 2023-12-22T13:55:33Z

flaky tests 100 ess + 100 serverless https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4673

rylnd

I would love an explanation for the problem/solution here; without that, I don't see how we can have confidence that this will fix the flakiness.

Also, on my investigation branch I was observing the failures at about a 1/75 rate, so I think a flakey run of more than 100 would be appropriate here to eliminate the potential for a false positive (or would it be a false negative? Either way, you get my meaning).

rylnd · 2023-12-22T18:08:26Z

x-pack/test/security_solution_cypress/cypress/e2e/entity_analytics/enrichments.cy.ts

@@ -37,7 +37,7 @@ const CURRENT_HOST_RISK_LEVEL = 'Current host risk level';
 const ORIGINAL_HOST_RISK_LEVEL = 'Original host risk level';

 // FLAKY: https://github.com/elastic/kibana/issues/169154


Suggested change

// FLAKY: https://github.com/elastic/kibana/issues/169154

rylnd · 2023-12-22T18:11:07Z

x-pack/test/security_solution_cypress/es_archives/risk_hosts_updated/mappings.json

@@ -31,10 +31,6 @@
    },
    "settings": {
      "index": {
-        "lifecycle": {


Can you please provide an explanation for the issue that caused the test failures, and why removing these ILM settings fixes them? Other developers (myself included; this one was tricky) would benefit greatly from your findings here.

My understanding, is that ILM is not supported in serverless, which mean they not supported in cypress tests, I put some related PR/error in description

Another idea, that maybe it was partially fixed by this PR, when EA tests were separated to a different execution

@nkhristinin it looks like link you provided to the error has expired. Can you please post something more permanent here for posterity? That way if this test fails again, we can validate whether the hypothesized ILM error is to blame.

RE the ILM hypothesis: in my investigation I saw failures in ESS as well, so I think that invalidates ILM as the underlying issue.

I also don't think ILM explains the behavior I was observing: the test fails due to the presence of unexpected data (specifically, historical risk scores).

Not sure, what was with the link, but I changed it to the link of failing build in this pr.

The serverless tests are failing because of ILM, I am not sure if it's related to your initial investigation, or if it's a new problem.

I had not encountered that ILM error during my investigation, and since it's specific to serverless it cannot be the root cause of #169154.

Yes, it's not the root cause of those issues. After more deep look into that, this is my understanding of the timeline:

We did have some flaky tests, which you investigate in this PR

After some time, EA tests were moved to separate executions

When I started to work on those tests, there were constant test failures in serverless, not flaky tests, which this PR if fixed.

Then I ran several flaky test runs, which were successful for enrichments, but had failures not related to this test case:

x100 - errors related to dashboard

x200 - errors related to dashboard

And then I saw that there was one more x200 test run, which had 1 error (test 175) for enrichment tests, but this error also not related to original issues, but something that happened in before all hook

Original error:

AssertionError: Timed out retrying after 150000ms: Expected not to find content: 'Original host risk level' within the element: <div.euiFlexGroup.StyledEuiFlexGroup-sc-1az7ye2-2.chTqHU.css-1a15qtm-euiFlexGroup-responsive-none-flexStart-center-row> but continuously found it.
at Context.eval (webpack:///./e2e/entity_analytics/enrichments.cy.ts:129:69)

New error:

CypressError: cy.task('esArchiverUnload') timed out after waiting 60000ms.
https://on.cypress.io/api/task
Because this error occurred during a before all hook we are skipping the remaining tests in the current suite: Enrichment

Conclusion:
We have not yet seen that original issue, but there were other flaky tests, so I will continue to run the flaky test runner, and we should have more details.

No, the last test run confirmed that it still flaky -

nkhristinin · 2023-12-27T07:30:00Z

@elasticmachine merge upstream

nkhristinin · 2023-12-27T07:37:10Z

200 ESS + 200 serverless flaky test runner
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4679

nkhristinin · 2024-01-03T08:33:23Z

@elasticmachine merge upstream

nkhristinin · 2024-01-04T07:20:41Z

@elasticmachine merge upstream

nkhristinin · 2024-01-05T08:29:57Z

@elasticmachine merge upstream

nkhristinin · 2024-01-08T06:35:45Z

@elasticmachine merge upstream

nkhristinin · 2024-01-08T09:34:45Z

250+247 flaky tests - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4782

rylnd

With the changes included in this branch, and the parallel changes discussed here, these tests look to be significantly less flaky (i.e. 1 failure in 1000). I'm happy to merge this, get our test coverage back, and iterate in the future if these do fail in the wild again.

Thanks for taking the time to dig deep here @nkhristinin !

nkhristinin · 2024-01-09T08:43:20Z

One more flaky test run
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4800

nkhristinin · 2024-01-11T08:29:54Z

@elasticmachine merge upstream

MadameSheema

security-engineering-productivity changes LGTM

…-fix'

kibana-ci · 2024-01-11T17:16:13Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 3afc42f

Failed CI Steps

Jest Tests #1

Test Failures

[job] [logs] Jest Tests #1 / AssigneesFilterPopover calls onSelectionChange when 1 user is selected
[job] [logs] Jest Tests #1 / AssigneesFilterPopover calls onSelectionChange with a single user when different users are selected

Metrics [docs]

✅ unchanged

History

💔 Build #187014 failed 8ea0b87
💚 Build #186933 succeeded efe503d
💚 Build #186278 succeeded 37800c9
💛 Build #186011 was flaky c40f012
💚 Build #185768 succeeded 989b5d3
💛 Build #185572 was flaky f450001

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

## Fix enrichments cypress tests [Related PR with removing ILM](elastic#167916) as it not supported in serverless. Here the [error](https://buildkite.com/elastic/kibana-pull-request/builds/179121) in the CI related to that --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Unskip enrichments tests

0ff1429

kibanamachine and others added 2 commits December 21, 2023 11:17

Merge branch 'main' into enrichment-cypress

444a483

update mappings and remove lifecycle

b106b04

nkhristinin marked this pull request as ready for review December 22, 2023 10:26

nkhristinin requested review from a team as code owners December 22, 2023 10:26

nkhristinin added the release_note:skip Skip the PR/issue when compiling release notes label Dec 22, 2023

Merge branch 'main' into enrichment-cypress

a36a58e

rylnd reviewed Dec 22, 2023

View reviewed changes

Merge branch 'main' into enrichment-cypress

738b65f

Remove comment

edde6de

Merge branch 'main' into enrichment-cypress

f450001

Merge branch 'main' into enrichment-cypress

989b5d3

Merge branch 'main' into enrichment-cypress

c40f012

Merge branch 'main' into enrichment-cypress

37800c9

rylnd approved these changes Jan 9, 2024

View reviewed changes

Merge branch 'main' into enrichment-cypress

efe503d

MadameSheema approved these changes Jan 11, 2024

View reviewed changes

nkhristinin and others added 2 commits January 11, 2024 16:19

Remove duplicated test checks

8ea0b87

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

3afc42f

…-fix'

nkhristinin merged commit d43e7e5 into elastic:main Jan 12, 2024
38 checks passed

kibanamachine added v8.13.0 backport:skip This commit does not require backporting labels Jan 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unskip enrichments tests #171983

Unskip enrichments tests #171983

nkhristinin commented Nov 27, 2023 •

edited

Loading

nkhristinin commented Dec 21, 2023

nkhristinin commented Dec 22, 2023

nkhristinin commented Dec 22, 2023

nkhristinin commented Dec 22, 2023

nkhristinin commented Dec 22, 2023

rylnd left a comment

rylnd Dec 22, 2023

rylnd Dec 22, 2023

nkhristinin Dec 27, 2023

nkhristinin Jan 3, 2024

rylnd Jan 4, 2024

rylnd Jan 4, 2024

nkhristinin Jan 5, 2024

rylnd Jan 5, 2024

nkhristinin Jan 8, 2024

nkhristinin Jan 8, 2024

nkhristinin commented Dec 27, 2023

nkhristinin commented Dec 27, 2023

nkhristinin commented Jan 3, 2024

nkhristinin commented Jan 4, 2024

nkhristinin commented Jan 5, 2024

nkhristinin commented Jan 8, 2024

nkhristinin commented Jan 8, 2024

rylnd left a comment

nkhristinin commented Jan 9, 2024

nkhristinin commented Jan 11, 2024

MadameSheema left a comment

kibana-ci commented Jan 11, 2024

		@@ -37,7 +37,7 @@ const CURRENT_HOST_RISK_LEVEL = 'Current host risk level';
		const ORIGINAL_HOST_RISK_LEVEL = 'Original host risk level';

		// FLAKY: https://github.com/elastic/kibana/issues/169154

Unskip enrichments tests #171983

Unskip enrichments tests #171983

Conversation

nkhristinin commented Nov 27, 2023 • edited Loading

Fix enrichments cypress tests

nkhristinin commented Dec 21, 2023

nkhristinin commented Dec 22, 2023

nkhristinin commented Dec 22, 2023

nkhristinin commented Dec 22, 2023

nkhristinin commented Dec 22, 2023

rylnd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nkhristinin commented Dec 27, 2023

nkhristinin commented Dec 27, 2023

nkhristinin commented Jan 3, 2024

nkhristinin commented Jan 4, 2024

nkhristinin commented Jan 5, 2024

nkhristinin commented Jan 8, 2024

nkhristinin commented Jan 8, 2024

rylnd left a comment

Choose a reason for hiding this comment

nkhristinin commented Jan 9, 2024

nkhristinin commented Jan 11, 2024

MadameSheema left a comment

Choose a reason for hiding this comment

kibana-ci commented Jan 11, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

nkhristinin commented Nov 27, 2023 •

edited

Loading