Jest flakiness testing #176005

jbudz · 2024-01-31T17:57:22Z

Not for merging

… Evaluate` API and migrates `Post Evaluate` API to OAS (elastic#175338)" This reverts commit 38f0a7a.

jbudz · 2024-01-31T17:57:32Z

buildkite test this

jbudz · 2024-01-31T18:12:54Z

buildkite test this

jbudz · 2024-01-31T18:17:29Z

buildkite test this

jbudz · 2024-01-31T18:32:05Z

buildkite test this

…t Evaluate` API and migrates `Post Evaluate` API to OAS (elastic#175338)" This reverts commit b201bf6.

jbudz · 2024-01-31T19:03:32Z

buildkite test this

jbudz · 2024-01-31T19:29:04Z

buildkite test this

This reverts commit 64f4908.

spong · 2024-01-31T20:52:26Z

buildkite test this

spong · 2024-01-31T21:07:05Z

Notes:

Bringing back changes from original PR (#175338) via 6eacc9b to reintroduce the issue and to start narrowing down to root cause.

It's interesting that the errors are coming out of x-pack/plugins/security_solution/public/common/jest.config.js. since the offending PR only changed assistant code. This leads me to believe some of the structural refactoring is what is affecting a separate security solution test. At this moment, I have not found a debugging method to surface the offending test that is resulting in an update in an asynchronous callback, so will audit security solution assistant tests.

Current suspicions:

🔴 Separation of evaluation api tests from api.tsx -> evaluate.test.tsx changed execution timing and surfaced pre-existing issue.
- Testing in ee8c315 -- build link.
🔴 Refactoring of types in elastic_assistant plugin's post_evaluate.test.ts to use types imported from @kbn/elastic-assistant-common. There's suspicion of imports causing the issue as discussed here.
- Testing in 6856015 -- build link
🟢 Refactoring of kbn-elastic-assistant package api tests from http.fetch -> http.get/post.
- Testing in 2d4161f -- build link
Audit security solution assistant tests
- To be tested...

Links:

🟢 Build with revert 25x tests
🔴 Build without revert 25x tests

Summary:

Debugging via the above steps, it was identified that the refactoring of the http.fetch calls within the kbn-elastic-assistant package was resulting in sporadic jest failures within the security solution plugin as http.get was mocked differently within the security solution's MockAssistantProvider. TBD as to why it was sporadic and not a consistent failure, but suspicion is that an existing test suite did not rely on the mocked value from http.get to pass, and the mis-match resulted from a timeout that surfaced after the test completed (thus causing the async error logged).

Upon further debugging locally, I was able to consistently get an output failure for the x-pack/plugins/security_solution/public/common/components/top_n/index.test.tsx suite when running all security solution jest tests. Upon further inspection, it looks like the last test in the suite would be problematic as it's doing a forEach of tests over an array of 4 items, mounting the above test providers (which contains the mock). Incidentally, the number of The document global was defined's logs were 4 per run as well. Skipping just this test ends up resulting in no error or failures, so I think we've isolated it to this test. I've included a fix for this by unmount()'ing the wrapper after each test

Re-submission of original PR, which includes fix for above test here: #176025

spong · 2024-01-31T21:13:57Z

buildkite test this

spong · 2024-01-31T21:47:03Z

buildkite test this

kibana-ci · 2024-01-31T22:24:35Z

💔 Build Failed

Failed CI Steps

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

… API and migrates Post Evaluate API to OAS (#176025) > [!IMPORTANT] > This PR is a reintroduction of #175338, which was [reverted](#175338 (comment)) due to sporadic jest failures within the `security_solution` plugin. Root cause was identified and detailed in #176005 (comment). ## Summary In #174317 we added support for OpenAPI codegen, this PR builds on that functionality by migrating the `Post Evaluate` route `/internal/elastic_assistant/evaluate` to be backed by an OAS, and adds a basic `Get Evaluate` route for rounding out the enhancements outlined in elastic/security-team#8167 (to be in a subsequent PR). Changes include: * Migration of `Post Evaluate` route to OAS * Migration of `Post Evaluate` route to use versioned router * Extracted `evaluate` API calls from `x-pack/packages/kbn-elastic-assistant/impl/assistant/api/api.tsx` to `x-pack/packages/kbn-elastic-assistant/impl/assistant/api/evaluate/evaluate.tsx` * Co-located relevant `use_perform_evaluation` hook * Adds `Get Evaluate` route, and corresponding `use_evaluation_data` hook. Currently only returns `agentExecutors` to be selected for evaluation. * API versioning constants added to `x-pack/packages/kbn-elastic-assistant-common/impl/schemas/index.ts` * Adds new `buildRouteValidationWithZod` function to `x-pack/plugins/elastic_assistant/server/schemas/common.ts` for validating routes against OAS generated zod schemas. ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

… API and migrates Post Evaluate API to OAS (elastic#176025) > [!IMPORTANT] > This PR is a reintroduction of elastic#175338, which was [reverted](elastic#175338 (comment)) due to sporadic jest failures within the `security_solution` plugin. Root cause was identified and detailed in elastic#176005 (comment). ## Summary In elastic#174317 we added support for OpenAPI codegen, this PR builds on that functionality by migrating the `Post Evaluate` route `/internal/elastic_assistant/evaluate` to be backed by an OAS, and adds a basic `Get Evaluate` route for rounding out the enhancements outlined in https://github.com/elastic/security-team/issues/8167 (to be in a subsequent PR). Changes include: * Migration of `Post Evaluate` route to OAS * Migration of `Post Evaluate` route to use versioned router * Extracted `evaluate` API calls from `x-pack/packages/kbn-elastic-assistant/impl/assistant/api/api.tsx` to `x-pack/packages/kbn-elastic-assistant/impl/assistant/api/evaluate/evaluate.tsx` * Co-located relevant `use_perform_evaluation` hook * Adds `Get Evaluate` route, and corresponding `use_evaluation_data` hook. Currently only returns `agentExecutors` to be selected for evaluation. * API versioning constants added to `x-pack/packages/kbn-elastic-assistant-common/impl/schemas/index.ts` * Adds new `buildRouteValidationWithZod` function to `x-pack/plugins/elastic_assistant/server/schemas/common.ts` for validating routes against OAS generated zod schemas. ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

… API and migrates Post Evaluate API to OAS (elastic#176025) > [!IMPORTANT] > This PR is a reintroduction of elastic#175338, which was [reverted](elastic#175338 (comment)) due to sporadic jest failures within the `security_solution` plugin. Root cause was identified and detailed in elastic#176005 (comment). ## Summary In elastic#174317 we added support for OpenAPI codegen, this PR builds on that functionality by migrating the `Post Evaluate` route `/internal/elastic_assistant/evaluate` to be backed by an OAS, and adds a basic `Get Evaluate` route for rounding out the enhancements outlined in elastic/security-team#8167 (to be in a subsequent PR). Changes include: * Migration of `Post Evaluate` route to OAS * Migration of `Post Evaluate` route to use versioned router * Extracted `evaluate` API calls from `x-pack/packages/kbn-elastic-assistant/impl/assistant/api/api.tsx` to `x-pack/packages/kbn-elastic-assistant/impl/assistant/api/evaluate/evaluate.tsx` * Co-located relevant `use_perform_evaluation` hook * Adds `Get Evaluate` route, and corresponding `use_evaluation_data` hook. Currently only returns `agentExecutors` to be selected for evaluation. * API versioning constants added to `x-pack/packages/kbn-elastic-assistant-common/impl/schemas/index.ts` * Adds new `buildRouteValidationWithZod` function to `x-pack/plugins/elastic_assistant/server/schemas/common.ts` for validating routes against OAS generated zod schemas. ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

jbudz added 2 commits January 31, 2024 11:54

Revert "[Security Solution] [Elastic AI Assistant] Adds internal `Get…

b201bf6

… Evaluate` API and migrates `Post Evaluate` API to OAS (elastic#175338)" This reverts commit 38f0a7a.

skip tests

e04f487

test

c57c220

fix errors

9de174f

jbudz added 2 commits January 31, 2024 12:29

set job number

046da29

bump parallelism

d6435e8

Reapply "[Security Solution] [Elastic AI Assistant] Adds internal `Ge…

b39cc87

…t Evaluate` API and migrates `Post Evaluate` API to OAS (elastic#175338)" This reverts commit b201bf6.

This reverts commit 38f0a7a.

64f4908

jbudz mentioned this pull request Jan 31, 2024

[Security Solution] [Elastic AI Assistant] Adds internal Get Evaluate API and migrates Post Evaluate API to OAS #175338

Merged

2 tasks

spong added 2 commits January 31, 2024 13:48

Revert "This reverts commit 38f0a7a."

6eacc9b

This reverts commit 64f4908.

Skips evaluate.test.tsx tests

ee8c315

Skips post_evaluate.test.ts test and removes common package import

6856015

Reverts to http.fetch

2d4161f

spong mentioned this pull request Jan 31, 2024

[Security Solution] [Elastic AI Assistant] Adds internal Get Evaluate API and migrates Post Evaluate API to OAS #176025

Merged

2 tasks

jbudz closed this Feb 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jest flakiness testing #176005

Jest flakiness testing #176005

jbudz commented Jan 31, 2024

jbudz commented Jan 31, 2024

jbudz commented Jan 31, 2024

jbudz commented Jan 31, 2024

jbudz commented Jan 31, 2024

jbudz commented Jan 31, 2024

jbudz commented Jan 31, 2024

spong commented Jan 31, 2024

spong commented Jan 31, 2024 •

edited

Loading

spong commented Jan 31, 2024

spong commented Jan 31, 2024

kibana-ci commented Jan 31, 2024

Jest flakiness testing #176005

Jest flakiness testing #176005

Conversation

jbudz commented Jan 31, 2024

jbudz commented Jan 31, 2024

jbudz commented Jan 31, 2024

jbudz commented Jan 31, 2024

jbudz commented Jan 31, 2024

jbudz commented Jan 31, 2024

jbudz commented Jan 31, 2024

spong commented Jan 31, 2024

spong commented Jan 31, 2024 • edited Loading

Notes:

Current suspicions:

Links:

Summary:

spong commented Jan 31, 2024

spong commented Jan 31, 2024

kibana-ci commented Jan 31, 2024

💔 Build Failed

Failed CI Steps

History

spong commented Jan 31, 2024 •

edited

Loading