Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Explore, fix flakey Cypress tests #161874

Closed
Tracked by #153664
stephmilovic opened this issue Jul 13, 2023 · 7 comments
Closed
Tracked by #153664

[Security Solution] Explore, fix flakey Cypress tests #161874

stephmilovic opened this issue Jul 13, 2023 · 7 comments
Assignees
@semd
Labels
bug Fixes for quality problems that affect the customer experience impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Threat Hunting:Explore Team:Threat Hunting Security Solution Threat Hunting Team technical debt Improvement of the software architecture and operational architecture

Comments

@stephmilovic
Copy link
Contributor

stephmilovic commented Jul 13, 2023 •
edited

Update: The noted tests are skipped. Unskip when fixed: #161898

I've noticed an increase in flakiness in our cypress tests. Looking at the #kibana-unsupported-ftrs-alerts slack channel along with the flakey test runner to compile the list below:

  • [job] [logs] Explore - Security Solution Tests / Entity Analytics Dashboard With anomalies data enables a job
  • [job] [logs] Explore - Security Solution Tests / Entity Analytics Dashboard With anomalies data renders table with pagination
  • [job] [logs] Security Solution Explore - Cypress Dashboard loading error #27 / Entity Analytics Dashboard With host risk data filters by risk classification
  • [job] [logs] Explore - Security Solution Tests / Hover actions Show topN
  • [job] [logs] Explore - Security Solution Tests / Hover actions Adds global filter - filter in
  • [job] [logs] Explore - Security Solution Tests / Hover actions Adds global filter - filter out
  • [job] [logs] Security Solution Tests / Hover actions Copy value <-- why is this not labeled explore?
  • [job] [logs] Explore - Security Solution Tests / Hover actions Adds to timeline
@stephmilovic stephmilovic added bug Fixes for quality problems that affect the customer experience Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Threat Hunting Security Solution Threat Hunting Team Team:Threat Hunting:Explore labels Jul 13, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

@stephmilovic stephmilovic added technical debt Improvement of the software architecture and operational architecture impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. labels Jul 13, 2023
@stephmilovic stephmilovic self-assigned this Jul 13, 2023
@stephmilovic stephmilovic mentioned this issue Jul 13, 2023
Merged
@MadameSheema MadameSheema mentioned this issue Jul 14, 2023
Closed
12 tasks
@machadoum machadoum mentioned this issue Aug 3, 2023
Merged
machadoum added a commit that referenced this issue Aug 7, 2023
@machadoum
[Security Solutions] Flaky entity analytics cypress (#162963)
5a3e33d 
Original issue: #161874

## Summary

Flaky entity analytics cypress test

Based on my analysis, it seems likely that the issue was caused by some
tests failing to close the filter after opening it. This could have had
an impact on the subsequent tests.
After the change, I could run the test successfully more than 60 times.
Here is a print from some of these runs: 
<img width="587" alt="Screenshot 2023-08-04 at 13 58 55"
src="https://github.com/elastic/kibana/assets/1490444/02367810-3616-4fa2-a82e-48c19e6ac4b6">
crespocarlos pushed a commit to crespocarlos/kibana that referenced this issue Aug 8, 2023
@machadoum @crespocarlos
[Security Solutions] Flaky entity analytics cypress (elastic#162963)
a3ee61f 
Original issue: elastic#161874

## Summary

Flaky entity analytics cypress test

Based on my analysis, it seems likely that the issue was caused by some
tests failing to close the filter after opening it. This could have had
an impact on the subsequent tests.
After the change, I could run the test successfully more than 60 times.
Here is a print from some of these runs: 
<img width="587" alt="Screenshot 2023-08-04 at 13 58 55"
src="https://github.com/elastic/kibana/assets/1490444/02367810-3616-4fa2-a82e-48c19e6ac4b6">
bryce-b pushed a commit to bryce-b/kibana that referenced this issue Aug 9, 2023
@machadoum @bryce-b
[Security Solutions] Flaky entity analytics cypress (elastic#162963)
d2952db 
Original issue: elastic#161874

## Summary

Flaky entity analytics cypress test

Based on my analysis, it seems likely that the issue was caused by some
tests failing to close the filter after opening it. This could have had
an impact on the subsequent tests.
After the change, I could run the test successfully more than 60 times.
Here is a print from some of these runs: 
<img width="587" alt="Screenshot 2023-08-04 at 13 58 55"
src="https://github.com/elastic/kibana/assets/1490444/02367810-3616-4fa2-a82e-48c19e6ac4b6">
@stephmilovic stephmilovic added impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. and removed impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. labels Aug 17, 2023
@stephmilovic
Copy link
Contributor Author

Tried for a few days on this and still cannot pass CI: #163989

I can only get the failures on CI, not locally. I can throttle CPU and/or network and get different failures… but I cannot reproduce the CI failure locally. I’m stuck, going to put this back on the shelf for now.

@stephmilovic
Copy link
Contributor Author

Resolved by #166492

@VidhiRambhia
Copy link
Contributor

Is this still an issue?

@stephmilovic stephmilovic mentioned this issue Dec 11, 2023
Open
@stephmilovic
Copy link
Contributor Author

latest flakey test run: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/4418

@stephmilovic
Copy link
Contributor Author

still valid:
#165692

@semd semd mentioned this issue Feb 23, 2024
Merged
@semd semd assigned semd and unassigned stephmilovic Feb 23, 2024
semd added a commit that referenced this issue Mar 4, 2024
@semd @kibanamachine @angorayc
[Security Solution] Unskip flacky cellActions test (#177577)
cd6bc5e 
## Summary

part of: #161874
Fixes hover actions flaky cypress test. 

Flaky test runner execution after changes:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5293

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Angela Chuang <6295984+angorayc@users.noreply.github.com>
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Mar 4, 2024
@semd
[Security Solution] Unskip flacky cellActions test (elastic#177577)
e6cd1f3 
## Summary

part of: elastic#161874
Fixes hover actions flaky cypress test.

Flaky test runner execution after changes:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5293

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Angela Chuang <6295984+angorayc@users.noreply.github.com>
(cherry picked from commit cd6bc5e)
semd added a commit to semd/kibana that referenced this issue Mar 4, 2024
@semd @kibanamachine @angorayc
[Security Solution] Unskip flacky cellActions test (elastic#177577)
2e1375d 
## Summary

part of: elastic#161874
Fixes hover actions flaky cypress test. 

Flaky test runner execution after changes:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5293

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Angela Chuang <6295984+angorayc@users.noreply.github.com>
fkanout pushed a commit to fkanout/kibana that referenced this issue Mar 4, 2024
@semd @kibanamachine
@angorayc @fkanout
[Security Solution] Unskip flacky cellActions test (elastic#177577)
fc1dea2 
## Summary

part of: elastic#161874
Fixes hover actions flaky cypress test. 

Flaky test runner execution after changes:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5293

---------

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Angela Chuang <6295984+angorayc@users.noreply.github.com>
@semd semd mentioned this issue Mar 6, 2024
Merged
semd added a commit that referenced this issue Mar 6, 2024
@semd
[Security Solution] Unskip flacky hover action test (#178109)
0f5b804 
## Summary

closes: #165692
part of: #161874

Hover action flakiness was fixed here
#177577

This PR unskips the test and enables it for serverless execution.

Flaky test runner execution after changes:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5401
@semd semd closed this as completed Mar 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@semd semd

Labels
bug Fixes for quality problems that affect the customer experience impact:medium Addressing this issue will have a medium level of impact on the quality/strength of our product. Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Threat Hunting:Explore Team:Threat Hunting Security Solution Threat Hunting Team technical debt Improvement of the software architecture and operational architecture
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@stephmilovic @elasticmachine @semd @VidhiRambhia