[BUG][Main] Fix flaky test cases for bundled OpenSearch Dashboards plugins

[tianleh]

What is the bug?

We have noticed flaky test when running the test cases for Dashboards plugins. To collect enough data points for analysis, we have set up multiple Github workflows against different browsers every three hours. By analyzing the past 10 failed runs, below is a summary of the frequency of failure.

Test File	Plugin Owner	Failed Times (out of 10)
plugins/anomaly-detection-dashboards-plugin/create_detector_spec.js	anomaly-detection-dashboards	8
plugins/anomaly-detection-dashboards-plugin/detector_list_spec.js	anomaly-detection-dashboards	2
plugins/index-management-dashboards-plugin/policies_spec.js	index-management-dashboards	6
plugins/observability-dashboards/trace_analytics_traces.spec.js	observability-dashboards	1
plugins/anomaly-detection-dashboards-plugin/overview_spec.js	anomaly-detection-dashboards	3
plugins/anomaly-detection-dashboards-plugin/detector_configuration_spec.js	anomaly-detection-dashboards	1
plugins/observability-dashboards/panels.spec.js	observability-dashboards	3
plugins/observability-dashboards/app_analytics.spec.js	observability-dashboards	2
plugins/reports-dashboards/03-details.spec.js	reports-dashboards	1

How can one reproduce the bug?
https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions?query=event%3Aschedule

What is the expected behavior?
We shall see all test cases pass.

What is your host/environment?

• OS: Github hosted Ubuntu latest

Do you have any screenshots?
Every workflow will upload the video. Use this as an example https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/1987477734

Do you have any additional context?
Note that when running from Jenkins Docker env, we are seeing more flaky test cases due to the less empowered execution env. We will consolidate and share the results later. E.g a recent Jenkins run has the following result (You may need to save the file locally or use curl to view it)
https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.0/1511/linux/x64/test-results/d550538e3386451fbcccea7c0ecf3d46/integ-test/functionalTestDashboards/with-security/test-results/stdout.txt

https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.0/1511/linux/x64/test-results/d550538e3386451fbcccea7c0ecf3d46/integ-test/functionalTestDashboards/without-security/test-results/stdout.txt

https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.0/1511/linux/arm64/test-results/f83b7e7b4ef14a4e9b5cc80a00745347/integ-test/functionalTestDashboards/with-security/test-results/stdout.txt

https://ci.opensearch.org/ci/dbc/integ-test-opensearch-dashboards/1.3.0/1511/linux/arm64/test-results/f83b7e7b4ef14a4e9b5cc80a00745347/integ-test/functionalTestDashboards/without-security/test-results/stdout.txt

[tianleh]

cc @dblock @peterzhuamazon @seraphjiang

[tianleh]

@ohltyler for AD
@bowenlan-amzn for ISM
@joshuali925 for observability
@davidcui1225 for reports

[tianleh]

@joshuali925 shared a tip to handle the interaction with dropdown component

opensearch-dashboards-functional-test/cypress/integration/plugins/observability-dashboards/notebooks.spec.js

Line 272 in c76fdc9

cy.get('.euiFieldSearch').focus().type('[Flights] Flight Count and Average Ticket Price{enter}');

[ohltyler]

AD flakiness was cleaned up in #144 besides one test that may be related to the Cypress upgrade (link). I've opened #147 to track that. I don't believe this is flaky since it consistently fails, so I'd consider it entirely related to the upgrade efforts.

[wanglam]

@tianleh We've run all tests based 2.11 branch for above plugins. After 5 attempts, no failed cases found.
Here is the job details:

• Without security enabled
  AD: https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/7244651329/job/20564369091
  Reports: https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/7244651322/job/20566660833
  Observability: https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/7244651334/job/20567098056
  Index Management: https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/7244651364/job/20566657988
• With security enabled
  AD: https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/7554212666/job/20599961908?pr=1000
  Reports: https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/7554212687/job/20599957422?pr=1000
  Observability: https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/7554212668/job/20599959519?pr=1000
  Index Management: https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/7554212684/job/20599956092?pr=1000

The test cases failed in the every 3 hours release sign off workflow may caused by share the same OpenSearch or OpenSearch Dashboards. Some tests may forgot to reset the operated data. There are two ways to make sure each functional tests of specific plugin run in isolate env:

1. Modify the workflow, run new OpenSearch and OpenSearch Dashboards before plugin tests start.
2. Inspired by this remote functional tests RFC, trigger functional tests in their own workflows.
   These two ways can provide an isolate OpenSearch and OpenSearch Dashboards for functional tests. It may reduce the differences between the every 3 hours workflow and the PR workflow.

[wanglam]

@tianleh Since there are no failed cases in these plugins, we will close this issue first. Will create new issue to track flaky tests case by case.