test strategy for OpenSearch 1.3.0 release for bundled OSD

[tianleh]

Currently the integration test running from infra's Jenkins job has flaky results. While we are continuously improving the Docker image and the browser, we can in parallel prepare some alternatives with some automation for 1.3.0 release, I am proposing #118.

This has been consistently successful running from the Github action of FTRep e.g https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/1959984086

Once we have the periodic Github action, it can show the test results on x64 bundled artifacts. However, for arm64, there is no official support from Github action yet. Alternative is to use self-hosted runners. https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners

[tianleh]

cc @seraphjiang @kavilla @peterzhuamazon for input

[tianleh]

Send a PR #125 to run the tests in this repo every 3 hours by using the latest url. Browser: Chrome. Architecture: x64

[tianleh]

I am still working on the improvement on the infra's Docker image opensearch-project/opensearch-build#1743 and enhance the test notification opensearch-project/opensearch-build#1728

[tianleh]

Analyzed the flaky tests from the last failed runs https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions?query=event%3Aschedule

take a note of flaky tests

https://github.com/opensearch-project/opensearch-dashboards-functional-test/runs/5555902862?check_suite_focus=true

plugins/anomaly-detection-dashboards-plugin/create_detector_spec.js
plugins/anomaly-detection-dashboards-plugin/detector_list_spec.js
plugins/index-management-dashboards-plugin/policies_spec.js

https://github.com/opensearch-project/opensearch-dashboards-functional-test/runs/5555880932?check_suite_focus=true

plugins/index-management-dashboards-plugin/policies_spec.js
plugins/observability-dashboards/trace_analytics_traces.spec.js
plugins/anomaly-detection-dashboards-plugin/create_detector_spec.js

https://github.com/opensearch-project/opensearch-dashboards-functional-test/runs/5553193656?check_suite_focus=true

plugins/anomaly-detection-dashboards-plugin/create_detector_spec.js
plugins/anomaly-detection-dashboards-plugin/overview_spec.js
plugins/index-management-dashboards-plugin/policies_spec.js

https://github.com/opensearch-project/opensearch-dashboards-functional-test/actions/runs/1986518373

plugins/anomaly-detection-dashboards-plugin/create_detector_spec.js
plugins/anomaly-detection-dashboards-plugin/detector_list_spec.js
plugins/index-management-dashboards-plugin/policies_spec.js
plugins/observability-dashboards/app_analytics.spec.js
plugins/observability-dashboards/panels.spec.js

https://github.com/opensearch-project/opensearch-dashboards-functional-test/runs/5553166830?check_suite_focus=true

plugins/anomaly-detection-dashboards-plugin/create_detector_spec.js
plugins/anomaly-detection-dashboards-plugin/overview_spec.js

https://github.com/opensearch-project/opensearch-dashboards-functional-test/runs/5550839236?check_suite_focus=true
plugins/anomaly-detection-dashboards-plugin/create_detector_spec.js
plugins/anomaly-detection-dashboards-plugin/detector_configuration_spec.js
plugins/anomaly-detection-dashboards-plugin/overview_spec.js
plugins/index-management-dashboards-plugin/policies_spec.js

https://github.com/opensearch-project/opensearch-dashboards-functional-test/runs/5550824472?check_suite_focus=true
plugins/anomaly-detection-dashboards-plugin/create_detector_spec.js

https://github.com/opensearch-project/opensearch-dashboards-functional-test/runs/5550820572?check_suite_focus=true
plugins/observability-dashboards/panels.spec.js

https://github.com/opensearch-project/opensearch-dashboards-functional-test/runs/5549122648?check_suite_focus=true
plugins/index-management-dashboards-plugin/policies_spec.js
plugins/observability-dashboards/app_analytics.spec.js
plugins/reports-dashboards/03-details.spec.js

https://github.com/opensearch-project/opensearch-dashboards-functional-test/runs/5549109928?check_suite_focus=true
plugins/anomaly-detection-dashboards-plugin/create_detector_spec.js
plugins/index-management-dashboards-plugin/policies_spec.js
plugins/observability-dashboards/panels.spec.js

[tianleh]

I can see some patterns for the frequent failed ones. I will send a PR to skip them for now.

[tianleh]

Flaky tests are tracked by plugin owners. Close this one who was originally for 1.3.0 release