[META] CI Improvement Plan #4019

seanneumann · 2023-05-12T17:15:33Z

We'd like to improve our CI for OpenSearch Dashboards. Creating this issue to start forming the plan.

Initial AI: baseline metrics to measure progress against and establish goal.

Standardized plugin CI
- Problem
  - Each plugin (especially in the OpenSearch project org) has defined their own GitHub CI workflows which vary from repo to repo. Sometimes a repo will lack verification within release builds which then leads to bugs only being caught close to release or do not utilize available scripts that can make their test runs more consistent or faster.
- Solution
  - Provide scripts (potentially housed and imported by the SDK or within the plugin helpers package) to allow for plugins to build and verify to standardized what is being executed and called from CI workflows.
- Tasks
  - [CI] Build and verify release artifacts for plugins for each OS #3918
Reduce monolith script
- Problem
  - Build and test CI workflow is massive and there is a lot of replication between the steps. It makes it difficult to contribute, improve, and fix whenever there are issues.
- Solution
  - Breakdown the script into components and templates
- Tasks
Reduce flakiness of tests
Ensure timeouts of tests
Migrate to Selenium tests to Cypress tests
- Screenshot/video upload to PR (already occurring but make it a comment for ease of access of maintainers to quickly review any potential UI regressions)
Failed backport label automation
Accommodate Different Node.js Versions (14, 16, 18) in Test Suite #4129

kavilla · 2023-09-01T17:32:40Z

Device matrix testing #4803

Utilizing the GitHub `workflow_dispatch`, we can trigger a job with a specific inputs. By default, we run from this repo: https://github.com/opensearch-project/opensearch-dashboards-functional-test This workflow uses the target branch of the PR to pull down from the FTRepo and run the tests from that specific branch. For example, PRs against `main` will pull down `main` from the FTRepo. The problem occurs when new functionality is opened or a bug is fixed and, per industry standard, we want to see tests added to ensure functionality, stability, and raise the bar. However, the cypress tests PR into the FTRepo depends on the new code to be merged for the CI within the FTRepo to work. So we have a stalemate that usually slows down PR review time. Here, a manual run can be triggered by a maintainer. It will utilize the source branch from the PR if provided. Related to: opensearch-project#4019 Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

Utilizing the GitHub `workflow_dispatch`, we can trigger a job with a specific inputs. By default, we run from this repo: https://github.com/opensearch-project/opensearch-dashboards-functional-test This workflow uses the target branch of the PR to pull down from the FTRepo and run the tests from that specific branch. For example, PRs against `main` will pull down `main` from the FTRepo. The problem occurs when new functionality is opened or a bug is fixed and, per industry standard, we want to see tests added to ensure functionality, stability, and raise the bar. However, the cypress tests PR into the FTRepo depends on the new code to be merged for the CI within the FTRepo to work. So we have a stalemate that usually slows down PR review time. Here, a manual run can be triggered by a maintainer. It will utilize the source branch from the PR if provided. Related to: #4019 Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

Utilizing the GitHub `workflow_dispatch`, we can trigger a job with a specific inputs. By default, we run from this repo: https://github.com/opensearch-project/opensearch-dashboards-functional-test This workflow uses the target branch of the PR to pull down from the FTRepo and run the tests from that specific branch. For example, PRs against `main` will pull down `main` from the FTRepo. The problem occurs when new functionality is opened or a bug is fixed and, per industry standard, we want to see tests added to ensure functionality, stability, and raise the bar. However, the cypress tests PR into the FTRepo depends on the new code to be merged for the CI within the FTRepo to work. So we have a stalemate that usually slows down PR review time. Here, a manual run can be triggered by a maintainer. It will utilize the source branch from the PR if provided. Related to: #4019 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit 34a8594) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md

Originally: opensearch-project#5134 I believed it to be a good idea to ensure that triggering the CI should always run a batch of tests. But I realized that some tests can be added and just be bad tests until updated. Or if we want to verify a flaky test in the CI. So instead of only making it additional tests, the spec input will complete replace the default tests (unless you append to the default). Empty will run the default spec still. This will append on to the PR comment with the spec that was run. So at least maintainers can see it passed for a set of tests but it did not run the ones we usually run. Also adding some formatting for empty values for the PR comment. Issue related: opensearch-project#4019 Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

Originally: #5134 I believed it to be a good idea to ensure that triggering the CI should always run a batch of tests. But I realized that some tests can be added and just be bad tests until updated. Or if we want to verify a flaky test in the CI. So instead of only making it additional tests, the spec input will complete replace the default tests (unless you append to the default). Empty will run the default spec still. This will append on to the PR comment with the spec that was run. So at least maintainers can see it passed for a set of tests but it did not run the ones we usually run. Also adding some formatting for empty values for the PR comment. Issue related: #4019 Signed-off-by: Kawika Avilla <kavilla414@gmail.com>

Originally: #5134 I believed it to be a good idea to ensure that triggering the CI should always run a batch of tests. But I realized that some tests can be added and just be bad tests until updated. Or if we want to verify a flaky test in the CI. So instead of only making it additional tests, the spec input will complete replace the default tests (unless you append to the default). Empty will run the default spec still. This will append on to the PR comment with the spec that was run. So at least maintainers can see it passed for a set of tests but it did not run the ones we usually run. Also adding some formatting for empty values for the PR comment. Issue related: #4019 Signed-off-by: Kawika Avilla <kavilla414@gmail.com> (cherry picked from commit 6154e2d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Originally: #5134 I believed it to be a good idea to ensure that triggering the CI should always run a batch of tests. But I realized that some tests can be added and just be bad tests until updated. Or if we want to verify a flaky test in the CI. So instead of only making it additional tests, the spec input will complete replace the default tests (unless you append to the default). Empty will run the default spec still. This will append on to the PR comment with the spec that was run. So at least maintainers can see it passed for a set of tests but it did not run the ones we usually run. Also adding some formatting for empty values for the PR comment. Issue related: #4019 (cherry picked from commit 6154e2d) Signed-off-by: Kawika Avilla <kavilla414@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

seanneumann added the enhancement New feature or request label May 12, 2023

github-actions bot added the untriaged label May 12, 2023

seanneumann removed the untriaged label May 12, 2023

seanneumann self-assigned this May 12, 2023

seanneumann added the ci label May 12, 2023

kavilla mentioned this issue May 16, 2023

Updates PR template for screenshots and test instructions #4042

Merged

8 tasks

sejli mentioned this issue Jun 13, 2023

[CI] Test builds from zip files opensearch-project/dashboards-search-relevance#184

Open

kavilla mentioned this issue Sep 27, 2023

[CI] allow for manual triggered cypress test jobs to accept inputs #5134

Merged

8 tasks

kavilla mentioned this issue Sep 28, 2023

[CI] Cypress test run spec input #5142

Merged

8 tasks

kavilla mentioned this issue Oct 2, 2023

[CI][Cypress] set NODE_OPTIONS and cluster disk allocation #5172

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[META] CI Improvement Plan #4019

[META] CI Improvement Plan #4019

seanneumann commented May 12, 2023 •

edited by kavilla

Loading

kavilla commented Sep 1, 2023 •

edited

Loading

[META] CI Improvement Plan #4019

[META] CI Improvement Plan #4019

Comments

seanneumann commented May 12, 2023 • edited by kavilla Loading

kavilla commented Sep 1, 2023 • edited Loading

seanneumann commented May 12, 2023 •

edited by kavilla

Loading

kavilla commented Sep 1, 2023 •

edited

Loading