Nightly FTR runs with Jenkins

[Swiddis]

Is your feature request related to a problem? Please describe

Currently, the FTR tests are primarily executed during release cycles and when a pull request is submitted to the FTR repository. Unfortunately, this approach means that potential test failures in FTR often go unnoticed during the development phase, preceding the generation of the Release Candidate. Consequently, these issues seem to typically remain unaddressed until the release window has already opened. This puts pressure on the release process, leading to hurried fixes.

Even if a dev checks on FTR occasionally, they might not see that their tests are failing because the page for them only shows green:

I think it would be more effective to identify and address these issues as soon as they arise, rather than waiting for the release window. Failures in FTR can occur for reasons like frontend modifications that aren't correctly ported, upstream changes, and actual bug introduction, all of which can be better diagnosed and addressed with a more regular testing dashboard.

Describe the solution you'd like

To enhance the efficiency of our development and release processes, I propose that FTR runs be incorporated into nightly CI routines. By doing so, issues with FTR tests can be detected and addressed promptly during the development, significantly reducing the amount of rushed repairs during release windows. I think this kind of proactive approach will help put less strain on the release process overall.

Describe alternatives you've considered

There are a few alternatives I've thought about:

1. Increased validation at the time of PR. This works well to find developer errors quickly and help to avoid tests within our repositories becoming obsolete over time, but doesn't really apply to the FTR build. It's also the case that we generally don't run our tests in local CI under the same conditions as the tests in Jenkins, which can lead to tests that are fine for us being flaky at release time.

2. Nightly runs in any plugin repository that's willing to set it up. This also could work, but I think that having a more universal process (bonus points for dashboards) would be more effective. We still have the issue of differing tests environments between GH actions and Jenkins (not to mention ToD), and this seems to rely a lot on good intentions compared to maintaining runs in one place.

Additional context

I put this issue here instead of FTR since it seems more related to the overall build process than an issue localized to the test repository, but I'm happy to have the issue moved.

[jordarlu]

Hello, @Swiddis ,if you don't mind me to ask a quick question, I learn that when you have a new FT test, you will have it run at FT repo CI checks when issuing a PR, and if you found it executed suceesfully before getting it merged? We want to make sure the errors if any been caught duing the CI checks first ... before reaching to the Jenkins ,,

[Swiddis]

Hi, yes, we make sure that the FTR tests are passing when the PR is merged, but it looks like this isn't enough when code changes over time or if tests are flaky and only pass during the initial checks. That's why I'm proposing we also have nightly runs, to track if tests are flaky or drifting out of date.

[jordarlu]

thanks for the feedback ,,, we actaully have the CI running the integ-test which will trigger FT test in evry build daily ( or triggerd by Github PRs ) .. for example : https://build.ci.opensearch.org/blue/organizations/jenkins/distribution-build-opensearch-dashboards/detail/distribution-build-opensearch-dashboards/6906/pipeline/726/ ( 3.0.0 for example ) ,,so, it should pick up the PR merged in FT repo.
CC : @kavilla for furtrher insight

[gaiksaya]

Based on above comment, moving the issue to FT repo. For CI based checks I believe FT repo maintainer's would the right people to take the call. For distribution level tests for all plugins hosted in FT repo, you check the status' here: https://build.ci.opensearch.org/view/Test/job/integ-test-opensearch-dashboards/
Related jenkins file: https://github.com/opensearch-project/opensearch-build/blob/main/jenkins/opensearch-dashboards/integ-test.jenkinsfile