[POC] Create POC for functional test repo to trigger remote runners running cypress tests

[manasvinibs]

Proof of Concept (POC) for Running Tests Outside the Functional Test Repository

Reference design: #5392

Changes in the Functional Test Repository:

In the Functional Test repository, I've implemented a wrapper script around a GitHub Actions dispatch event. This script triggers a remote workflow outside the Functional Test repository. For this POC, I've utilized my forked Opensearch-Dashboards repository, which includes functional tests for Dashboards.

1. GitHub Workflow in the Functional Test Repo:

   • Functional Test Repo Workflow

2. GitHub Workflow in OpenSearch-Dashboards Repository:

   • This workflow is triggered from the Functional Test repo using the GitHub workflow_dispatch event. It hosts a GitHub runner to execute Cypress tests located in the same repository.
   • OpenSearch-Dashboards Repo Workflow

Code Changes in Functional Test Repo:

Remote Cypress Script (remoteCypress.sh):

The script builds the payload required to trigger remote workflows, executing Cypress functional tests in plugin repositories. Details for the payload are typically provided by the manifest.yml file, which includes information such as GitHub endpoint, ref, name, and matrix information for a specific build type.

This script returns either 0 (success) or 1 (failure) based on the status of the remote runner execution completion status by constantly polling the remote workflow for a set period of time. This can be used by build CI to know if the component(plugin) tests part of the release workflow has succeeded or not.

• PR with Code Changes

Sample Cypress Test Setup in OSD Repo:

• Sample Cypress Test Setup

Sample manifest.yml file provided by build CI:

schema-version: '1.1'
build:
  name: OpenSearch Dashboards
  version: 2.11.0
  platform: linux
  architecture: x64
  distribution: tar
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.11.0/6665/linux/x64/tar/dist/opensearch-dashboards/opensearch-dashboards-2.11.0-linux-x64.tar.gz
  id: '6665'
components:
  - name: OpenSearch-Dashboards
    repository: https://github.com/opensearch-project/OpenSearch-Dashboards.git
    ref: 3c838160fe6b453881ba326c6c8f57e11cf10fdd
    commit_id: 3c838160fe6b453881ba326c6c8f57e11cf10fdd
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/2.11.0/6665/linux/x64/tar/builds/opensearch-dashboards/dist/opensearch-dashboards-min-2.11.0-linux-x64.tar.gz
   .............
   ...............
   .............
 

OpenSearch-Dashboards Changes:

I've added a simple sanity test to a component repository. For the POC, I've pulled some existing sanity Dashboards tests from the functional test repo and included them within the OpenSearch-Dashboards repo. In the POC, I've defined a dependency on the Cypress library in the package.json file, with the Cypress version matching the one in the FT repo. In the actual implementation, the Functional Test repo should install all necessary libraries, including Cypress, which all plugin components indirectly depend on through the Functional Test repo dependency.

Test Results:

1. A sample GitHub workflow template is provided, uploading test artifacts to S3. Infra/build teams can read the results from the S3 bucket. Permissions required for S3 uploads can be managed by creating a GitHub action in one repository and reused by plugin teams' workflows.
2. Alternatively, each plugin component can use a test artifacts analytics tool like Codecov to upload test results. This requires infra to modify their existing integration test script to collect results from the specific tool.