[META] Test orchestration framework

[setiah]

This is a meta issue for defining the test orchestration framework as part of automated test infra

Summary of Action plan

todo

Roles and Responsibilities

1. Orchestration framework will provide s3 location to publish logs as part of configuration parameters for a job. It is the responsibility of each job to publish its own logs to that s3 location.

Tasks

Please note these P0s are defined according to prod readiness for 1.1 release

P0

• Define syntax of manifest file and S3 location. See Notifications/Hooks to different systems. #111
• Create pipeline with 2 stages - artifact reader, test executor #173
• Create artifact reader stage, that reads artifacts from s3, reads manifest file for each artifact and extracts parameters to run testing jobs.
• Create test executor stage, that uses those parameters to kick off multiple parallel testing jobs on provided artifact.
• Setup git credentials for Jenkins staging #176
• Setup orchestration pipeline on staging #175
• Jobs to publish test results to S3 bucket #174
• Setting up right environment on jenkins for running the tests #317
• Implement "DependencyInstaller" #283
• Setup local test cluster from the s3 bundle #384
• Make pipeline parameterized to take S3 manifest as input parameter #262
• Make all test jobs parameterized #263
• Setup maven and build dependencies using DependencyInstaller #276
• Copy manifest and bundle from S3 #260
• Support notification/hooks to different systems Test Orchestrator for OpenSearch project #107
• Meta: Implement results publisher #207

P0 - non surge

• Run jobs on agent nodes instead of master #221

P1

• Integrate with cdk based cluster to parallelize tests for plugins #257
• Trigger automated tests after build completes #1184
• Trigger notifications after test run #211