- OpenSearch Releas Process
OpenSearch Project: OpenSearch is a community-driven, Apache 2.0-licensed open source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data.
OpenSearch/OpenSearch Dashboards Bundle: An OpenSearch/OpenSearch Dashboards bundle refers to a compressed file that encompasses both the OpenSearch/OpenSearch Dashboards distribution and accompanying plugins. This compressed file acts as a container for the OpenSearch/OpenSearch Dashboards software and its associated extensions, enabling users to conveniently package and deploy the entire system.
OpenSearch/OpenSearch Dashboards Components/Plugins: OpenSearch/OpenSearch Dashboards components/plugins are extensions that can be added to OpenSearch/OpenSearch Dashboards, to add new features or functionality. There are a wide variety of plugins available refer OpenSearch available plugins and OpenSearch Dashboards available plugins sections.
These manifests serve as the initial input for driving the build workflow. These manifests are located in build repository and are generated by the manifest workflow. Once the manifest is created, the maintainers of the build repository review and merge it into the main branch. The entire build repository operates exclusively on the main branch.
Output of the build workflow, used as input to the assemble workflow.
| deb | rpm | tar | windows |
|---|---|---|---|
| os-arm64, os-x64 | os-arm64, os-x64 | os-x64, os-arm64 | os-x64 |
The final output of the assemble workflow and manifest that is added to the final distribution, this has the commitID (can be used for reproducible builds) and the artifact file links. This final bundle manifest incorporates the assembled components and ensures traceability through the commit ID and accessibility to the artifact files.
| deb | rpm | tar | windows |
|---|---|---|---|
| os-arm64, os-x64 | os-arm64, os-x64 | os-x64, os-arm64 | os-x64 |
These are the issues created by automation with the distribution build and integ-test worklows failure, the automation detects the component failure and raises an issue in the respective component repo. Sample integ-test failure AUTOCUT issue and distribution build failure AUTOCUT issue.
| Wokflow | Description |
|---|---|
| Check for Build | Workflow that peridically triggers the distribution workflows using parameterized cron. |
| OpenSearch Distribution Build | Workflow that is responsible to build/assemble the OpenSearch and its components. |
| OpenSearch Dashboards Distribution Build | Workflow that is responsible to build/assemble the OpenSearch Dashboards and its components. |
| OpenSearch Integ Test | Workflow that runs integ tests for OpenSearch and its components. |
| OpenSearch Dashboards Integ Test | Workflow that runs integ tests for OpenSearch Dashboards and its components. |
| Performance Tests | Workflow that runs Performance tests on a cluster created with a given version. |
| BWC Tests | Workflow that runs backward compatibility tests on a cluster created with a given version. |
| RPM Validation | Workflow that validates the RPM distribution |
| Docker Build | Workflow that builds the OpenSearch and OpenSearch Dashboards docker images |
| Docker Copy | Workflow that copies the created docker images to multiple DockerHub and ECR repositories |
| Docker Scan | Workflow that checks vulnerabilities for a given docker image as an input. |
| Wokflow | Description |
|---|---|
| Release Notes Tracker | Workflow that identifies if a component has a release notes added based on the commit history. |
| Promote Repos | Workflow that signs and promotes the APT/YUM repos to the production buckets accessed via the cloudfront. |
| Promote artifacts | Workflow that signs and promotes all the release artifcats to the production buckets accessed via the cloudfront. |
| Publish to Maven | Workflow that signs and publishes to the central maven repository. |
| Docker Promotion | Workflow that promotoes the docker images to production docker repositories. |
| Validation Workflow | Workflow that validates the released distribution. |
Each new OpenSearch release process starts when any one component increments a version, typically on the main branch. For example, OpenSearch#1192 incremented the version to 2.0. The version check automation workflow will notice this change or it can be triggered manually, and make a pull request (e.g. opensearch-build#514) that adds a new manifest (e.g. opensearch-2.0.0.yml. After that's merged, a GitHub issue is automatically opened by this workflow to make a new release using this release template (e.g. opensearch-build#566). Existing and new components (re)onboard into every release by submitting pull requests to each version's manifest.
The release managers to a specific OpenSearch release will be assigned through volunteer model. The request for release managers will be posted in OpenSearch public slack workspace (under releases channel) and selected on first come first served (FCFS) model. Note: The release managers should be maintainers of a repo under OpenSearch GitHub organization.
Ensure the label is created in each component repo for this new version, and the next minor release. Create a version label. This is usally handled by each respective component repo owner, however it is essential for a release manager to verify.
This issue captures the state of the OpenSearch release, its assignee is responsible for driving the release. Please contact them or @mention them on this issue for help. There are linked issues on components of the release where individual components can be tracked. More details are included in the Maintainers Release owner section.
The release issue is created by an automation workflow. Once the release manager is finalized, the release manager should be updating the created release issue. Sample Release Issue 2.8.0. Update the release issue issue so all __REPLACE_RELEASE-__ placeholders have actual dates.
Increase the build frequency for the this release from once a day (H 1 * * *) to once every hour (H/60 * * * *) in check-for-build.jenkinsfile. This will ensure the Distribution Build workflow is called every hour to build and detect the components failure early which are part of the Input Manifest.
This section is not required for a patch release.
Create a release issue in each component repository and then update the "components" section of the release issue with the corresponding release issues generated in each component repository. Refer to the sample component issues for a specific release. Utilize the existing component_release_template.md to generate the component release issue. Make sure that all the release issues created earlier are linked to this release issue and assigned to a designated owner within the component team.
If exists any release specific issues/campaigns link it back to the release issue. Sample linked issues/campaigns
The Release Branch Readiness date is determined as the release date minus 14 days.
Not applicable for patch release.
This step requires both OpenSearch and OpenSearch Dashboards to create a release branch that will be used for the release.
This step requires that every team participating in a release has their release branch created for the corresponding release by the date listed on this step. The Distribution Build workflow will also start using the release branch to create release candidate instead of .x branches.
Versions are incremented as soon as development starts on a given version to avoid confusion. Following is the example that depicts the version increment.
- OpenSearch:
main= 3.0.0,2.x= 2.9.0, and2.8= 2.8.1 - common-utils:
main= 3.0.0.0,2.x= 2.9.0.0 and2.8= 2.8.1.0
To ensure the version incrementation process is handled correctly, it is important to follow to increment the version of the release branch. Currently, the version incrementation is being done manually by an individual from the core repositories. Sample OpenSearch Version Increment PR for the release branch.
The next step is to identify the pending component version increment pull requests. The creation of component version increment pull requests is automated with workflows running in the build repo OpenSearch Version Increment Workflow, OpenSearch Dashboards Version Increment Workflow. Sample component version increment PR.
The objective is to merge these pull requests in order to synchronize all the components with the core version. It is not as simple as submitting a pull request and having it merged. When a pull request is submitted, CI checks are triggered for each repository. These CI checks assess the dependencies. Thus, prior to merging the pull requests that involve version increments, the release manager must confirm that the dependencies have been updated, built, and pushed to Maven and S3. This crucial stage requires the release manager to coordinate with the component teams, address any dependency issues, and ultimately merge the pull requests that involve version increments.
Coordinate with the Core and component teams to ensure that the code for this particular release version is fully prepared and that the corresponding branch has been included in the release version Input Manifest. Update Jenkins workflows that execute daily snapshot builds for both OpenSearch and OpenSearch Dashboards (Ref Increase the build frequency). Submit pull requests to incorporate each component into the respective version level Input Manifest along with the necessary checks. Sample PR. The Feature Freeze date is determined as the release date minus 12 days and the Code Complete date is determined as the release date minus 10 days.
Refer the Build Workflows section to get the details about the distribution build job for OpenSearch and OpenSearch Dashboards. This section covers how to sequentially handle the distribution build job during the release.
Ensure the proper inputs are used to initiate the distribution. For instance, here's an example for OpenSearch. It's important that the OpenSearch Dashboard distribution aligns accordingly.
COMPONENT_NAME: To trigger a specific component, this includes standalone OpenSearch or specific plugin.
INPUT_MANIFEST: The release input manifest that drives the workflow.
TEST_MANIFEST: The release test input manifest that is used for the integ tests.
INTEG_TEST_JOB_NAME: The integ test job name. Default already added to this input integ-test for OpenSearch and integ-test-opensearch-dashboards for OpenSearch Dashboards.
BUILD_PLATFORM: The input used to build for a specific platform, followed by its own distributions within the platform.
BUILD_DISTRIBUTION: Input to build selected distribution related artifacts, choices include 'tar', 'rpm', 'deb', 'zip'. Can combine multiple distributions with space in between (docker is only available on tar).
BUILD_DOCKER: Input with a dropdown that has 3 options build_docker, build_docker_with_build_number_tag, do_not_build_docker, the release manager has to take a call with right inputs.
UPDATE_LATEST_URL: To update the /latest CFN URL, Visit latest-distribution-url for more details.
Following is the order of execution of the distribution build to address the components dependencies.
Note: The execution order specified is necessary only for versions up to 1.3.x. For 2.x and above Maven dependencies are published through each component repository using the GH workflow. For more details check the META issue.
OpenSearch
OpenSearch + common-utils + job-scheduler
OpenSearch + common-utils + job-scheduler + performance-analyzer
OpenSearch + common-utils + job-scheduler + performance-analyzer + security
OpenSearch + common-utils + job-scheduler + ml-commons + performance-analyzer + security
All components (which are ready after completion of version increment)
OpenSearch Dashboards
All components (which are ready after completion of version increment)
Now once all the version increment PR鈥檚 are completed and all the components are part of the input manifest, now its time to generate the RC. Use the Distribution Build to generate the release candidate. Use the following section as the reference to generate the RC, validate it and broadcast it for a given release. The process of Release Candidate Generation and Testing should commence at least 6 days prior to the release date.
Following is the generated build number after triggering the Distribution Build workflow. The distribution build number denotes the RC, now with the example below the finalized RC鈥檚 are OS: 7848, OSD: 6126.
| OpenSearch | OpenSearch Dashboards |
|---|---|
| build_7858 | build_6126 |
Following are the TAR integ test jobs for a given RC based on the above section Sample Build details. The integ tests are executed for artifacts generated as part of the builds OS: 7848, OSD: 6126.
| Integ Test Tar | x64 | arm64 |
|---|---|---|
| OpenSearch | x64 | arm64 |
| OpenSearch Dashboards | x64 | arm64 |
Following are the RPM integ test jobs for a given RC based on the above section Sample Build details. The integ tests for rpm are executed for artifacts generated as part of the builds OS: 7848, OSD: 6126.
| Integ Test RPM | x64 | arm64 |
|---|---|---|
| OpenSearch | x64 | arm64 |
| OpenSearch Dashboards | x64 | arm64 |
Following are the details for the docker image build and scan. The docker images are built using the TAR artifact generated as part of the Distribution Build (From the above example OS: 7848, OSD: 6126). The Distribution Build workflow with input BUILD_DOCKER (Ref Workflow Trigger) triggers the [docker-build] workflow as dowmstream.
| Docker | build | scan |
|---|---|---|
| OpenSearch | Build | Scan |
| OpenSearch Dashboards | Build | Scan |
This to ensure that check-for-build.jenkinsfile wont re-build periodically and override the docker, the RC docker is created with build number. This step can be skipped if the input BUILD_DOCKER: build_docker_with_build_number_tag (Ref Workflow Trigger used in the Distribution Build.
| Docker Freeze | copy |
|---|---|
| OpenSearch | docker-copy |
| OpenSearch Dashboards | docker-copy |
For more details in running the performance tests refer Performance Tests section part of the test workflow.
| test_workload | sample_run |
|---|---|
| nyc_taxis | perf-test |
For more details in running the BWC tests refer Backwards Compatibility Tests section.
Currently, the windows integration tests for a release is manual. The manually tested windows zip is being evaluated and approved by the plugin teams for sign off. In order to test the windows distribution, two instances need to be created: one with security features enabled and another without security. Afterward, API calls should be tested by launching the OpenSearch and Dashboard processes through direct execution of the .bat file.
Broadcast the release candidate in OpenSearch public slack workspace and the release GitHub issue using format sample broadcast mesaage to gather votes.
As a release manager it is essential to ensure the successful completion of all the above mentioned jobs. In the event of failures during integration tests or scans, the release manager should collaborate with the component teams and initiate a re-run to ensure that all jobs are executed successfully.
Post all the job related failures in the Release issue, Sample post.
Note: Sometimes the integ-test jobs are flaky and might not pass due to several reasons with the component code, in that case coordinate with the respective component team and get a manual sign off. Sample manual sign off. All the failed logs are in s3 accessed through the cloudfront. Sample link.
Stop builds for this version of OpenSearch and/or OpenSearch Dashboards in order to avoid accidental commits going in unknowingly. Restart only if necessary else manually run the build workflow and declare new release candidate.
Once the RC is finalized, in order to exclude the release from running periodically, at this point it is necessary for the release manager to lock the input manifest and update the check-for-build.jenkins to remove it from the scheduled execution, sample PR.
Ensure that all pre-release activities listed below are completed before proceeding with the final release. These activities hold great significance as they determine whether the release should progress further and, if approved, they contribute to saving a substantial amount of time during the release window.
Verify all issues labeled with with this release have been resolved. Coordinate with the core/components team to close the gaps in resolving the issues labeled with with this release.
Get the Go / No-Go votes from project management committee (PMC) before staging the release artifacts for production publishing process,
| Repo | OpenSearch (Sample Runs) | OpenSearch Dashboards (Sample Runs) |
|---|---|---|
| YUM | yum-os | yum-osd |
| APT | apt-os | apt-osd |
| Artifacts | OpenSearch (Sample Runs) | OpenSearch Dashboards (Sample Runs) |
|---|---|---|
| Windows | os-windows-zip-x64 | osd-windows-zip-x64 |
| Debian | os-deb-arm64, os-deb-x64 | osd-deb-arm64, osd-deb-x64 |
| TAR | os-tar-arm64, os-tar-x64 | osd-tar-x64, osd-tar-arm64 |
| RPM | os-rpm-arm64, os-rpm-x64 | osd-rpm-arm64, osd-rpm-x64 |
Coordinate with the plugin teams and create a consolidates release notes. Sample PR. Release manager can check if a plugin team has created a release notes or not using the release notes tracker tool. Sample run.
Release the artifacts to production distribution channels, update the website and inform the community of the release.
Promote OpenSearch to maven, trigger the publish-to-maven worfklow (Ref Release Workflows), sample run.
Publish the images to docker and ECR, trigger the docker promotion workflow (Ref Release Workflows), sample run.
While not directly under the responsibility of a Release Manager, the following sections require coordination with the Project Management Team to ensure that the tasks are successfully fulfilled.
Coordinate with the documentation website team to ensure the changes are in place and close the gaps in promoting the website changes to prod. At this point the maintainers of the repos documentation-website and project-website will handle this task.
Coordinate with the project management team to ensure the social media advertisement is completed
Use the validation workflow (Ref Release Workflows) to validate the published artifcats, sample validation workflow run.
Once the release is completed following are the activities that needs to be completed by the release manager:
Create release tags for each OpenSearch and Dashboard components. Sample OpenSearch, OpenSearch Dashboards workflow runs.
Replace refs in input manifest with tags. The refs can be identified from the bundle or build manifest, sample PR that updates the input manifest.
Generate distribution release notes for opensearch-build repository, sample 1.3.10 release details.
Lower the frequency of builds for the release version of OpenSearch and/or OpenSearch Dashboards, sample PR.
Create an issue for a retrospective, solicit feedback, and publish a summary. Sample retro issue.
Update and release the Helm chart and ansible playbook with the new OpenSearch and Dashboard version. Sample helm chart PR and ansible PR.