Releasing the Distribution
This document explains the OpenSearch Process in detail.
OpenSearch Project: OpenSearch is a community-driven, Apache 2.0-licensed open source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data.
OpenSearch/OpenSearch Dashboards Bundle: An OpenSearch/OpenSearch Dashboards bundle refers to a compressed file that encompasses both the OpenSearch/OpenSearch Dashboards distribution and accompanying plugins. This compressed file acts as a container for the OpenSearch/OpenSearch Dashboards software and its associated extensions, enabling users to conveniently package and deploy the entire system.
OpenSearch/OpenSearch Dashboards Components/Plugins: OpenSearch/OpenSearch Dashboards components/plugins are extensions that can be added to OpenSearch/OpenSearch Dashboards, to add new features or functionality. There are a wide variety of plugins available refer OpenSearch available plugins and OpenSearch Dashboards available plugins sections.
These manifests serve as the initial input for driving the build workflow. These manifests are located in build repository and are generated by the manifest workflow. Once the manifest is created, the maintainers of the build repository review and merge it into the main branch. The entire build repository operates exclusively on the main branch.
These manifests are integral to the comprehensive testing of the core and components/plugins, as they contain numerous settings and configurations that facilitate various tests within the test_workflow.
| OpenSearch | OpenSearch Dashboards |
|---|---|
| opensearch-2.8.0-test.yml | opensearch-dashboards-2.8.0-test.yml |
Output of the build workflow, used as input to the assemble workflow.
| deb | rpm | tar | windows |
|---|---|---|---|
| os-arm64, os-x64 | os-arm64, os-x64 | os-x64, os-arm64 | os-x64 |
The final output of the assemble workflow and manifest that is added to the final distribution, this has the commitID (can be used for reproducible builds) and the artifact file links. This final bundle manifest incorporates the assembled components and ensures traceability through the commit ID and accessibility to the artifact files.
| deb | rpm | tar | windows |
|---|---|---|---|
| os-arm64, os-x64 | os-arm64, os-x64 | os-x64, os-arm64 | os-x64 |
These are the issues created by automation with the distribution build and integ-test workflows failure, the automation detects the component failure and raises an issue in the respective component repo. Sample integ-test failure AUTOCUT issue and distribution build failure AUTOCUT issue. The created AUTOCUT issues will have the updated information with latest build failure details, the automation also detects if the component build has passed and closes the issues automatically. For more details refer the closeBuildSuccessGithubIssue.groovy and createGithubIssue.groovy libraries part of the distribution build and integ-test workflows.
| Workflow | Description |
|---|---|
| Check for Build | Workflow that periodically triggers the distribution workflows using parameterized cron. |
| OpenSearch Distribution Build | Workflow that is responsible to build/assemble the OpenSearch and its components. |
| OpenSearch Dashboards Distribution Build | Workflow that is responsible to build/assemble the OpenSearch Dashboards and its components. |
| OpenSearch Integ Test | Workflow that runs integ tests for OpenSearch and its components. |
| OpenSearch Dashboards Integ Test | Workflow that runs integ tests for OpenSearch Dashboards and its components. |
| Benchmark Tests | Workflow that runs Performance tests using opensearch-benchmark on a cluster created with a given version. |
| BWC Tests | Workflow that runs backward compatibility tests on a cluster created with a given version. |
| RPM Validation | Workflow that validates the RPM distribution |
| Docker Build | Workflow that builds the OpenSearch and OpenSearch Dashboards docker images |
| Docker Copy | Workflow that copies the created docker images to multiple DockerHub and ECR repositories |
| Docker Scan | Workflow that checks vulnerabilities for a given docker image as an input. |
| 1.x Maven Publish | Workflow that published snapshot maven artifacts, used only for 1.3.x versions. For more details check this issue-319. |
| Workflow | Description |
|---|---|
| Release Notes Tracker | Workflow that identifies if a component has a release notes added based on the commit history. |
| Promote Repos | Workflow that signs and promotes the APT/YUM repos to the production buckets accessed via the cloudfront. |
| Promote artifacts | Workflow that signs and promotes all the release artifacts to the production buckets accessed via the cloudfront. |
| Publish to Maven | Workflow that signs and publishes to the central maven repository. |
| Docker Promotion | Workflow that promotes the docker images to production docker repositories. |
| Validation Workflow | Workflow that validates the released distribution. |
Each new OpenSearch release process starts when any one component increments a version, typically on the main branch. For example, OpenSearch#1192 incremented the version to 2.0. The version check automation workflow will notice this change or it can be triggered manually, and make a pull request (e.g. opensearch-build#514) that adds a new manifest (e.g. opensearch-2.9.0.yml). After that's merged, a GitHub issue is automatically opened by this workflow to make a new release using this release template (e.g. opensearch-build#566). Existing and new components (re)onboard into every release by submitting pull requests to each version's manifest.
The release manager to a specific OpenSearch release will be assigned through volunteer model. The request for a release manager will be posted in OpenSearch public Slack #releases channel and selected on first come first served (FCFS) model. Note: The release manager should be a maintainer of a repo under OpenSearch GitHub organization.
The release label creation is part of the version increment workflows running in the build repo OpenSearch Version Increment Workflow, OpenSearch Dashboards Version Increment Workflow. These workflows not only raise a pull request for a version increment but also verify if the release label exists for a given version. If it doesn't, they proceed to create it.
This issue captures the state of the OpenSearch release, its assignee is responsible for driving the release. Please contact them or @mention them on this issue for help. There are linked issues on components of the release where individual components can be tracked. More details are included in the Maintainers Release owner section.
The release issue is created by an automation workflow. Once the release manager is finalized, the release manager should be updating the created release issue. Sample Release Issue 2.8.0. Update the release issue so all __REPLACE_RELEASE-__ placeholders have actual dates.
Increase the build frequency for the release from once a day (H 1 * * *) to once every hour (H/60 * * * *) in check-for-build.jenkinsfile. This will ensure the Distribution Build workflow is called every hour to build and detect the components failure early that are part of the Input Manifest.
This step is necessary solely for the 1.3.x release process; you can find more information in the issue. Please make sure to update and execute the 1.x Maven Publish workflow; you can find the workflow specifics in the Build Workflows section. Additionally, you can refer to a relevant sample PR.
This section is not required for a patch release.
The component release issues are auto created by the workflows part of the build repo OpenSearch components, OpenSearch Dashboards components. These workflows create the release issues based on the template component_release_template.md and links back the global release issue part of the build. Sample component release issues created for 2.10.0. Sample link https://github.com/issues?q=is%3Aopen+is%3Aissue+user%3Aopensearch-project+%5BRELEASE%5D+Release+version+2.10.0+in%3Atitle+.
Inside the template component_release_template.md, replace the fields RELEASE_VERSION, RELEASE_BRANCH_X, RELEASE_BRANCH and RELEASE_ISSUE to desired release values before creating the release issues across the component/plugin repos. Once the fields are replaced, use the meta and gh cli to create the issues. Find the list of components/plugins from the opensearch-plugins repo (for OpenSearch, for OpenSearch Dashboards) and use the meta cli to create the release issues. For more details check the create-an-issue-in-all-plugin-repos section.
meta exec "gh issue create --label v2.8.0 --title 'Release version 2.8.0' --body-file /tmp/opensearch-build/.github/ISSUE_TEMPLATE/component_release_template.md"
If exists any release specific issues/campaigns, link it back to the release issue. Sample linked issues/campaigns
The Release Branch Readiness date is determined as the release date minus 4 days.
This release branch creation is not applicable for patch release.
This step requires both OpenSearch and OpenSearch Dashboards to create a release branch that will be used for the release.
This step requires that every team participating in a release has their release branch created for the corresponding release by the date listed on this step. The Distribution Build workflow will also start using the release branch to create release candidate instead of .x branches.
Versions are incremented as soon as development starts on a given version to avoid confusion. Following is the example that depicts the version increment.
- OpenSearch:
main= 3.0.0,2.x= 2.9.0, and2.8= 2.8.1 - common-utils:
main= 3.0.0.0,2.x= 2.9.0.0 and2.8= 2.8.1.0
To ensure the version incrementation process is handled correctly, it is important to follow to increment the version of the release branch. Currently, the version incrementation is being done manually by an individual from the core repositories. Sample OpenSearch Version Increment PR for the release branch.
The next step is to identify the pending component version increment pull requests. The creation of component version increment pull requests is automated with workflows running in the build repo OpenSearch Version Increment Workflow, OpenSearch Dashboards Version Increment Workflow. Sample component version increment PR.
The objective is to merge these pull requests in order to synchronize all the components with the core version. It is not as simple as submitting a pull request and having it merged. When a pull request is submitted, CI checks are triggered for each repository. These CI checks assess the dependencies. Thus, prior to merging the pull requests that involve version increments, the release manager must confirm that the dependencies have been updated, built, and pushed to Maven and S3. This crucial stage requires the release manager to coordinate with the component teams, address any dependency issues, and ultimately merge the pull requests that involve version increments.
Coordinate with the Core and component teams to ensure that the code for this particular release version is fully prepared and that the corresponding branch has been included in the release version Input Manifest. Update Jenkins workflows that execute daily snapshot builds for both OpenSearch and OpenSearch Dashboards (Ref Increase the build frequency). Submit pull requests to incorporate each component into the respective version level Input Manifest along with the necessary checks. Sample PR. The Feature Freeze date and the Code Complete date is determined as the release date minus 4 days.
Refer the Build Workflows section to get the details about the distribution build job for OpenSearch and OpenSearch Dashboards. This section covers how to sequentially handle the distribution build job during the release.
Ensure the proper inputs are used to initiate the distribution. For instance, here's an example for OpenSearch. It's important that the OpenSearch Dashboard distribution aligns accordingly.
COMPONENT_NAME: To trigger a specific component, this includes standalone OpenSearch or specific plugin.
INPUT_MANIFEST: The release input manifest that drives the workflow.
TEST_MANIFEST: The release test input manifest that is used for the integ tests.
INTEG_TEST_JOB_NAME: The integ test job name. Default already added to this input integ-test for OpenSearch and integ-test-opensearch-dashboards for OpenSearch Dashboards.
BUILD_PLATFORM: The input used to build for a specific platform, followed by its own distributions within the platform.
BUILD_DISTRIBUTION: Input to build selected distribution related artifacts, choices include 'tar', 'rpm', 'deb', 'zip'. Can combine multiple distributions with space in between (docker is only available on tar).
BUILD_DOCKER: Input with a dropdown that has 3 options build_docker, build_docker_with_build_number_tag, do_not_build_docker, the release manager has to take a call with right inputs.
UPDATE_LATEST_URL: To update the /latest CFN URL, Visit latest-distribution-url for more details.
Following is the order of execution of the distribution build to address the components dependencies.
Note: The execution order specified is necessary only for versions up to 1.3.x. For 2.x and above Maven dependencies are published through each component repository using the GH workflow. For more details check the META issue.
OpenSearch
OpenSearch + common-utils + job-scheduler
OpenSearch + common-utils + job-scheduler + performance-analyzer
OpenSearch + common-utils + job-scheduler + performance-analyzer + security
OpenSearch + common-utils + job-scheduler + ml-commons + performance-analyzer + security
All components (which are ready after completion of version increment)
OpenSearch Dashboards
All components (which are ready after completion of version increment)
Now once all the version increment PRs are completed and all the components are part of the input manifest, now it's time to generate the RC. Use the Distribution Build to generate the release candidate. Use the following section as the reference to generate the RC, validate it and broadcast it for a given release. The process of Release Candidate Generation and Testing should commence at least 6 days prior to the release date.
Following is the generated build number after triggering the Distribution Build workflow. The distribution build number denotes the RC, now with the example below the finalized RC’s are OS: 7848, OSD: 6126.
| OpenSearch | OpenSearch Dashboards |
|---|---|
| build_7848 | build_6126 |
Following are the details for the docker image build and scan. The docker images are built using the TAR artifact generated as part of the Distribution Build (From the above example OS: 7848, OSD: 6126). The Distribution Build workflow with input BUILD_DOCKER (Ref Workflow Trigger) triggers the [docker-build] workflow as downstream.
| Docker | build | scan |
|---|---|---|
| OpenSearch | Build | Scan |
| OpenSearch Dashboards | Build | Scan |
This to ensure that check-for-build.jenkinsfile won't re-build periodically and override the docker, the RC docker is created with build number. This step can be skipped if the input BUILD_DOCKER: build_docker_with_build_number_tag (Ref Workflow Trigger used in the Distribution Build).
| Docker Freeze | copy |
|---|---|
| OpenSearch | docker-copy |
| OpenSearch Dashboards | docker-copy |
For running the benchmark tests, use the benchmark-test job part of the Build Workflows. For more details in running the benchmark tests refer Benchmarking Tests section part of the test workflow. This job offers multiple options to test the performance of a specific version cluster using various metrics. The benchmark performance results can be accessed via the OpenSearch Performance Benchmarks dashboard. Sample benchmark tests for 2.9.0 release.
For more details in running the BWC tests refer Backwards Compatibility Tests section.
On board the components/plugins to the test Test Manifest with the bwc-test setting. Example as follows:
- name: index-management
bwc-test:
test-configs:
- with-security
Currently, the windows integration tests for a release is manual. The manually tested windows zip is being evaluated and approved by the plugin teams for sign off. In order to test the windows distribution, two instances need to be created: one with security features enabled and another without security. Afterward, API calls should be tested by launching the OpenSearch and Dashboard processes through direct execution of the .bat file.
Broadcast the release candidate in OpenSearch public slack workspace and the release GitHub issue using format sample broadcast message to gather votes.
As a release manager, it is essential to ensure the successful completion of all the above mentioned jobs. In the event of failures during integration tests or scans, the release manager should collaborate with the component teams and initiate a re-run to ensure that all jobs are executed successfully.
Post all the job related failures in the Release issue, Sample post.
Note: Sometimes the integ-test jobs are flaky and might not pass due to several reasons with the component code, in that case coordinate with the respective component team and get a manual sign off. Sample manual sign off. All the failed logs are in s3 accessed through the cloudfront. Sample link.
Stop builds for this version of OpenSearch and/or OpenSearch Dashboards in order to avoid accidental commits going in unknowingly. Restart only if necessary, else manually run the build workflow and declare new release candidate.
Once the RC is finalized, in order to exclude the release from running periodically, at this point it is necessary for the release manager to lock the input manifest and update the check-for-build.jenkins to remove it from the scheduled execution, sample PR.
Ensure that all pre-release activities listed below are completed before proceeding with the final release. These activities hold great significance as they determine whether the release should progress further and, if approved, they contribute to saving a substantial amount of time during the release window.
Verify all issues labeled with this release have been resolved. Coordinate with the core/components team to close the gaps in resolving the issues labeled with this release.
Get the Go / No-Go votes from project management committee (PMC) before staging the release artifacts for production publishing process,
| Repo | OpenSearch (Sample Runs) | OpenSearch Dashboards (Sample Runs) |
|---|---|---|
| YUM | yum-os | yum-osd |
| APT | apt-os | apt-osd |
| Artifacts | OpenSearch (Sample Runs) | OpenSearch Dashboards (Sample Runs) |
|---|---|---|
| Windows | os-windows-zip-x64 | osd-windows-zip-x64 |
| Debian | os-deb-arm64, os-deb-x64 | osd-deb-arm64, osd-deb-x64 |
| TAR | os-tar-arm64, os-tar-x64 | osd-tar-arm64, osd-tar-x64 |
| RPM | os-rpm-arm64, os-rpm-x64 | osd-rpm-arm64, osd-rpm-x64 |
Coordinate with the plugin teams and create a consolidates release notes. Sample PR. Release manager can check if a plugin team has created a release notes or not using the release notes tracker tool. Sample run.
Release the artifacts to production distribution channels, update the website and inform the community of the release.
Promote OpenSearch to maven, trigger the publish-to-maven workflow (Ref Release Workflows), sample run.
Publish the images to docker and ECR, trigger the docker promotion workflow (Ref Release Workflows), sample run.
While not directly under the responsibility of a Release Manager, the following sections require coordination with the Project Management Team to ensure that the tasks are successfully fulfilled.
Coordinate with the documentation website team to ensure the changes are in place and close the gaps in promoting the website changes to prod. At this point the maintainers of the repos documentation-website and project-website will handle this task.
Coordinate with the project management team to ensure the social media advertisement is completed.
Use the validation workflow (Ref Release Workflows) to validate the published artifacts, sample validation workflow run.
Please update the checklist either in the release issue body or as a new comment to the release issue. By following and updating the release checklist, we can ensure the success of the release. Sample release checklist for 2.9.0 release.
Once the release is completed following are the activities that needs to be completed by the release manager:
Create release tags for each OpenSearch and Dashboard components. Sample OpenSearch, OpenSearch Dashboards workflow runs.
Replace refs in input manifest with tags. The refs can be identified from the bundle or build manifest, sample PR that updates the input manifest.
Generate distribution release notes for opensearch-build repository, sample 1.3.10 release details.
Lower the frequency of builds for the release version of OpenSearch and/or OpenSearch Dashboards, sample PR.
Create an issue for a retrospective, solicit feedback, and publish a summary. Sample retro issue.
Update and release the Helm chart and ansible playbook with the new OpenSearch and Dashboard version. Sample helm chart PR and ansible PR.
The release manager for the current release should ensure that a release manager is assigned for the upcoming release. This can be achieved by coordinating in a Slack channel and @opensearch-project/engineering-effectiveness team should assist in assigning the user to the upcoming Release issue.
Please utilize the provided communication templates to effectively coordinate with the teams involved in the release process. These templates are designed to assist you in communicating through the GitHub release issue and the public Slack #releases channel.
We are starting the process of <RELEASE_VERSION> release.
Release Issue: <RELEASE_ISSUE>
Release Manager: <RELEASE_MANAGER>
Kindly review the following information provided below regarding the release dates:
Release Branch and Version Increment: <REPLACE_RELEASE-minus-14-days>
Feature freeze: <REPLACE_RELEASE-minus-12-days>
Code Complete: <REPLACE_RELEASE-minus-10-days>
RC creation and : <REPLACE_RELEASE-minus-6-days>
Pre Release: <REPLACE_RELEASE-minus-1-days>
Release date: <RELEASE_DATE>
Post Release: <RELEASE_DATE>
The version increment pull requests are still awaiting attention. We kindly request the plugin/component owners to review them and make an effort to merge them promptly.
OpenSearch: https://github.com/pulls?q=is%3Aopen+is%3Apr+org%3Aopensearch-project+increment+<RELEASE_VERSION>+in%3Atitle+
OpenSearch Dashboards: https://github.com/pulls?q=is%3Aopen+is%3Apr+org%3Aopensearch-project+increment+<RELEASE_VERSION>.0+in%3Atitle+
Refer the sample rc announcement from past 1.3.10 release.
OpenSearch 2.8.0 version has been released to public :tada:!
Thanks everyone for the help to bring <RELEASE_VERSION> release out.
Component repo owners please create a github release based on the tags of <RELEASE_VERSION.0>.
Presenting the retrospective <RELEASE_RETRO_ISSUE_LINK> for the release. Please feel free to provide your valuable feedback for further improvements in the upcoming release.