[RELEASE] Release version 2.11.1

[github-actions]

Release OpenSearch and OpenSearch Dashboards 2.11.1

I noticed that a manifest was automatically created in manifests/2.11.1. Please follow the following checklist to make a release.

How to use this issue

This Release Issue

This issue captures the state of the OpenSearch release, its assignee (Release Manager) is responsible for driving the release. Please contact them or @mention them on this issue for help. There are linked issues on components of the release where individual components can be tracked. For more information check the the Release Process OpenSearch Guide.

Please refer to the following link for the release version dates: Release Schedule and Maintenance Policy.

Preparation

• Release manager assigned.
• Existence of label in each component repo. For more information check the release-label section.
• Increase the build frequency.
• Release Issue.

Campaigns

• Component Release Issue.
• Release Campaigns.

Release Branch and Version Increment - Ends Nov 14 2023

• Core Release Branch.
• Core Version Increment.
• Components Release Branch.
• Components Version Increment.

Feature Freeze

• OpenSearch / OpenSearch-Dashboards core and components teams finalize their features.

Code Complete - Ends date

• Mark this as done once the Code Complete is reviewed.
• Create/Verify pull requests to add each component to relase input manifests/2.11.1/opensearch-2.11.1.yml and manifests/2.11.1/opensearch-dashboards-2.11.1.yml.

Release Candidate Creation and Testing - Ends Nov 21 2023

• Generate Release Candidate.
• Integ Test TAR.
• Integ Test RPM.
• Docker Build and Scan.
• Backwards Compatibility Tests.
• Windows Integration Test.
• Broadcast and Communication.
• Release Candidate Lock.

Pre Release - Ends Nov 24 2023

• Release Labeled Issues.
• Go or No-Go.
• Promote Repos.
• Promote artifacts.
• Release Notes.

Release - Ends Nov 27 2023

• Maven Promotion.
• Docker Promotion.
• Release Validation.
• Collaboration with the Project Management Team.

Release Checklist.

Release Checklist

Pre-Release activities

• Promote Repos.
  • • OS
  • • OSD
• Promote Artifacts.
  • • Windows
  • • Linux Debian
  • • Linux RPM
  • • Linux TAR
• Consolidated Release Notes.

Release activities

• Docker Promotion.
• Release Validation part 1.
  • • OpenSearch and OpenSearch Dashboard Validation.
  • • Validate the native plugin installation.
• Merge consolidated release notes PR.
• Website and Documentation Changes.
  • • Merge staging website PR.
  • • Promote the website changes to prod.
  • • Add website alert.
• Release Validation part 2.
  • • Validate the artifact download URL's and signatures.
• Release Validation part 3.
  • • Trigger the validation build (Search for Completed validation for <> in the logs).
• Maven Promotion.
• Publish blog posts.
• Advertise on Social Media.
• Post on public slack and Github Release issue.

Post-Release activities

• Release Tags.
• Input Manifest Update.
• Decrease the Build Frequency.
• OpenSearch Build Release notes.
• Retrospective Issue.
• Helm and Ansible Playbook release.
• Upcoming Release Preparation.

Post Release

• Release Tags.
• Input Manifest Update.
• OpenSearch Build Release notes.
• Decrease the Build Frequency.
• Retrospective Issue.
• Helm and Ansible Playbook release.
• Upcoming Release Preparation.

Components

Replace with links to all component tracking issues.

Component	On track	Release Notes
{COMPONENT_ISSUE_LINK}	{INDICATOR}}	{STATUS}

Legend

Symbol	Meaning
🟢	On track with overall release
🟡	Missed last milestone
🔴	Missed multiple milestones

[bbarani]

Bug fixes targeted for 2.11.1 release:

OpenSearch:

• [Regression] Request decompression was broken in 2.11 OpenSearch#10802

[peternied]

For reference the fix for that issue was made inside the security plugin, it is in the 2.11 branch:

• [Backport 2.11] Use new instance of Decompressor on channel initialization security#3599

Having the ref for the security plugin point to 27dee2 to be sure the fix is included in this patch release.

Thank you @Divyaasm

[bbarani]

List of repos that have changes merged in to 2.11 branch post 2.11 release.

Repo	Branch	CommitID	Commit Date	PRs that went in after 2.11.0
OpenSearch	[2.11]	ac4de44	2023-10-30	opensearch-project/OpenSearch#10677, opensearch-project/OpenSearch#10719, opensearch-project/OpenSearch#11005
common-utils	[2.11]	dcf288d	2023-10-24	opensearch-project/common-utils#551, opensearch-project/common-utils#555
job-scheduler	[2.11]
k-NN	[2.11]
geospatial	[2.11]	07fd490	2023-10-27	opensearch-project/geospatial#576
security	[2.11]	f27dee2	2023-10-25	opensearch-project/security#3599, opensearch-project/security#3486, opensearch-project/security#3480
cross-cluster-replication	[2.11]	cf633fa	2023-10-18	opensearch-project/cross-cluster-replication#1276
ml-commons	[2.11]	1a31c5f	2023-10-30	opensearch-project/ml-commons#1524
neural-search	[2.11]
notifications-core	[2.11]
notifications	[2.11]
opensearch-observability	[2.11]	3b6d40e	2023-10-16
opensearch-reports	[2.11]	ab8d86c	2023-10-16	opensearch-project/reporting#915
sql	[2.11]	6e17ae6	2023-10-30	22 PRs (including multiple features)
asynchronous-search	[2.11]
anomaly-detection	[2.11]
alerting	[2.11]	c00c86d	2023-10-24	opensearch-project/alerting#1286
security-analytics	[2.11]	5b4ab6c	2023-10-31	7 commits
index-management	[2.11]	d1f9bce	2023-10-13	opensearch-project/index-management#1013, opensearch-project/index-management#1012
performance-analyzer	[2.11]
custom-codecs	[2.11]

Repo	Branch	CommitID	Commit Date	PRs that went in after 2.11.0
OpenSearch-Dashboards	[2.11]	227a412	2023-10-27	opensearch-project/OpenSearch-Dashboards#5288, opensearch-project/OpenSearch-Dashboards#5301, opensearch-project/OpenSearch-Dashboards#5330, opensearch-project/OpenSearch-Dashboards#5390
functionalTestDashboards	[2.11]	668a8b5	2023-10-30	opensearch-project/opensearch-dashboards-functional-test#929, opensearch-project/opensearch-dashboards-functional-test#933, opensearch-project/opensearch-dashboards-functional-test#938, opensearch-project/opensearch-dashboards-functional-test#945
observabilityDashboards	[2.11]	e0a509c	2023-10-30	26 commits
reportsDashboards	[2.11]	82d154a	2023-10-19	opensearch-project/dashboards-reporting#229, opensearch-project/dashboards-reporting#223
ganttChartDashboards	[2.11]
queryWorkbenchDashboards	[2.11]	94805c5	2023-10-30	12 commits
customImportMapDashboards	[2.11]
anomalyDetectionDashboards	[2.11]
mlCommonsDashboards	[2.11]
indexManagementDashboards	[2.11]
notificationsDashboards	[2.11]
alertingDashboards	[2.11]	8a55e43	2023-10-25	opensearch-project/alerting-dashboards-plugin#767,
securityAnalyticsDashboards	[2.11]	de6001a	2023-10-25	opensearch-project/security-analytics-dashboards-plugin#770, opensearch-project/security-analytics-dashboards-plugin#762, opensearch-project/security-analytics-dashboards-plugin#766, opensearch-project/security-analytics-dashboards-plugin#752
securityDashboards	[2.11]	ffc793f	2023-10-23	opensearch-project/security-dashboards-plugin#1626
searchRelevanceDashboards	[2.11]

[CEHENKLE]

So we need to make a decision about 2.11.1.

A little background first:

1/ We need this release because of a critical breaking bug in the security plug-in. This bug impacts everyone using logstash and the only work around is to turn off compression (which is expensive)

2/ As a refresher, our normal process is to commit changes to main, then backport to 2.x.. Since the 1.0 release, we only backport bug fixes and security changes to the current 2.x branch (in this case 2.11) so that if we need to we can quickly do a patch release we can.

3/ There are two reason we don’t back port features to the current release branch (right now 2.11):
a) Because we want patch release to be as thin as possible, and the more changes you make, the more verification we need to do
b) Adding features in a patch breaks semver.

However, looking at the list of changes for 2.11, we currently have a mix of features that have been merged in.

Here are the ways forward that we’ve thought of (feel free to suggest another way):

0. Ship nothing, wait until Jan 2.12

• Pros:
  -- Least disruptive
  -- Follows SemVer

• Cons:
  -- We are broken with logstash, and the only work around in place is not scalable (turn compression off) and costs more money
  -- The high severity fixes will not be released quickly

1. Ship 2.11.1 as-is with features using 2.11 branch

• Pros:
  -- This is a quick fix with limited change to build infrastructure and workflows
  -- We will be able to follow the normal patch release process
  -- The high severity fixes will be released quickly
• Cons:
  -- This is against SemVar guidelines as we are introducing features in patch release
  -- Features might get released without proper testing

2. Create 2.11.1 branch and use commits from 2.11 release ( and cherry pick only necessary bug fixes in to 2.11.1 branch)

• Pros:
  -- This is a quick fix with limited blast radius.
  -- We will be able to adhere to SemVar guidelines
  -- Patch release will be lean with minimal changes (as expected out of patch release)
  -- Limited participation required from all the teams
  -- The high severity fixes will be released quickly
• Cons:
  -- This requires manual creation of branches followed by cherry picking the code commits
  -- This might require additional changes in build scripts to use 2.11.1 patch branch (3 digit branches were never used before)
  -- We will deviate from our normal patch release process, which adds risks
  -- We would need to tag 2.11.1 in 2.11.1 branch instead of 2.11 branch (and this might cause confusion)

3. Release 2.12 version instead of 2.11.1 version to accommodate new features

• Pros:
  -- We will be able to adhere to SemVar guidelines
  -- This will be a full blown release with comprehensive testing so we will have high confidence in the features
  -- We will be able to release additional features in this release
  -- Community trust and transparency will not be impacted
• Cons:
  -- This requires a full blown release so would take more time compared to a patch release
  -- The high severity fixes has to wait until we release minor version
  -- This might delay the release planning to accommodate any feature that is not ready at this point in time

4. Revert all feature related changes from 2.11 branch

• Pros:
  -- We will be able to adhere to SemVar guidelines
  -- We will not deviate from our normal patch release process
  -- Patch release will be lean with minimal changes (as expected out of patch release)
  -- Limited participation required from all plug-ins
  -- The high severity fixes will be released quickly
• Cons:
  -- This requires manual revert of features from 2.11 release branch ( so there is a decent effort required from all plugin / core owners)
  -- Downstream teams that picked up 2.11 and built from it will be left in a weird state
  -- The reverts has be coordinated across multiple teams to avoid introducing additional issues.

As you can see, there’s no solution without cons, but given all the trade-offs, I’m leaning towards 1. WDYT?

Thanks,

PS, because this is impacting people today, I’d like to get this release out ASAP. So please put your comments up in the next 48 hours so we can prep a release for early next week (if that's what we're going to do). I also want to better understand why features were backported to the release branch, and what we should do about this in the future, but I’d like to tackle that after 2.11.1 is out.

[reta]

Thanks @CEHENKLE

1. Ship 2.11.1 as-is with features using 2.11 branch

+1 to this option, for OpenSearch core at least, out of 3 merged pull requests, 2 are non-functional changes and 1 is a bugfix (with accordance to semver and signed off by the team before the merge). For a plugins though (except security one) would be good to hear maintainers.

[smortex]

Regarding back-ported features in release branches

However, looking at the list of changes for 2.11, we currently have a mix of features that have been merged in.

😶 That's unexpected. It feels like a lot of these backports PR where opened with some automation due to improper labels being put on the initial PR (i.e. a feature should only have been labeled backport-2.x and a bugfix should have been labeled backport-2.x and backport-2.10). Maybe ensuring each PR is labelled as a backwards-incompatible, enhancement or bugfix can be used with this automation to detect inconsistencies and avoid this situation in the future? Is there a PR to discuss this?

Regarding what to do with the next release

#	Scenario	My feeling
0	Ship nothing, wait until Jan 2.12	LGTM but does not looks acceptable for some users 🙃
1	Ship 2.11.1 as-is with features using 2.11 branch	Strong no go: if we do this on purpose we should really stop pretending we are doing something that connects even marginally with SemVer. When this happen by accident, SemVer has a FAQ entry to describe how it should be handled (basically scenario 5)
2	Create 2.11.1 branch and use commits from 2.11 release ( and cherry pick only necessary bug fixes in to 2.11.1 branch)	That is basically having a 2.11.1 branch that contains what 2.11 was supposed to have. I wonder if keeping the 2.11 branch at it is makes sense in this case, because a 2.11.2 version would be branched out of the 2.11.1 branch is another patch release is to be issued
3	Release 2.12 version instead of 2.11.1 version to accommodate new features	LGTM
4	Revert all feature related changes from 2.11 branch	LGTM

Given the situation, 0 & 3 seems the most practical (bump minor version and release), 4 is probably the cleanest thing we can do but require more efforts (allows us to bump the patch version and release).

Branching 2.12 from 2.11 instead of 2.x and releasing the 2.12.0 version from there seems to require the minimum efforts and would have my preference because it looks to require reasonable effort (scenario 3).

If the cost regarding community trust of releasing 2.12 earlier than announced and without the announced features of 2.12 seems problematic, scenario 4 is probably what must be done IMHO.

If both are too much pain, as a last resort scenario 0 is acceptable.

[peternied]

Chiming in for Security, I would advocate for option 1, as its the smallest deviation from our existing process and I trust that maintainers are making the right choices.

For me the most important trait of following Semver is signaling breaking changes on major version changes, secondarily is signaling potential risk between minor/patch versions. The sematic argument of 'is it a feature or a bug' is hard to judge on a product of this scale.

Changing the process mid-flight is risky - I'd advocate for doing a retro afterwards to drill into misaligned expectations around branches / release.

[macohen]

I think that this should be the 2.12 release. We know there are features and bugs in this release. We need to be careful with testing and make sure we're confident in the release. It's a slippery slope to make exceptions on semver, but continue to say that we follow it. We should also have a retro meeting after the fact that involves representatives from interested parties who take these releases into their products/services.

[joshuarrrr]

In terms of level of effort, options 2 and 4 seem comparable/equivalent, but 4 gets us to a proper/desired state where we'd prefer to be, whereas option 2 deviates our process quite a bit.

• Option 0 - Prolongs user pain unnecessarily
• Option 1 - I think this is wrong and damages trust
• Option 2 - I think this is fine, but worse than option 4, which is similar level of effort
• Option 3 - I also think this is preferable to options 0 and 1, but will mean a slower fix for the critical bug
• Option 4 - My preferred option to solve pain while restoring us to a correct state

I trust that maintainers are making the right choices.

But we have evidence that's not the case - maintainers are responsible for adding the labels which triggers the backport automation and have (intentionally or not) violated semver by improperly backporting features to the 2.11 branch.