[RELEASE] Release version 1.3.11

[github-actions]

Release OpenSearch and OpenSearch Dashboards 1.3.11

I noticed that a manifest was automatically created in manifests/1.3.11. Please follow the following checklist to make a release.

How to use this issue

This Release Issue

This issue captures the state of the OpenSearch release, its assignee is responsible for driving the release. Please contact them or @mention them on this issue for help. There are linked issues on components of the release where individual components can be tracked. More details are included in the Maintainers Release owner section.

Release Steps

There are several steps to the release process, these steps are completed as the whole release and components that are behind present risk to the release. The release owner completes the tasks in this ticket, whereas component owners resolve tasks on their ticket in their repositories.

Steps have completion dates for coordinating efforts between the components of a release; components can start as soon as they are ready far in advance of a future release.

Component List

To aid in understanding the state of the release there is a table with status indicating each component state. This is updated based on the status of the component issues.

Preparation

• Assign this issue to a release owner.
• Declare a pencils down date for new features to be merged.
• June 22nd, 2023 is pencils down date for feature freeze.
• Update this issue so all 1.3.11 placeholders have actual dates.
• Document any new quality requirements or changes.
• Finalize scope and feature set and update the Public Roadmap.
• Create a release issue in every component repo based on component release issue template and link back to this issue, update Components section with these links.
• Ensure the label is created in each component repo for this new version, and the next minor release. Create a version label
• Ensure that all release issues created above are assigned to an owner in the component team.
• Increase the build frequency for the this release from once a day (H 1 * * *) to once every hour (H/60 * * * *) in jenkinsFile.

Code Complete - Ends June 22nd, 2023

• Code Complete: Make sure that the code for this specific version of the release is ready and the branch corresponding to this release has been added to this release version manifest.
• Verify pull requests to add each component to manifests/1.3.11/opensearch-1.3.11.yml and manifests/1.3.11/opensearch-dashboards-1.3.11.yml have been merged.
• Gather, review and combine the release notes from components repositories.

Release testing - Ends June 27th, 2023

• Declare a release candidate build, and provide the instructions with the release candidates for teams on testing (June 23rd, 2023).
• Stop builds for this version of OpenSearch and/or OpenSearch Dashboards in order to avoid accidental commits going in unknowingly. Restart only if necessary else manually run the build workflow and declare new release candidate.
• After generate the release candidates, raise PR to lock input manifest refs of both OS and OSD builds with the commit ids from distribution manifest of the release candidates.
• Sanity Testing (June 23rd, 2023 - June 27th, 2023): Sanity testing and fixing of critical issues found by teams. Teams test their components within the distribution, ensuring integration, backwards compatibility, and perf tests pass.
• Publish all test results in the comments of this issue.

Performance testing validation - Ends June 27th, 2023

• Performance tests do not show a regression.
• Longevity tests do not show any issues.

Release - Ends June 29th, 2023

• Verify all issues labeled v1.3.11 in all projects have been resolved.
• Complete documentation for this release.
• Author blog post for this release.
• June 29th, 2023 - Publish this release on opensearch.org.
• June 29th, 2023 - Publish a blog post - release is launched!

Post Release

• Create release tags for each component (Jenkins job name: release-tag-creation).
• Replace refs in manifests/1.3.11 with tags and remove checks.
• If this is a major or minor version release, stop building previous patch version.
• Generate distribution release notes reviewed by PM team for opensearch-build repository.
• Increment version for Helm Charts (sample PR) for the 1.3.11 release.
• Increment version for Ansible Charts (sample PR) for the 1.3.11 release.
• Prepare for next patch release by incrementing patch versions for each component.
• Update this template with any new or missed steps.
• Create an issue for a retrospective, solicit feedback, and publish a summary.

Components

Replace with links to all component tracking issues.

Component	On track	Release Notes
{COMPONENT_ISSUE_LINK}	{INDICATOR}}	{STATUS}

Legend

Symbol	Meaning
🟢	On track with overall release
🟡	Missed last milestone
🔴	Missed multiple milestones

[jordarlu]

We have release candidates built and ready for testing.

---

OpenSearch Docker

Start without security
Docker command docker pull docker pull opensearchstaging/opensearch:1.3.11.8029 && docker run -it -d -p 9200:9200 -e "discovery.type=single-node" -e "DISABLE_SECURITY_PLUGIN=true" opensearchstaging/opensearch:1.3.11.8029
Connect command curl http://localhost:9200

Start with security
Docker command docker pull opensearchstaging/opensearch:1.3.11.8029 && docker run -it -d -p 9200:9200 -e "discovery.type=single-node" opensearchstaging/opensearch:1.3.11.8029
Connect command curl --insecure https://admin:admin@localhost:9200

OpenSearch Dashboards Docker

Start without security
Docker command docker pull opensearchstaging/opensearch-dashboards:1.3.11.6320 && docker run -it -d --network="host" -e "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" opensearchstaging/opensearch-dashboards:1.3.11.6320
URL http://localhost:5601

Start with security
Docker command docker pull opensearchstaging/opensearch-dashboards:1.3.11.6320 && docker run -it -d --network="host" opensearchstaging/opensearch-dashboards:1.3.11.6320
URL http://localhost:5601

Use below artifacts to deploy OpenSearch and OpenSearch Dashboards on different platforms

OpenSearch - Build 8029

• x64 artifacts: [manifest] [tar] [rpm] [yum] [windows]

• arm64 artifacts: [manifest] [tar] [rpm] [yum]

OpenSearch Dashboards - Build 6320

• x64 artifacts: [manifest] [tar] [rpm] [yum] [windows]

• arm64 artifacts: [manifest] [tar] [rpm] [yum]

How to install via yum, tarball, windows etc?]

[jordarlu]

Integ Test Results

OpenSearch 8029 integ-test result

Failure with rpm x64 🔴

2023-06-23 21:37:38 ERROR    | index-management     | with-security        | FAIL  |
2023-06-23 21:37:38 ERROR    | index-management     | without-security     | FAIL  |

Success with rpm arm64 ✅

Failure with tar x64 🔴

2023-06-23 21:40:40 ERROR    | index-management     | without-security     | FAIL  |

Failure with tar arm64 🔴

2023-06-23 21:46:51 ERROR    | index-management     | without-security     | FAIL  |

OpenSearch Dashboards 6320 integ-test result

Failure with rpm x64 🔴

2023-06-23 21:23:22 ERROR    | alertingDashboards   | without-security     | FAIL  |
2023-06-23 21:44:13 ERROR    | observabilityDashboards | with-security        | FAIL  |
2023-06-23 21:44:13 ERROR    | observabilityDashboards | without-security     | FAIL  |
2023-06-23 21:29:06 ERROR    | reportsDashboards    | with-security        | FAIL  |
2023-06-23 21:29:06 ERROR    | reportsDashboards    | without-security     | FAIL  |

Failure with rpm arm64 🔴

2023-06-23 21:27:32 ERROR    | alertingDashboards   | without-security     | FAIL  |
2023-06-23 21:56:23 ERROR    | observabilityDashboards | with-security        | FAIL  |
2023-06-23 21:32:29 ERROR    | reportsDashboards    | with-security        | FAIL  |
2023-06-23 21:32:29 ERROR    | reportsDashboards    | without-security     | FAIL  |

Failure with tar x64 🔴

2023-06-23 21:45:11 ERROR    | observabilityDashboards | without-security     | FAIL  |
2023-06-23 21:32:35 ERROR    | reportsDashboards    | with-security        | FAIL  |
2023-06-23 21:32:35 ERROR    | reportsDashboards    | without-security     | FAIL  |
2023-06-23 21:14:00 ERROR    | securityDashboards   | with-security        | FAIL  |

Failure with tar arm64 🔴

2023-06-23 21:28:32 ERROR    | alertingDashboards   | without-security     | FAIL  |
2023-06-23 21:54:57 ERROR    | observabilityDashboards | without-security     | FAIL  |
2023-06-23 21:34:14 ERROR    | reportsDashboards    | with-security        | FAIL  |
2023-06-23 21:34:14 ERROR    | reportsDashboards    | without-security     | FAIL  |

[stephen-crawford]

Do we have any updates on the status of this? It looks like all the failures are related or similar. For instance, the SecurityDashboards failure comes after successful cluster checks with an eventual output of

available","since":"2023-06-23T21:10:28.208Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:visTypeTable@1.3.11","message":"All dependencies are available","since":"2023-06-23T21:10:28.208Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:visTypeVega@1.3.11","message":"All dependencies are available","since":"2023-06-23T21:10:28.208Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:visTypeVislib@1.3.11","message":"All dependencies are available","since":"2023-06-23T21:10:28.208Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:alertingDashboards@1.3.11","message":"All dependencies are available","since":"2023-06-23T21:10:28.208Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:anomalyDetectionDashboards@1.3.11","message":"All dependencies are available","since":"2023-06-23T21:10:28.208Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:indexManagementDashboards@1.3.11","message":"All dependencies are available","since":"2023-06-23T21:10:28.208Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:reportsDashboards@1.3.11","message":"All dependencies are available","since":"2023-06-23T21:10:28.208Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:securityDashboards@1.3.11","message":"All dependencies are available","since":"2023-06-23T21:10:28.208Z","state":"green","icon":"success","uiColor":"secondary"}]},"metrics":{"last_updated":"2023-06-23T21:10:25.155Z","collection_interval_in_millis":5000,"os":{"platform":"linux","platformRelease":"linux-5.10.178-162.673.amzn2.x86_64","load":{"1m":6.86376953125,"5m":4.615234375,"15m":2.19677734375},"memory":{"total_in_bytes":32809476096,"free_in_bytes":4303745024,"used_in_bytes":28505731072},"uptime_in_millis":892000,"distro":"RHEL","distroRelease":"RHEL-undefined"},"process":{"memory":{"heap":{"total_in_bytes":106291200,"used_in_bytes":85187552,"size_limit":1526909922},"resident_set_size_in_bytes":145252352},"pid":255,"event_loop_delay":0.12347099999897182,"uptime_in_millis":32223.999999999996},"response_times":{"avg_in_millis":0,"max_in_millis":0},"concurrent_connections":0,"requests":{"disconnects":0,"total":0,"statusCodes":{},"status_codes":{}}}}

2023-06-23 21:10:28 INFO     Service is available

2023-06-23 21:10:28 INFO     Waiting for service to become available

2023-06-23 21:10:28 INFO     Pinging service attempt 0

2023-06-23 21:10:28 INFO     Pinging https://localhost:9200/_cluster/health

2023-06-23 21:10:28 INFO     200: {"cluster_name":"opensearch","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"discovered_master":true,"active_primary_shards":2,"active_shards":2,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

2023-06-23 21:10:28 INFO     Service is available

2023-06-23 21:10:28 INFO     ===============================================

2023-06-23 21:10:28 INFO     Running integration tests for securityDashboards

2023-06-23 21:10:28 INFO     ===============================================

2023-06-23 21:10:28 INFO     Executing "bash /tmp/tmpawx33_jb/securityDashboards/integtest.sh -b localhost -p 5601 -s true -t securityDashboards -v 1.3.11 -o default" in /tmp/tmpawx33_jb/securityDashboards

2023-06-23 21:12:48 INFO     Stderr reported for component: securityDashboards

2023-06-23 21:12:48 INFO     npm WARN old lockfile 

npm WARN old lockfile The package-lock.json file was created with an old version of npm,

npm WARN old lockfile so supplemental metadata must be fetched from the registry.

npm WARN old lockfile 

npm WARN old lockfile This is a one-time fix-up, please be patient...

npm WARN old lockfile 

npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.

Couldn't determine Mocha version

tput: No value for $TERM and no -T specified

Browserslist: caniuse-lite is outdated. Please run:

npx browserslist@latest --update-db


Why you should do it regularly:

https://github.com/browserslist/browserslist#browsers-data-updating

Couldn't determine Mocha version

tput: No value for $TERM and no -T specified

error Command failed with exit code 1.

In comparison, alertign dashboards for tar arm x64 comes out with

available","since":"2023-06-23T21:20:26.838Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:visTypeVislib@1.3.11","message":"All dependencies are available","since":"2023-06-23T21:20:26.838Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:alertingDashboards@1.3.11","message":"All dependencies are available","since":"2023-06-23T21:20:26.838Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:anomalyDetectionDashboards@1.3.11","message":"All dependencies are available","since":"2023-06-23T21:20:26.838Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:indexManagementDashboards@1.3.11","message":"All dependencies are available","since":"2023-06-23T21:20:26.838Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:reportsDashboards@1.3.11","message":"All dependencies are available","since":"2023-06-23T21:20:26.838Z","state":"green","icon":"success","uiColor":"secondary"}]},"metrics":{"last_updated":"2023-06-23T21:20:22.950Z","collection_interval_in_millis":5000,"os":{"platform":"linux","platformRelease":"linux-5.10.178-162.673.amzn2.aarch64","load":{"1m":6.6806640625,"5m":6.12353515625,"15m":3.68408203125},"memory":{"total_in_bytes":33023156224,"free_in_bytes":3338104832,"used_in_bytes":29685051392},"uptime_in_millis":1485000,"distro":"RHEL","distroRelease":"RHEL-undefined"},"process":{"memory":{"heap":{"total_in_bytes":80080896,"used_in_bytes":68770360,"size_limit":1526909922},"resident_set_size_in_bytes":112513024},"pid":3477,"event_loop_delay":0.24886799999512732,"uptime_in_millis":22639},"response_times":{"avg_in_millis":0,"max_in_millis":0},"concurrent_connections":0,"requests":{"disconnects":0,"total":0,"statusCodes":{},"status_codes":{}}}}

2023-06-23 21:20:26 INFO     Service is available

2023-06-23 21:20:26 INFO     Waiting for service to become available

2023-06-23 21:20:26 INFO     Pinging service attempt 0

2023-06-23 21:20:26 INFO     Pinging http://localhost:9200/_cluster/health

2023-06-23 21:20:26 INFO     200: {"cluster_name":"opensearch","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"discovered_master":true,"active_primary_shards":1,"active_shards":1,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

2023-06-23 21:20:26 INFO     Service is available

2023-06-23 21:20:26 INFO     ===============================================

2023-06-23 21:20:26 INFO     Running integration tests for alertingDashboards

2023-06-23 21:20:26 INFO     ===============================================

2023-06-23 21:20:26 INFO     Executing "bash /tmp/tmp142eium8/alertingDashboards/integtest.sh -b localhost -p 5601 -s false -t alertingDashboards -v 1.3.11 -o default" in /tmp/tmp142eium8/alertingDashboards

2023-06-23 21:28:30 INFO     Stderr reported for component: alertingDashboards

2023-06-23 21:28:30 INFO     tput: No value for $TERM and no -T specified

Browserslist: caniuse-lite is outdated. Please run:

  npx browserslist@latest --update-db

  Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating

tput: No value for $TERM and no -T specified

error Command failed with exit code 1.

So the same issue on the surface but nothing obviously specific to one plugin or another.

[jordarlu]

Docker-Scan CVEs: OpenSearch, OpenSearch DashBoards

[jordarlu]

Docker-Scan CVEs re-run after rebuilding to mitigate CVEs related with AL2 : OpenSearch, OpenSearch DashBoards

[jordarlu]

Perfomance Test on 'nyc_taxis' https://ci.opensearch.org/ci/dbc/perf-test/1.3.11/8029/linux/x64/tar/test-results/4914/perf-test/without-security/perf-test.html
Perfomance Test on 'http_logs' https://ci.opensearch.org/ci/dbc/perf-test/1.3.11/8029/linux/arm64/tar/test-results/4913/perf-test/with-security/perf-test.html

[peterzhuamazon]

Native plugin installation:

./opensearch-plugin install repository-s3
-> Installing repository-s3
-> Downloading repository-s3 from opensearch
[=================================================] 100%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.RuntimePermission getClassLoader
* java.lang.reflect.ReflectPermission suppressAccessChecks
* java.net.NetPermission setDefaultAuthenticator
* java.net.SocketPermission * connect,resolve
* java.util.PropertyPermission opensearch.allow_insecure_settings read,write
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed repository-s3 with folder name repository-s3