Merge branch 'main' into extension-handshake

CHANGELOG.md

@@ -27,7 +27,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Update Apache Lucene to 9.5.0-snapshot-d5cef1c ([#5570](https://github.com/opensearch-project/OpenSearch/pull/5570))
 - Bump `maven-model` from 3.6.2 to 3.8.6 ([#5599](https://github.com/opensearch-project/OpenSearch/pull/5599))
 - Bump `maxmind-db` from 2.1.0 to 3.0.0 ([#5601](https://github.com/opensearch-project/OpenSearch/pull/5601))
-- Bump `protobuf-java` from 3.21.11 to 3.21.12 ([#5603](https://github.com/opensearch-project/OpenSearch/pull/5603))
 - Bump `wiremock-jre8-standalone` from 2.33.2 to 2.35.0
 - Bump `gson` from 2.10 to 2.10.1
 - Bump `json-schema-validator` from 1.0.73 to 1.0.76
@@ -39,6 +38,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `com.maxmind.geoip2:geoip2` from 4.0.0 to 4.0.1
 - Bump `com.networknt:json-schema-validator` from 1.0.76 to 1.0.78
 - Bump `com.netflix.nebula:gradle-info-plugin` from 12.0.0 to 12.0.1
+- Bump `com.avast.gradle:gradle-docker-compose-plugin` from 0.16.11 to 0.16.12
+- Bump `org.apache.commons:commons-compress` from 1.22 to 1.23.0
+- Bump `org.apache.commons:commons-configuration2` from 2.8.0 to 2.9.0
+- Bump `com.netflix.nebula:nebula-publishing-plugin` from 19.2.0 to 20.2.0
+- Bump `net.minidev:json-smart` from 2.4.9 to 2.4.10
+- Bump `com.netflix.nebula.ospackage-base` from 11.0.0 to 11.2.0
 
 ### Changed
 - [CCR] Add getHistoryOperationsFromTranslog method to fetch the history snapshot from translogs ([#3948](https://github.com/opensearch-project/OpenSearch/pull/3948))
@@ -47,6 +52,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Change http code on create index API with bad input raising NotXContentException from 500 to 400 ([#4773](https://github.com/opensearch-project/OpenSearch/pull/4773))
 - Change http code for DecommissioningFailedException from 500 to 400 ([#5283](https://github.com/opensearch-project/OpenSearch/pull/5283))
 - Improve summary error message for invalid setting updates ([#4792](https://github.com/opensearch-project/OpenSearch/pull/4792))
+- Changed `opensearch-env` to respect already set `OPENSEARCH_HOME` environment variable ([#6956](https://github.com/opensearch-project/OpenSearch/pull/6956/))
 
 ### Deprecated
 
@@ -70,6 +76,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fix 'org.apache.hc.core5.http.ParseException: Invalid protocol version' under JDK 16+ ([#4827](https://github.com/opensearch-project/OpenSearch/pull/4827))
 - Fix compression support for h2c protocol ([#4944](https://github.com/opensearch-project/OpenSearch/pull/4944))
 - Support OpenSSL Provider with default Netty allocator ([#5460](https://github.com/opensearch-project/OpenSearch/pull/5460))
+- Avoid negative memory result in IndicesQueryCache stats calculation ([#6917](https://github.com/opensearch-project/OpenSearch/pull/6917))
 
 ### Security
 
@@ -85,10 +92,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Add wait_for_completion parameter to resize, open, and forcemerge APIs ([#6434](https://github.com/opensearch-project/OpenSearch/pull/6434))
 - [Segment Replication] Apply backpressure when replicas fall behind ([#6563](https://github.com/opensearch-project/OpenSearch/pull/6563))
 - [Remote Store] Integrate remote segment store in peer recovery flow ([#6664](https://github.com/opensearch-project/OpenSearch/pull/6664))
-- [Segment Replication] Add new cluster setting to set replication strategy by default for all indices in cluster. ([#6791](https://github.com/opensearch-project/OpenSearch/pull/6791))
 - Enable sort optimization for all NumericTypes ([#6464](https://github.com/opensearch-project/OpenSearch/pull/6464)
 - Remove 'cluster_manager' role attachment when using 'node.master' deprecated setting ([#6331](https://github.com/opensearch-project/OpenSearch/pull/6331))
 - Add new cluster settings to ignore weighted round-robin routing and fallback to default behaviour. ([#6834](https://github.com/opensearch-project/OpenSearch/pull/6834))
+- Add experimental support for ZSTD compression. ([#3577](https://github.com/opensearch-project/OpenSearch/pull/3577))
+- [Segment Replication] Add point in time and scroll query compatibility. ([#6644](https://github.com/opensearch-project/OpenSearch/pull/6644))
 
 ### Dependencies
 - Bump `org.apache.logging.log4j:log4j-core` from 2.18.0 to 2.20.0 ([#6490](https://github.com/opensearch-project/OpenSearch/pull/6490))
@@ -97,17 +105,21 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `io.projectreactor.netty:reactor-netty` from 1.1.3 to 1.1.4
 - Bump `com.avast.gradle:gradle-docker-compose-plugin` from 0.15.2 to 0.16.11
 - Bump `net.minidev:json-smart` from 2.4.8 to 2.4.9
-- Bump `com.google.protobuf:protobuf-java` from 3.22.0 to 3.22.2
+- Bump `com.google.protobuf:protobuf-java` to 3.22.2 ([#6994](https://github.com/opensearch-project/OpenSearch/pull/6994))
 - Bump Netty to 4.1.90.Final ([#6677](https://github.com/opensearch-project/OpenSearch/pull/6677)
 - Bump `com.diffplug.spotless` from 6.15.0 to 6.17.0
 - Bump `org.apache.zookeeper:zookeeper` from 3.8.0 to 3.8.1
 - Bump `net.minidev:json-smart` from 2.4.7 to 2.4.10
 - Bump `org.apache.maven:maven-model` from 3.6.2 to 3.9.1
 - Bump `org.codehaus.jettison:jettison` from 1.5.3 to 1.5.4 ([#6878](https://github.com/opensearch-project/OpenSearch/pull/6878))
+- Add `com.github.luben:zstd-jni:1.5.4-1` ([#3577](https://github.com/opensearch-project/OpenSearch/pull/3577))
+- Bump: Netty from 4.1.90.Final to 4.1.91.Final , ASM 9.4 to ASM 9.5, ByteBuddy 1.14.2 to 1.14.3 ([#6981](https://github.com/opensearch-project/OpenSearch/pull/6981))
+- Bump `com.azure:azure-storage-blob` from 12.15.0 to 12.21.1
 
 ### Changed
 - Require MediaType in Strings.toString API ([#6009](https://github.com/opensearch-project/OpenSearch/pull/6009))
 - [Refactor] XContent base classes from xcontent to core library ([#5902](https://github.com/opensearch-project/OpenSearch/pull/5902))
+- Added a new field type: flat_object ([#6507](https://github.com/opensearch-project/OpenSearch/pull/6507))
 
 ### Deprecated
 - Map, List, and Set in org.opensearch.common.collect ([#6609](https://github.com/opensearch-project/OpenSearch/pull/6609))
@@ -119,6 +131,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Added equals/hashcode for named DocValueFormat.DateTime inner class ([#6357](https://github.com/opensearch-project/OpenSearch/pull/6357))
 - Fixed bug for searchable snapshot to take 'base_path' of blob into account ([#6558](https://github.com/opensearch-project/OpenSearch/pull/6558))
 - Fix fuzziness validation ([#5805](https://github.com/opensearch-project/OpenSearch/pull/5805))
+- Fix GetSnapshots to not return non-existent snapshots with ignore_unavailable=true ([#6839](https://github.com/opensearch-project/OpenSearch/pull/6839))
 
 ### Security
 

benchmarks/src/main/java/org/opensearch/benchmark/store/remote/filecache/FileCacheBenchmark.java

@@ -27,6 +27,8 @@
 import org.openjdk.jmh.annotations.Threads;
 import org.openjdk.jmh.annotations.Warmup;
 import org.openjdk.jmh.infra.Blackhole;
+import org.opensearch.common.breaker.CircuitBreaker;
+import org.opensearch.common.breaker.NoopCircuitBreaker;
 import org.opensearch.index.store.remote.filecache.CachedIndexInput;
 import org.opensearch.index.store.remote.filecache.FileCache;
 import org.opensearch.index.store.remote.filecache.FileCacheFactory;
@@ -91,7 +93,8 @@ public static class CacheParameters {
         public void setup() {
             fileCache = FileCacheFactory.createConcurrentLRUFileCache(
                 (long) maximumNumberOfEntries * INDEX_INPUT.length(),
-                concurrencyLevel
+                concurrencyLevel,
+                new NoopCircuitBreaker(CircuitBreaker.REQUEST)
             );
             for (long i = 0; i < maximumNumberOfEntries; i++) {
                 final Path key = Paths.get(Long.toString(i));

buildSrc/build.gradle

@@ -106,7 +106,7 @@ dependencies {
   api 'org.apache.commons:commons-compress:1.22'
   api 'org.apache.ant:ant:1.10.13'
   api 'com.netflix.nebula:gradle-extra-configurations-plugin:9.0.0'
-  api 'com.netflix.nebula:nebula-publishing-plugin:19.2.0'
+  api 'com.netflix.nebula:nebula-publishing-plugin:20.2.0'
   api 'com.netflix.nebula:gradle-info-plugin:12.0.1'
   api 'org.apache.rat:apache-rat:0.15'
   api 'commons-io:commons-io:2.11.0'
@@ -115,7 +115,7 @@ dependencies {
   api 'org.jdom:jdom2:2.0.6.1'
   api "org.jetbrains.kotlin:kotlin-stdlib-jdk8:${props.getProperty('kotlin')}"
   api 'de.thetaphi:forbiddenapis:3.5.1'
-  api 'com.avast.gradle:gradle-docker-compose-plugin:0.16.11'
+  api 'com.avast.gradle:gradle-docker-compose-plugin:0.16.12'
   api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}"
   api 'org.apache.maven:maven-model:3.9.1'
   api 'com.networknt:json-schema-validator:1.0.78'

buildSrc/src/main/java/org/opensearch/gradle/test/DistroTestPlugin.java

@@ -119,8 +119,8 @@ public void apply(Project project) {
             TaskProvider<?> depsTask = project.getTasks().register(taskname + "#deps");
             depsTask.configure(t -> t.dependsOn(distribution, examplePlugin));
             depsTasks.put(taskname, depsTask);
-            // TODO - suppressing failure temporarily where duplicate tasks are created for docker.
-            try {
+            // Avoid duplicate tasks such as docker registered in lifecycleTasks
+            if (project.getTasksByName(taskname, false).isEmpty()) {
                 TaskProvider<Test> destructiveTask = configureTestTask(project, taskname, distribution, t -> {
                     t.onlyIf(t2 -> distribution.isDocker() == false || dockerSupport.get().getDockerAvailability().isAvailable);
                     addSysprop(t, DISTRIBUTION_SYSPROP, distribution::getFilepath);
@@ -134,8 +134,6 @@ public void apply(Project project) {
                 }
                 destructiveDistroTest.configure(t -> t.dependsOn(destructiveTask));
                 lifecycleTasks.get(distribution.getType()).configure(t -> t.dependsOn(destructiveTask));
-            } catch (Exception ex) {
-                System.out.println(ex.getMessage());
             }
 
             if ((distribution.getType() == OpenSearchDistribution.Type.DEB || distribution.getType() == OpenSearchDistribution.Type.RPM)

buildSrc/version.properties

@@ -1,5 +1,5 @@
 opensearch        = 3.0.0
-lucene            = 9.5.0
+lucene            = 9.6.0-snapshot-a3ae27f
 
 bundled_jdk_vendor = adoptium
 bundled_jdk = 19.0.2+7
@@ -16,7 +16,7 @@ supercsv          = 2.4.0
 # Update to 2.17.2+ is breaking OpenSearchJsonLayout (see https://issues.apache.org/jira/browse/LOG4J2-3562)
 log4j             = 2.17.1
 slf4j             = 1.7.36
-asm               = 9.4
+asm               = 9.5
 jettison          = 1.5.4
 woodstox          = 6.4.0
 kotlin            = 1.7.10
@@ -26,7 +26,7 @@ guava             = 31.1-jre
 # when updating the JNA version, also update the version in buildSrc/build.gradle
 jna               = 5.5.0
 
-netty             = 4.1.90.Final
+netty             = 4.1.91.Final
 joda              = 2.12.2
 
 # client dependencies
@@ -51,7 +51,7 @@ junit             = 4.13.2
 hamcrest          = 2.1
 mockito           = 5.2.0
 objenesis         = 3.2
-bytebuddy         = 1.14.2
+bytebuddy         = 1.14.3
 
 # benchmark dependencies
 jmh               = 1.35

client/rest-high-level/src/test/java/org/opensearch/client/OpenSearchRestHighLevelClientTestCase.java

@@ -55,7 +55,7 @@
 import org.opensearch.common.xcontent.XContentHelper;
 import org.opensearch.common.xcontent.XContentType;
 import org.opensearch.common.xcontent.json.JsonXContent;
-import org.opensearch.core.internal.io.IOUtils;
+import org.opensearch.common.util.io.IOUtils;
 import org.opensearch.ingest.Pipeline;
 import org.opensearch.search.SearchHit;
 import org.opensearch.search.SearchModule;

distribution/packages/build.gradle

@@ -63,7 +63,7 @@ import java.util.regex.Pattern
  */
 
 plugins {
-  id "com.netflix.nebula.ospackage-base" version "11.0.0"
+  id "com.netflix.nebula.ospackage-base" version "11.2.0"
 }
 
 void addProcessFilesTask(String type, boolean jdk) {

distribution/src/bin/opensearch-env

@@ -28,18 +28,20 @@ while [ -h "$SCRIPT" ] ; do
   fi
 done
 
-# determine OpenSearch home; to do this, we strip from the path until we find
-# bin, and then strip bin (there is an assumption here that there is no nested
-# directory under bin also named bin)
-OPENSEARCH_HOME=`dirname "$SCRIPT"`
-
-# now make OPENSEARCH_HOME absolute
-OPENSEARCH_HOME=`cd "$OPENSEARCH_HOME"; pwd`
-
-while [ "`basename "$OPENSEARCH_HOME"`" != "bin" ]; do
+if [[ -z "$OPENSEARCH_HOME" ]]; then
+  # determine OpenSearch home; to do this, we strip from the path until we find
+  # bin, and then strip bin (there is an assumption here that there is no nested
+  # directory under bin also named bin)
+  OPENSEARCH_HOME=`dirname "$SCRIPT"`
+
+  # now make OPENSEARCH_HOME absolute
+  OPENSEARCH_HOME=`cd "$OPENSEARCH_HOME"; pwd`
+
+  while [ "`basename "$OPENSEARCH_HOME"`" != "bin" ]; do
+    OPENSEARCH_HOME=`dirname "$OPENSEARCH_HOME"`
+  done
   OPENSEARCH_HOME=`dirname "$OPENSEARCH_HOME"`
-done
-OPENSEARCH_HOME=`dirname "$OPENSEARCH_HOME"`
+fi
 
 # now set the classpath
 OPENSEARCH_CLASSPATH="$OPENSEARCH_HOME/lib/*"

distribution/src/bin/opensearch-env.bat

@@ -3,15 +3,17 @@ set SCRIPT=%0
 rem determine OpenSearch home; to do this, we strip from the path until we
 rem find bin, and then strip bin (there is an assumption here that there is no
 rem nested directory under bin also named bin)
-for %%I in (%SCRIPT%) do set OPENSEARCH_HOME=%%~dpI
+if not defined OPENSEARCH_HOME (
+  for %%I in (%SCRIPT%) do set OPENSEARCH_HOME=%%~dpI
 
-:opensearch_home_loop
-for %%I in ("%OPENSEARCH_HOME:~1,-1%") do set DIRNAME=%%~nxI
-if not "%DIRNAME%" == "bin" (
+  :opensearch_home_loop
+  for %%I in ("%OPENSEARCH_HOME:~1,-1%") do set DIRNAME=%%~nxI
+  if not "%DIRNAME%" == "bin" (
+    for %%I in ("%OPENSEARCH_HOME%..") do set OPENSEARCH_HOME=%%~dpfI
+    goto opensearch_home_loop
+  )
   for %%I in ("%OPENSEARCH_HOME%..") do set OPENSEARCH_HOME=%%~dpfI
-  goto opensearch_home_loop
 )
-for %%I in ("%OPENSEARCH_HOME%..") do set OPENSEARCH_HOME=%%~dpfI
 
 rem now set the classpath
 set OPENSEARCH_CLASSPATH=!OPENSEARCH_HOME!\lib\*

distribution/tools/keystore-cli/src/test/java/org/opensearch/bootstrap/BootstrapTests.java

@@ -31,7 +31,7 @@
 
 package org.opensearch.bootstrap;
 
-import org.opensearch.core.internal.io.IOUtils;
+import org.opensearch.common.util.io.IOUtils;
 import org.opensearch.common.settings.KeyStoreCommandTestCase;
 import org.opensearch.common.settings.KeyStoreWrapper;
 import org.opensearch.common.settings.SecureSettings;

distribution/tools/keystore-cli/src/test/java/org/opensearch/common/settings/KeyStoreCommandTestCase.java

@@ -42,7 +42,7 @@
 
 import com.google.common.jimfs.Configuration;
 import com.google.common.jimfs.Jimfs;
-import org.opensearch.core.internal.io.IOUtils;
+import org.opensearch.common.util.io.IOUtils;
 import org.apache.lucene.tests.util.LuceneTestCase;
 import org.opensearch.cli.CommandTestCase;
 import org.opensearch.common.io.PathUtilsForTesting;

distribution/tools/keystore-cli/src/test/java/org/opensearch/common/settings/KeyStoreWrapperTests.java

@@ -39,7 +39,7 @@
 import org.apache.lucene.store.IndexOutput;
 import org.apache.lucene.store.NIOFSDirectory;
 import org.opensearch.common.Randomness;
-import org.opensearch.core.internal.io.IOUtils;
+import org.opensearch.common.util.io.IOUtils;
 import org.opensearch.env.Environment;
 import org.opensearch.test.OpenSearchTestCase;
 import org.hamcrest.Matchers;

distribution/tools/plugin-cli/src/main/java/org/opensearch/plugins/InstallPluginCommand.java

@@ -58,7 +58,7 @@
 import org.opensearch.common.SuppressForbidden;
 import org.opensearch.common.collect.Tuple;
 import org.opensearch.common.hash.MessageDigests;
-import org.opensearch.core.internal.io.IOUtils;
+import org.opensearch.common.util.io.IOUtils;
 import org.opensearch.env.Environment;
 
 import java.io.BufferedReader;

distribution/tools/plugin-cli/src/main/java/org/opensearch/plugins/PluginCli.java

@@ -32,7 +32,7 @@
 
 package org.opensearch.plugins;
 
-import org.opensearch.core.internal.io.IOUtils;
+import org.opensearch.common.util.io.IOUtils;
 import org.opensearch.cli.Command;
 import org.opensearch.cli.LoggingAwareMultiCommand;
 import org.opensearch.cli.Terminal;

distribution/tools/plugin-cli/src/main/java/org/opensearch/plugins/RemovePluginCommand.java

@@ -38,7 +38,7 @@
 import org.opensearch.cli.ExitCodes;
 import org.opensearch.cli.Terminal;
 import org.opensearch.cli.UserException;
-import org.opensearch.core.internal.io.IOUtils;
+import org.opensearch.common.util.io.IOUtils;
 import org.opensearch.env.Environment;
 
 import java.io.IOException;

distribution/tools/upgrade-cli/src/test/java/org/opensearch/upgrade/UpgradeCliTests.java

@@ -18,7 +18,7 @@
 import org.opensearch.common.io.PathUtilsForTesting;
 import org.opensearch.common.settings.KeyStoreWrapper;
 import org.opensearch.common.settings.Settings;
-import org.opensearch.core.internal.io.IOUtils;
+import org.opensearch.common.util.io.IOUtils;
 import org.opensearch.env.Environment;
 import org.opensearch.env.TestEnvironment;
 

libs/cli/build.gradle

@@ -34,7 +34,6 @@ apply plugin: 'opensearch.publish'
 dependencies {
   api 'net.sf.jopt-simple:jopt-simple:5.0.4'
   api project(':libs:opensearch-common')
-  api project(':libs:opensearch-core')
 }
 
 test.enabled = false

libs/cli/src/main/java/org/opensearch/cli/MultiCommand.java

@@ -36,7 +36,7 @@
 import joptsimple.OptionSet;
 import joptsimple.OptionSpec;
 import joptsimple.util.KeyValuePair;
-import org.opensearch.core.internal.io.IOUtils;
+import org.opensearch.common.util.io.IOUtils;
 
 import java.io.IOException;
 import java.util.ArrayList;

libs/core/src/main/java/org/opensearch/core/internal/io/IOUtils.java → libs/common/src/main/java/org/opensearch/common/util/io/IOUtils.java

@@ -30,7 +30,7 @@
  * GitHub history for details.
  */
 
-package org.opensearch.core.internal.io;
+package org.opensearch.common.util.io;
 
 import org.opensearch.common.Nullable;
 

libs/core/src/main/java/org/opensearch/core/internal/io/Streams.java → libs/common/src/main/java/org/opensearch/common/util/io/Streams.java

@@ -30,7 +30,7 @@
  * GitHub history for details.
  */
 
-package org.opensearch.core.internal.io;
+package org.opensearch.common.util.io;
 
 import java.io.IOException;
 import java.io.InputStream;

libs/core/src/main/java/org/opensearch/core/internal/io/package-info.java → libs/common/src/main/java/org/opensearch/common/util/io/package-info.java

@@ -6,5 +6,5 @@
  * compatible open source license.
  */
 
-/** Common internal I/O classes */
-package org.opensearch.core.internal.io;
+/** Common internal I/O utility classes */
+package org.opensearch.common.util.io;

libs/core/src/main/java/org/opensearch/core/internal/net/NetUtils.java → libs/common/src/main/java/org/opensearch/common/util/net/NetUtils.java

@@ -30,7 +30,7 @@
  * GitHub history for details.
  */
 
-package org.opensearch.core.internal.net;
+package org.opensearch.common.util.net;
 
 import java.io.IOException;
 import java.lang.reflect.Field;

libs/core/src/main/java/org/opensearch/core/internal/net/package-info.java → libs/common/src/main/java/org/opensearch/common/util/net/package-info.java

@@ -7,4 +7,4 @@
  */
 
 /** Common Network Utility Classes */
-package org.opensearch.core.internal.net;
+package org.opensearch.common.util.net;

libs/core/build.gradle

@@ -79,6 +79,9 @@ dependencies {
 
   api "com.fasterxml.jackson.core:jackson-core:${versions.jackson}"
 
+  // lucene
+  api "org.apache.lucene:lucene-core:${versions.lucene}"
+
   testImplementation "com.carrotsearch.randomizedtesting:randomizedtesting-runner:${versions.randomizedrunner}"
   testImplementation "junit:junit:${versions.junit}"
   testImplementation "org.hamcrest:hamcrest:${versions.hamcrest}"
@@ -96,4 +99,5 @@ tasks.named('forbiddenApisMain').configure {
 
 tasks.named("dependencyLicenses").configure {
   mapping from: /jackson-.*/, to: 'jackson'
+  mapping from: /lucene-.*/, to: 'lucene'
 }