Merge branch 'main' into updateGoogleJimfs

Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com>

CHANGELOG.md

@@ -40,8 +40,6 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `org.bouncycastle:bcprov-jdk15on` to `org.bouncycastle:bcprov-jdk15to18` version 1.75 ([#8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
 - Bump `org.bouncycastle:bcmail-jdk15on` to `org.bouncycastle:bcmail-jdk15to18` version 1.75 ([#8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
 - Bump `org.bouncycastle:bcpkix-jdk15on` to `org.bouncycastle:bcpkix-jdk15to18` version 1.75 ([#8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
-- Bump `com.google.jimfs:jimfs` from 1.2 to 1.3.0 ([#8585](https://github.com/opensearch-project/OpenSearch/pull/8585))
-- Bump `com.google.cloud:google-cloud-core-http` from 2.17.0 to 2.21.0 ([#8586](https://github.com/opensearch-project/OpenSearch/pull/8586))
 
 ### Changed
 - [CCR] Add getHistoryOperationsFromTranslog method to fetch the history snapshot from translogs ([#3948](https://github.com/opensearch-project/OpenSearch/pull/3948))
@@ -53,6 +51,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Pass localNode info to all plugins on node start ([#7919](https://github.com/opensearch-project/OpenSearch/pull/7919))
 - Improved performance of parsing floating point numbers ([#7909](https://github.com/opensearch-project/OpenSearch/pull/7909))
 - Move span actions to Scope ([#8411](https://github.com/opensearch-project/OpenSearch/pull/8411))
+- Add wrapper tracer implementation
 
 ### Deprecated
 
@@ -79,6 +78,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Replaces ZipInputStream with ZipFile to fix Zip Slip vulnerability ([#7230](https://github.com/opensearch-project/OpenSearch/pull/7230))
 - Add missing validation/parsing of SearchBackpressureMode of SearchBackpressureSettings ([#7541](https://github.com/opensearch-project/OpenSearch/pull/7541))
 - Adds log4j configuration for telemetry LogSpanExporter ([#8393](https://github.com/opensearch-project/OpenSearch/pull/8393))
+- Fix painless casting bug, which crashes the OpenSearch process ([#8315](https://github.com/opensearch-project/OpenSearch/pull/8315))
 
 ### Security
 
@@ -102,6 +102,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Support transport action names when registering NamedRoutes ([#7957](https://github.com/opensearch-project/OpenSearch/pull/7957))
 - Create concept of persistent ThreadContext headers that are unstashable ([#8291]()https://github.com/opensearch-project/OpenSearch/pull/8291)
 - Enable Partial Flat Object ([#7997](https://github.com/opensearch-project/OpenSearch/pull/7997))
+- Add jdk.incubator.vector module support for JDK 20+ ([#8601](https://github.com/opensearch-project/OpenSearch/pull/8601))
 
 ### Dependencies
 - Bump `com.azure:azure-storage-common` from 12.21.0 to 12.21.1 (#7566, #7814)
@@ -137,6 +138,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Update Apache HttpCore/ HttpClient and Apache HttpCore5 / HttpClient5 dependencies ([#8434](https://github.com/opensearch-project/OpenSearch/pull/8434))
 - Bump `org.apache.maven:maven-model` from 3.9.2 to 3.9.3 (#8403)
 - Bump `io.projectreactor.netty:reactor-netty` and `io.projectreactor.netty:reactor-netty-core` from 1.1.7 to 1.1.8 (#8405)
+- Bump `com.azure:azure-storage-blob` from 12.22.2 to 12.22.3 (#8572)
+- Bump `net.minidev:json-smart` from 2.4.11 to 2.5.0 ([#8576](https://github.com/opensearch-project/OpenSearch/pull/8576))
+- Bump `com.google.jimfs:jimfs` from 1.2 to 1.3.0 (#8577, #8571)
+- Bump `com.networknt:json-schema-validator` from 1.0.85 to 1.0.86 ([#8573](https://github.com/opensearch-project/OpenSearch/pull/8573))
+- Bump `com.google.cloud:google-cloud-core-http` from 2.17.0 to 2.21.0 ([#8586](https://github.com/opensearch-project/OpenSearch/pull/8586))
+- Bump `com.google.jimfs:jimfs` from 1.2 to 1.3.0 ([#8585](https://github.com/opensearch-project/OpenSearch/pull/8585))
 
 ### Changed
 - Replace jboss-annotations-api_1.2_spec with jakarta.annotation-api ([#7836](https://github.com/opensearch-project/OpenSearch/pull/7836))
@@ -159,6 +166,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Enabling compression levels for zstd and zstd_no_dict ([#8312](https://github.com/opensearch-project/OpenSearch/pull/8312))
 - Optimize Metadata build() to skip redundant computations as part of ClusterState build ([#7853](https://github.com/opensearch-project/OpenSearch/pull/7853))
 - Add safeguard limits for file cache during node level allocation ([#8208](https://github.com/opensearch-project/OpenSearch/pull/8208))
+- Move span actions to Scope ([#8411](https://github.com/opensearch-project/OpenSearch/pull/8411))
+- Add wrapper tracer implementation ([#8565](https://github.com/opensearch-project/OpenSearch/pull/8565))
 
 ### Deprecated
 
@@ -173,8 +182,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fix mapping char_filter when mapping a hashtag ([#7591](https://github.com/opensearch-project/OpenSearch/pull/7591))
 - Fix NPE in multiterms aggregations involving empty buckets ([#7318](https://github.com/opensearch-project/OpenSearch/pull/7318))
 - Precise system clock time in MasterService debug logs ([#7902](https://github.com/opensearch-project/OpenSearch/pull/7902))
+- Improve indexing performance for flat_object type ([#7855](https://github.com/opensearch-project/OpenSearch/pull/7855))
 
 ### Security
 
 [Unreleased 3.0]: https://github.com/opensearch-project/OpenSearch/compare/2.x...HEAD
-[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.8...2.x
+[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.8...2.x

build.gradle

@@ -424,6 +424,9 @@ gradle.projectsEvaluated {
         if (BuildParams.runtimeJavaVersion > JavaVersion.VERSION_17) {
           task.jvmArgs += ["-Djava.security.manager=allow"]
         }
+        if (BuildParams.runtimeJavaVersion >= JavaVersion.VERSION_20) {
+          task.jvmArgs += ["--add-modules=jdk.incubator.vector"]
+        }
       }
     }
 

buildSrc/build.gradle

@@ -34,6 +34,7 @@ import org.gradle.util.GradleVersion
 plugins {
   id 'java-gradle-plugin'
   id 'groovy'
+  id 'java-test-fixtures'
 }
 
 group = 'org.opensearch.gradle'
@@ -78,17 +79,9 @@ if (JavaVersion.current() < JavaVersion.VERSION_11) {
 }
 
 sourceSets {
-  test {
-    java {
-      srcDirs += ['src/testFixtures/java']
-    }
-  }
   integTest {
     compileClasspath += sourceSets["main"].output + configurations["testRuntimeClasspath"]
     runtimeClasspath += output + compileClasspath
-    java {
-      srcDirs += ['src/testFixtures/java']
-    }
   }
 }
 
@@ -125,16 +118,16 @@ dependencies {
   api 'com.avast.gradle:gradle-docker-compose-plugin:0.16.12'
   api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}"
   api 'org.apache.maven:maven-model:3.9.3'
-  api 'com.networknt:json-schema-validator:1.0.85'
+  api 'com.networknt:json-schema-validator:1.0.86'
   api 'org.jruby.jcodings:jcodings:1.0.58'
   api 'org.jruby.joni:joni:2.2.1'
   api "com.fasterxml.jackson.core:jackson-databind:${props.getProperty('jackson_databind')}"
   api "org.ajoberstar.grgit:grgit-core:5.2.0"
 
-  testImplementation "junit:junit:${props.getProperty('junit')}"
-  testImplementation "com.carrotsearch.randomizedtesting:randomizedtesting-runner:${props.getProperty('randomizedrunner')}"
-  testRuntimeOnly gradleApi()
-  testRuntimeOnly gradleTestKit()
+  testFixturesApi "junit:junit:${props.getProperty('junit')}"
+  testFixturesApi "com.carrotsearch.randomizedtesting:randomizedtesting-runner:${props.getProperty('randomizedrunner')}"
+  testFixturesApi gradleApi()
+  testFixturesApi gradleTestKit()
   testImplementation 'com.github.tomakehurst:wiremock-jre8-standalone:2.35.0'
   testImplementation "org.mockito:mockito-core:${props.getProperty('mockito')}"
   integTestImplementation('org.spockframework:spock-core:2.3-groovy-3.0') {
@@ -183,7 +176,7 @@ if (project != rootProject) {
   // build-tools is not ready for primetime with these...
   tasks.named("dependencyLicenses").configure { it.enabled = false }
   dependenciesInfo.enabled = false
-  disableTasks('forbiddenApisMain', 'forbiddenApisTest', 'forbiddenApisIntegTest')
+  disableTasks('forbiddenApisMain', 'forbiddenApisTest', 'forbiddenApisIntegTest', 'forbiddenApisTestFixtures')
   jarHell.enabled = false
   thirdPartyAudit.enabled = false
   if (org.opensearch.gradle.info.BuildParams.inFipsJvm) {
@@ -250,6 +243,12 @@ if (project != rootProject) {
     }
   }
 
+  // disable fail-on-warnings for this specific task which trips Java 11 bug
+  // https://bugs.openjdk.java.net/browse/JDK-8209058
+  tasks.named("compileTestFixturesJava").configure {
+    options.compilerArgs -= '-Werror'
+  }
+
   tasks.register("integTest", Test) {
     inputs.dir(file("src/testKit")).withPropertyName("testkit dir").withPathSensitivity(PathSensitivity.RELATIVE)
     systemProperty 'test.version_under_test', version

distribution/src/config/jvm.options

@@ -78,3 +78,7 @@ ${error.file}
 
 # Explicitly allow security manager (https://bugs.openjdk.java.net/browse/JDK-8270380)
 18-:-Djava.security.manager=allow
+
+# JDK 20+ Incubating Vector Module for SIMD optimizations;
+# disabling may reduce performance on vector optimized lucene
+20:--add-modules=jdk.incubator.vector

gradle/wrapper/gradle-wrapper.properties

@@ -11,7 +11,7 @@
 
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-8.2-all.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-8.2.1-all.zip
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
-distributionSha256Sum=5022b0b25fe182b0e50867e77f484501dba44feeea88f5c1f13b6b4660463640
+distributionSha256Sum=7c3ad722e9b0ce8205b91560fd6ce8296ac3eadf065672242fd73c06b8eeb6ee

server/src/main/java/org/opensearch/common/io/InputStreamContainer.java → libs/common/src/main/java/org/opensearch/common/io/InputStreamContainer.java

@@ -19,16 +19,18 @@ public class InputStreamContainer {
 
     private final InputStream inputStream;
     private final long contentLength;
+    private final long offset;
 
     /**
      * Construct a new stream object
      *
      * @param inputStream   The input stream that is to be encapsulated
      * @param contentLength The total content length that is to be read from the stream
      */
-    public InputStreamContainer(InputStream inputStream, long contentLength) {
+    public InputStreamContainer(InputStream inputStream, long contentLength, long offset) {
         this.inputStream = inputStream;
         this.contentLength = contentLength;
+        this.offset = offset;
     }
 
     /**
@@ -44,4 +46,11 @@ public InputStream getInputStream() {
     public long getContentLength() {
         return contentLength;
     }
+
+    /**
+     * @return offset of the source content.
+     */
+    public long getOffset() {
+        return offset;
+    }
 }

libs/telemetry/src/main/java/org/opensearch/telemetry/Telemetry.java

@@ -13,6 +13,8 @@
 
 /**
  * Interface defining telemetry
+ *
+ * @opensearch.internal
  */
 public interface Telemetry {
 

libs/telemetry/src/main/java/org/opensearch/telemetry/tracing/AbstractSpan.java

@@ -10,6 +10,8 @@
 
 /**
  * Base span
+ *
+ * @opensearch.internal
  */
 public abstract class AbstractSpan implements Span {
 

libs/telemetry/src/main/java/org/opensearch/telemetry/tracing/DefaultSpanScope.java

@@ -12,8 +12,10 @@
 
 /**
  * Default implementation of Scope
+ *
+ * @opensearch.internal
  */
-public class DefaultSpanScope implements SpanScope {
+final class DefaultSpanScope implements SpanScope {
 
     private final Span span;
 

libs/telemetry/src/main/java/org/opensearch/telemetry/tracing/DefaultTracer.java

@@ -16,9 +16,9 @@
  * The default tracer implementation. It handles tracing context propagation between spans by maintaining
  * current active span in its storage
  *
- *
+ *  @opensearch.internal
  */
-public class DefaultTracer implements Tracer {
+class DefaultTracer implements Tracer {
     static final String THREAD_NAME = "th_name";
 
     private final TracingTelemetry tracingTelemetry;

libs/telemetry/src/main/java/org/opensearch/telemetry/tracing/Span.java

@@ -11,7 +11,9 @@
 /**
  * An interface that represents a tracing span.
  * Spans are created by the Tracer.startSpan method.
- * Span must be ended by calling Tracer.endSpan which internally calls Span's endSpan.
+ * Span must be ended by calling SpanScope.close which internally calls Span's endSpan.
+ *
+ * @opensearch.internal
  */
 public interface Span {
 

libs/telemetry/src/main/java/org/opensearch/telemetry/tracing/SpanReference.java

@@ -10,8 +10,10 @@
 
 /**
  * Wrapper class to hold reference of Span
+ *
+ * @opensearch.internal
  */
-public class SpanReference {
+final class SpanReference {
 
     private Span span;
 

libs/telemetry/src/main/java/org/opensearch/telemetry/tracing/TracerContextStorage.java

@@ -12,6 +12,8 @@
  * Storage interface used for storing tracing context
  * @param <K> key type
  * @param <V> value type
+ *
+ * @opensearch.internal
  */
 public interface TracerContextStorage<K, V> {
     /**

libs/telemetry/src/main/java/org/opensearch/telemetry/tracing/TracingContextPropagator.java

@@ -13,6 +13,8 @@
 
 /**
  * Interface defining the tracing related context propagation
+ *
+ * @opensearch.internal
  */
 public interface TracingContextPropagator {
 

libs/telemetry/src/main/java/org/opensearch/telemetry/tracing/TracingTelemetry.java

@@ -12,6 +12,8 @@
 
 /**
  * Interface for tracing telemetry providers
+ *
+ * @opensearch.internal
  */
 public interface TracingTelemetry extends Closeable {
 

libs/telemetry/src/main/java/org/opensearch/telemetry/tracing/noop/NoopSpanScope.java

@@ -12,6 +12,8 @@
 
 /**
  * No-op implementation of SpanScope
+ *
+ * @opensearch.internal
  */
 public final class NoopSpanScope implements SpanScope {
 

libs/telemetry/src/main/java/org/opensearch/telemetry/tracing/noop/NoopTracer.java

@@ -13,6 +13,8 @@
 
 /**
  * No-op implementation of Tracer
+ *
+ * @opensearch.internal
  */
 public class NoopTracer implements Tracer {
 

modules/lang-painless/src/main/java/org/opensearch/painless/AnalyzerCaster.java

@@ -412,7 +412,7 @@ public static PainlessCast getLegalCast(Location location, Class<?> actual, Clas
             }
         }
 
-        if (actual == def.class
+        if ((actual == def.class && expected != void.class)
             || (actual != void.class && expected == def.class)
             || expected.isAssignableFrom(actual)
             || (actual.isAssignableFrom(expected) && explicit)) {

modules/lang-painless/src/test/java/org/opensearch/painless/FactoryTests.java

@@ -360,6 +360,14 @@ public void testVoidReturn() {
         assertEquals(iae.getMessage(), "not a statement: result not used from addition operation [+]");
     }
 
+    public void testDefToVoidReturnThrowsException() {
+        ClassCastException exception = expectScriptThrows(
+            ClassCastException.class,
+            () -> getEngine().compile("def_return_in_void", "def x=1;return x;", VoidReturnTestScript.CONTEXT, Collections.emptyMap())
+        );
+        assertEquals(exception.getMessage(), "Cannot cast from [def] to [void].");
+    }
+
     public abstract static class FactoryTestConverterScript {
         private final Map<String, Object> params;