Merge branch 'opensearch-project:main' into painless-bug

CHANGELOG.md

@@ -37,10 +37,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - OpenJDK Update (April 2023 Patch releases) ([#7344](https://github.com/opensearch-project/OpenSearch/pull/7344)
 - Bump `com.google.http-client:google-http-client:1.43.2` from 1.42.0 to 1.43.2 ([7928](https://github.com/opensearch-project/OpenSearch/pull/7928)))
 - Add Opentelemetry dependencies ([#7543](https://github.com/opensearch-project/OpenSearch/issues/7543))
-- Bump `org.bouncycastle:bcprov-jdk15on` to `org.bouncycastle:bcprov-jdk15to18` version 1.75 ([8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
-- Bump `org.bouncycastle:bcmail-jdk15on` to `org.bouncycastle:bcmail-jdk15to18` version 1.75 ([8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
-- Bump `org.bouncycastle:bcpkix-jdk15on` to `org.bouncycastle:bcpkix-jdk15to18` version 1.75 ([8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
-
+- Bump `org.bouncycastle:bcprov-jdk15on` to `org.bouncycastle:bcprov-jdk15to18` version 1.75 ([#8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
+- Bump `org.bouncycastle:bcmail-jdk15on` to `org.bouncycastle:bcmail-jdk15to18` version 1.75 ([#8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
+- Bump `org.bouncycastle:bcpkix-jdk15on` to `org.bouncycastle:bcpkix-jdk15to18` version 1.75 ([#8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
+- Bump `com.google.cloud:google-cloud-core-http` from 2.17.0 to 2.21.0 ([#8586](https://github.com/opensearch-project/OpenSearch/pull/8586))
 
 
 ### Changed
@@ -103,10 +103,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Support transport action names when registering NamedRoutes ([#7957](https://github.com/opensearch-project/OpenSearch/pull/7957))
 - Create concept of persistent ThreadContext headers that are unstashable ([#8291]()https://github.com/opensearch-project/OpenSearch/pull/8291)
 - Enable Partial Flat Object ([#7997](https://github.com/opensearch-project/OpenSearch/pull/7997))
+- Add jdk.incubator.vector module support for JDK 20+ ([#8601](https://github.com/opensearch-project/OpenSearch/pull/8601))
 
 ### Dependencies
 - Bump `com.azure:azure-storage-common` from 12.21.0 to 12.21.1 (#7566, #7814)
-- Bump `com.google.guava:guava` from 30.1.1-jre to 32.1.1-jre (#7565, #7811, #7807, #7808, #8402, #8400, #8401)
+- Bump `com.google.guava:guava` from 30.1.1-jre to 32.1.1-jre (#7565, #7811, #7807, #7808, #8402, #8400, #8401, #8581)
 - Bump `net.minidev:json-smart` from 2.4.10 to 2.4.11 (#7660, #7812)
 - Bump `org.gradle.test-retry` from 1.5.2 to 1.5.3 (#7810)
 - Bump `com.diffplug.spotless` from 6.17.0 to 6.18.0 (#7896)
@@ -126,7 +127,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `io.projectreactor:reactor-core` from 3.4.18 to 3.5.6 in /plugins/repository-azure ([#8016](https://github.com/opensearch-project/OpenSearch/pull/8016))
 - Bump `spock-core` from 2.1-groovy-3.0 to 2.3-groovy-3.0 ([#8122](https://github.com/opensearch-project/OpenSearch/pull/8122))
 - Bump `com.networknt:json-schema-validator` from 1.0.83 to 1.0.84 (#8141)
-- Bump `com.netflix.nebula:gradle-info-plugin` from 12.1.3 to 12.1.4 (#8139)
+- Bump `com.netflix.nebula:gradle-info-plugin` from 12.1.3 to 12.1.5 (#8139, #8568)
 - Bump `commons-io:commons-io` from 2.12.0 to 2.13.0 in /plugins/discovery-azure-classic ([#8140](https://github.com/opensearch-project/OpenSearch/pull/8140))
 - Bump `mockito` from 5.2.0 to 5.4.0 ([#8181](https://github.com/opensearch-project/OpenSearch/pull/8181))
 - Bump `netty` from 4.1.93.Final to 4.1.94.Final ([#8191](https://github.com/opensearch-project/OpenSearch/pull/8191))
@@ -138,6 +139,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Update Apache HttpCore/ HttpClient and Apache HttpCore5 / HttpClient5 dependencies ([#8434](https://github.com/opensearch-project/OpenSearch/pull/8434))
 - Bump `org.apache.maven:maven-model` from 3.9.2 to 3.9.3 (#8403)
 - Bump `io.projectreactor.netty:reactor-netty` and `io.projectreactor.netty:reactor-netty-core` from 1.1.7 to 1.1.8 (#8405)
+- Bump `com.azure:azure-storage-blob` from 12.22.2 to 12.22.3 (#8572)
+- Bump `com.google.jimfs:jimfs` from 1.2 to 1.3.0 (#8577, #8571)
+- Bump `com.networknt:json-schema-validator` from 1.0.85 to 1.0.86 ([#8573](https://github.com/opensearch-project/OpenSearch/pull/8573))
 
 ### Changed
 - Replace jboss-annotations-api_1.2_spec with jakarta.annotation-api ([#7836](https://github.com/opensearch-project/OpenSearch/pull/7836))
@@ -159,6 +163,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - [Search Pipelines] Pass pipeline creation context to processor factories ([#8164](https://github.com/opensearch-project/OpenSearch/pull/8164))
 - Enabling compression levels for zstd and zstd_no_dict ([#8312](https://github.com/opensearch-project/OpenSearch/pull/8312))
 - Optimize Metadata build() to skip redundant computations as part of ClusterState build ([#7853](https://github.com/opensearch-project/OpenSearch/pull/7853))
+- Add safeguard limits for file cache during node level allocation ([#8208](https://github.com/opensearch-project/OpenSearch/pull/8208))
 
 ### Deprecated
 
@@ -173,6 +178,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fix mapping char_filter when mapping a hashtag ([#7591](https://github.com/opensearch-project/OpenSearch/pull/7591))
 - Fix NPE in multiterms aggregations involving empty buckets ([#7318](https://github.com/opensearch-project/OpenSearch/pull/7318))
 - Precise system clock time in MasterService debug logs ([#7902](https://github.com/opensearch-project/OpenSearch/pull/7902))
+- Improve indexing performance for flat_object type ([#7855](https://github.com/opensearch-project/OpenSearch/pull/7855))
 
 ### Security
 

build.gradle

@@ -424,6 +424,9 @@ gradle.projectsEvaluated {
         if (BuildParams.runtimeJavaVersion > JavaVersion.VERSION_17) {
           task.jvmArgs += ["-Djava.security.manager=allow"]
         }
+        if (BuildParams.runtimeJavaVersion >= JavaVersion.VERSION_20) {
+          task.jvmArgs += ["--add-modules=jdk.incubator.vector"]
+        }
       }
     }
 

buildSrc/build.gradle

@@ -34,6 +34,7 @@ import org.gradle.util.GradleVersion
 plugins {
   id 'java-gradle-plugin'
   id 'groovy'
+  id 'java-test-fixtures'
 }
 
 group = 'org.opensearch.gradle'
@@ -78,17 +79,9 @@ if (JavaVersion.current() < JavaVersion.VERSION_11) {
 }
 
 sourceSets {
-  test {
-    java {
-      srcDirs += ['src/testFixtures/java']
-    }
-  }
   integTest {
     compileClasspath += sourceSets["main"].output + configurations["testRuntimeClasspath"]
     runtimeClasspath += output + compileClasspath
-    java {
-      srcDirs += ['src/testFixtures/java']
-    }
   }
 }
 
@@ -114,7 +107,7 @@ dependencies {
   api 'org.apache.ant:ant:1.10.13'
   api 'com.netflix.nebula:gradle-extra-configurations-plugin:10.0.0'
   api 'com.netflix.nebula:nebula-publishing-plugin:20.3.0'
-  api 'com.netflix.nebula:gradle-info-plugin:12.1.4'
+  api 'com.netflix.nebula:gradle-info-plugin:12.1.5'
   api 'org.apache.rat:apache-rat:0.15'
   api 'commons-io:commons-io:2.13.0'
   api "net.java.dev.jna:jna:5.13.0"
@@ -125,16 +118,16 @@ dependencies {
   api 'com.avast.gradle:gradle-docker-compose-plugin:0.16.12'
   api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}"
   api 'org.apache.maven:maven-model:3.9.3'
-  api 'com.networknt:json-schema-validator:1.0.85'
+  api 'com.networknt:json-schema-validator:1.0.86'
   api 'org.jruby.jcodings:jcodings:1.0.58'
   api 'org.jruby.joni:joni:2.2.1'
   api "com.fasterxml.jackson.core:jackson-databind:${props.getProperty('jackson_databind')}"
   api "org.ajoberstar.grgit:grgit-core:5.2.0"
 
-  testImplementation "junit:junit:${props.getProperty('junit')}"
-  testImplementation "com.carrotsearch.randomizedtesting:randomizedtesting-runner:${props.getProperty('randomizedrunner')}"
-  testRuntimeOnly gradleApi()
-  testRuntimeOnly gradleTestKit()
+  testFixturesApi "junit:junit:${props.getProperty('junit')}"
+  testFixturesApi "com.carrotsearch.randomizedtesting:randomizedtesting-runner:${props.getProperty('randomizedrunner')}"
+  testFixturesApi gradleApi()
+  testFixturesApi gradleTestKit()
   testImplementation 'com.github.tomakehurst:wiremock-jre8-standalone:2.35.0'
   testImplementation "org.mockito:mockito-core:${props.getProperty('mockito')}"
   integTestImplementation('org.spockframework:spock-core:2.3-groovy-3.0') {
@@ -183,7 +176,7 @@ if (project != rootProject) {
   // build-tools is not ready for primetime with these...
   tasks.named("dependencyLicenses").configure { it.enabled = false }
   dependenciesInfo.enabled = false
-  disableTasks('forbiddenApisMain', 'forbiddenApisTest', 'forbiddenApisIntegTest')
+  disableTasks('forbiddenApisMain', 'forbiddenApisTest', 'forbiddenApisIntegTest', 'forbiddenApisTestFixtures')
   jarHell.enabled = false
   thirdPartyAudit.enabled = false
   if (org.opensearch.gradle.info.BuildParams.inFipsJvm) {
@@ -250,6 +243,12 @@ if (project != rootProject) {
     }
   }
 
+  // disable fail-on-warnings for this specific task which trips Java 11 bug
+  // https://bugs.openjdk.java.net/browse/JDK-8209058
+  tasks.named("compileTestFixturesJava").configure {
+    options.compilerArgs -= '-Werror'
+  }
+
   tasks.register("integTest", Test) {
     inputs.dir(file("src/testKit")).withPropertyName("testkit dir").withPathSensitivity(PathSensitivity.RELATIVE)
     systemProperty 'test.version_under_test', version

buildSrc/version.properties

@@ -22,7 +22,7 @@ jettison          = 1.5.4
 woodstox          = 6.4.0
 kotlin            = 1.7.10
 antlr4            = 4.11.1
-guava             = 32.0.1-jre
+guava             = 32.1.1-jre
 protobuf          = 3.22.3
 jakarta_annotation = 1.3.5
 

distribution/src/config/jvm.options

@@ -78,3 +78,7 @@ ${error.file}
 
 # Explicitly allow security manager (https://bugs.openjdk.java.net/browse/JDK-8270380)
 18-:-Djava.security.manager=allow
+
+# JDK 20+ Incubating Vector Module for SIMD optimizations;
+# disabling may reduce performance on vector optimized lucene
+20:--add-modules=jdk.incubator.vector

distribution/tools/keystore-cli/build.gradle

@@ -34,8 +34,8 @@ dependencies {
   compileOnly project(":server")
   compileOnly project(":libs:opensearch-cli")
   testImplementation project(":test:framework")
-  testImplementation 'com.google.jimfs:jimfs:1.2'
-  testRuntimeOnly('com.google.guava:guava:32.1.1-jre') {
+  testImplementation 'com.google.jimfs:jimfs:1.3.0'
+  testRuntimeOnly("com.google.guava:guava:${versions.guava}") {
     transitive = false
   }
 }

distribution/tools/plugin-cli/build.gradle

@@ -41,7 +41,7 @@ dependencies {
   api "org.bouncycastle:bc-fips:1.0.2.3"
   testImplementation project(":test:framework")
   testImplementation 'com.google.jimfs:jimfs:1.2'
-  testRuntimeOnly('com.google.guava:guava:32.1.1-jre') {
+  testRuntimeOnly("com.google.guava:guava:${versions.guava}") {
     transitive = false
   }
 

distribution/tools/upgrade-cli/build.gradle

@@ -20,8 +20,8 @@ dependencies {
   implementation "com.fasterxml.jackson.core:jackson-databind:${versions.jackson_databind}"
   implementation "com.fasterxml.jackson.core:jackson-annotations:${versions.jackson}"
   testImplementation project(":test:framework")
-  testImplementation 'com.google.jimfs:jimfs:1.2'
-  testRuntimeOnly('com.google.guava:guava:32.1.1-jre') {
+  testImplementation 'com.google.jimfs:jimfs:1.3.0'
+  testRuntimeOnly("com.google.guava:guava:${versions.guava}") {
     transitive = false
   }
 }

gradle/wrapper/gradle-wrapper.properties

@@ -11,7 +11,7 @@
 
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-8.2-all.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-8.2.1-all.zip
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
-distributionSha256Sum=5022b0b25fe182b0e50867e77f484501dba44feeea88f5c1f13b6b4660463640
+distributionSha256Sum=7c3ad722e9b0ce8205b91560fd6ce8296ac3eadf065672242fd73c06b8eeb6ee

modules/lang-painless/src/yamlRestTest/resources/rest-api-spec/test/painless/30_search.yml

@@ -482,100 +482,3 @@
                         }]
     - match: { error.root_cause.0.type: "illegal_argument_exception" }
     - match: { error.root_cause.0.reason: "script score function must not produce negative scores, but got: [-9.0]"}
-
----
-
-"Flat-object fields from within the scripting":
-  - skip:
-      version: " - 2.6.99"
-      reason: "flat_object is introduced in 2.7.0"
-
-  - do:
-      indices.create:
-        index: test
-        body:
-          mappings:
-            properties:
-              flat:
-                type : "flat_object"
-
-  # This document has 6 distinct parts in its flat_object field paths:
-  # - flat.field_1
-  # - flat.field_2
-  # - flat.field_3
-  # - flat.inner
-  # - flat.field_A
-  # - flat.field_B
-  - do:
-      index:
-        index: test
-        id:    1
-        body:  {
-          "flat": {
-            "field_1": "John Doe",
-            "field_2": 33,
-            "field_3": false,
-            "inner": {
-              "field_A": ["foo", "bar"],
-              "field_B": false
-            }
-          }
-        }
-
-  - do:
-      index:
-        index: test
-        id:    2
-        body:  {
-          "flat": {
-            "field_1": "Joe Public",
-            "field_2": 45
-          }
-        }
-
-  - do:
-      indices.refresh:
-        index: test
-
-  # It is possible to filter based on the number of distinct parts of flat_object field paths
-  - do:
-      search:
-        body: {
-          _source: true,
-          query: {
-            bool: {
-              filter: {
-                script: {
-                  script: {
-                    source: "doc['flat'].size() == 6",
-                    lang: "painless"
-                  }
-                }
-              }
-            }
-          }
-        }
-
-  - length:   { hits.hits: 1  }
-  - match: { hits.hits.0._source.flat.field_1: "John Doe" }
-
-  - do:
-      search:
-        body: {
-          _source: true,
-          query: {
-            bool: {
-              filter: {
-                script: {
-                  script: {
-                    source: "doc['flat'].size() < 6",
-                    lang: "painless"
-                  }
-                }
-              }
-            }
-          }
-        }
-
-  - length:   { hits.hits: 1  }
-  - match: { hits.hits.0._source.flat.field_1: "Joe Public" }

plugins/ingest-attachment/licenses/guava-32.1.1-jre.jar.sha1

@@ -0,0 +1 @@
+ad575652d84153075dd41ec6177ccb15251262b2

plugins/repository-azure/build.gradle

@@ -55,7 +55,7 @@ dependencies {
   api "io.netty:netty-resolver-dns:${versions.netty}"
   api "io.netty:netty-transport-native-unix-common:${versions.netty}"
   implementation project(':modules:transport-netty4')
-  api 'com.azure:azure-storage-blob:12.22.2'
+  api 'com.azure:azure-storage-blob:12.22.3'
   api 'org.reactivestreams:reactive-streams:1.0.4'
   api 'io.projectreactor:reactor-core:3.5.6'
   api 'io.projectreactor.netty:reactor-netty:1.1.8'

plugins/repository-azure/licenses/azure-storage-blob-12.22.3.jar.sha1

@@ -0,0 +1 @@
+0df12462c2eac3beaf25d283f707a0560853228b

plugins/repository-azure/licenses/guava-32.1.1-jre.jar.sha1

@@ -0,0 +1 @@
+ad575652d84153075dd41ec6177ccb15251262b2

plugins/repository-gcs/build.gradle

@@ -67,7 +67,7 @@ dependencies {
   api "com.google.auth:google-auth-library-oauth2-http:${versions.google_auth}"
 
   api 'com.google.cloud:google-cloud-core:2.5.10'
-  api 'com.google.cloud:google-cloud-core-http:2.17.0'
+  api 'com.google.cloud:google-cloud-core-http:2.21.0'
   api 'com.google.cloud:google-cloud-storage:1.113.1'
 
   api 'com.google.code.gson:gson:2.9.0'

plugins/repository-gcs/licenses/google-cloud-core-http-2.21.0.jar.sha1

@@ -0,0 +1 @@
+07da4710ccdbcfee253672c0b9e00e7370626c26

plugins/repository-gcs/licenses/guava-32.1.1-jre.jar.sha1

@@ -0,0 +1 @@
+ad575652d84153075dd41ec6177ccb15251262b2

plugins/repository-hdfs/licenses/guava-32.1.1-jre.jar.sha1

@@ -0,0 +1 @@
+ad575652d84153075dd41ec6177ccb15251262b2