Skip to content

Commit

Permalink
Merge branch 'opensearch-project:main' into painless-bug
Browse files Browse the repository at this point in the history
  • Loading branch information
shiv0408 committed Jul 10, 2023
2 parents b73ae52 + 397069f commit c4519e0
Show file tree
Hide file tree
Showing 43 changed files with 566 additions and 187 deletions.
18 changes: 12 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- OpenJDK Update (April 2023 Patch releases) ([#7344](https://github.com/opensearch-project/OpenSearch/pull/7344)
- Bump `com.google.http-client:google-http-client:1.43.2` from 1.42.0 to 1.43.2 ([7928](https://github.com/opensearch-project/OpenSearch/pull/7928)))
- Add Opentelemetry dependencies ([#7543](https://github.com/opensearch-project/OpenSearch/issues/7543))
- Bump `org.bouncycastle:bcprov-jdk15on` to `org.bouncycastle:bcprov-jdk15to18` version 1.75 ([8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
- Bump `org.bouncycastle:bcmail-jdk15on` to `org.bouncycastle:bcmail-jdk15to18` version 1.75 ([8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
- Bump `org.bouncycastle:bcpkix-jdk15on` to `org.bouncycastle:bcpkix-jdk15to18` version 1.75 ([8247](https://github.com/opensearch-project/OpenSearch/pull/8247))

- Bump `org.bouncycastle:bcprov-jdk15on` to `org.bouncycastle:bcprov-jdk15to18` version 1.75 ([#8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
- Bump `org.bouncycastle:bcmail-jdk15on` to `org.bouncycastle:bcmail-jdk15to18` version 1.75 ([#8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
- Bump `org.bouncycastle:bcpkix-jdk15on` to `org.bouncycastle:bcpkix-jdk15to18` version 1.75 ([#8247](https://github.com/opensearch-project/OpenSearch/pull/8247))
- Bump `com.google.cloud:google-cloud-core-http` from 2.17.0 to 2.21.0 ([#8586](https://github.com/opensearch-project/OpenSearch/pull/8586))


### Changed
Expand Down Expand Up @@ -103,10 +103,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Support transport action names when registering NamedRoutes ([#7957](https://github.com/opensearch-project/OpenSearch/pull/7957))
- Create concept of persistent ThreadContext headers that are unstashable ([#8291]()https://github.com/opensearch-project/OpenSearch/pull/8291)
- Enable Partial Flat Object ([#7997](https://github.com/opensearch-project/OpenSearch/pull/7997))
- Add jdk.incubator.vector module support for JDK 20+ ([#8601](https://github.com/opensearch-project/OpenSearch/pull/8601))

### Dependencies
- Bump `com.azure:azure-storage-common` from 12.21.0 to 12.21.1 (#7566, #7814)
- Bump `com.google.guava:guava` from 30.1.1-jre to 32.1.1-jre (#7565, #7811, #7807, #7808, #8402, #8400, #8401)
- Bump `com.google.guava:guava` from 30.1.1-jre to 32.1.1-jre (#7565, #7811, #7807, #7808, #8402, #8400, #8401, #8581)
- Bump `net.minidev:json-smart` from 2.4.10 to 2.4.11 (#7660, #7812)
- Bump `org.gradle.test-retry` from 1.5.2 to 1.5.3 (#7810)
- Bump `com.diffplug.spotless` from 6.17.0 to 6.18.0 (#7896)
Expand All @@ -126,7 +127,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Bump `io.projectreactor:reactor-core` from 3.4.18 to 3.5.6 in /plugins/repository-azure ([#8016](https://github.com/opensearch-project/OpenSearch/pull/8016))
- Bump `spock-core` from 2.1-groovy-3.0 to 2.3-groovy-3.0 ([#8122](https://github.com/opensearch-project/OpenSearch/pull/8122))
- Bump `com.networknt:json-schema-validator` from 1.0.83 to 1.0.84 (#8141)
- Bump `com.netflix.nebula:gradle-info-plugin` from 12.1.3 to 12.1.4 (#8139)
- Bump `com.netflix.nebula:gradle-info-plugin` from 12.1.3 to 12.1.5 (#8139, #8568)
- Bump `commons-io:commons-io` from 2.12.0 to 2.13.0 in /plugins/discovery-azure-classic ([#8140](https://github.com/opensearch-project/OpenSearch/pull/8140))
- Bump `mockito` from 5.2.0 to 5.4.0 ([#8181](https://github.com/opensearch-project/OpenSearch/pull/8181))
- Bump `netty` from 4.1.93.Final to 4.1.94.Final ([#8191](https://github.com/opensearch-project/OpenSearch/pull/8191))
Expand All @@ -138,6 +139,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Update Apache HttpCore/ HttpClient and Apache HttpCore5 / HttpClient5 dependencies ([#8434](https://github.com/opensearch-project/OpenSearch/pull/8434))
- Bump `org.apache.maven:maven-model` from 3.9.2 to 3.9.3 (#8403)
- Bump `io.projectreactor.netty:reactor-netty` and `io.projectreactor.netty:reactor-netty-core` from 1.1.7 to 1.1.8 (#8405)
- Bump `com.azure:azure-storage-blob` from 12.22.2 to 12.22.3 (#8572)
- Bump `com.google.jimfs:jimfs` from 1.2 to 1.3.0 (#8577, #8571)
- Bump `com.networknt:json-schema-validator` from 1.0.85 to 1.0.86 ([#8573](https://github.com/opensearch-project/OpenSearch/pull/8573))

### Changed
- Replace jboss-annotations-api_1.2_spec with jakarta.annotation-api ([#7836](https://github.com/opensearch-project/OpenSearch/pull/7836))
Expand All @@ -159,6 +163,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- [Search Pipelines] Pass pipeline creation context to processor factories ([#8164](https://github.com/opensearch-project/OpenSearch/pull/8164))
- Enabling compression levels for zstd and zstd_no_dict ([#8312](https://github.com/opensearch-project/OpenSearch/pull/8312))
- Optimize Metadata build() to skip redundant computations as part of ClusterState build ([#7853](https://github.com/opensearch-project/OpenSearch/pull/7853))
- Add safeguard limits for file cache during node level allocation ([#8208](https://github.com/opensearch-project/OpenSearch/pull/8208))

### Deprecated

Expand All @@ -173,6 +178,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Fix mapping char_filter when mapping a hashtag ([#7591](https://github.com/opensearch-project/OpenSearch/pull/7591))
- Fix NPE in multiterms aggregations involving empty buckets ([#7318](https://github.com/opensearch-project/OpenSearch/pull/7318))
- Precise system clock time in MasterService debug logs ([#7902](https://github.com/opensearch-project/OpenSearch/pull/7902))
- Improve indexing performance for flat_object type ([#7855](https://github.com/opensearch-project/OpenSearch/pull/7855))

### Security

Expand Down
3 changes: 3 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -424,6 +424,9 @@ gradle.projectsEvaluated {
if (BuildParams.runtimeJavaVersion > JavaVersion.VERSION_17) {
task.jvmArgs += ["-Djava.security.manager=allow"]
}
if (BuildParams.runtimeJavaVersion >= JavaVersion.VERSION_20) {
task.jvmArgs += ["--add-modules=jdk.incubator.vector"]
}
}
}

Expand Down
29 changes: 14 additions & 15 deletions buildSrc/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ import org.gradle.util.GradleVersion
plugins {
id 'java-gradle-plugin'
id 'groovy'
id 'java-test-fixtures'
}

group = 'org.opensearch.gradle'
Expand Down Expand Up @@ -78,17 +79,9 @@ if (JavaVersion.current() < JavaVersion.VERSION_11) {
}

sourceSets {
test {
java {
srcDirs += ['src/testFixtures/java']
}
}
integTest {
compileClasspath += sourceSets["main"].output + configurations["testRuntimeClasspath"]
runtimeClasspath += output + compileClasspath
java {
srcDirs += ['src/testFixtures/java']
}
}
}

Expand All @@ -114,7 +107,7 @@ dependencies {
api 'org.apache.ant:ant:1.10.13'
api 'com.netflix.nebula:gradle-extra-configurations-plugin:10.0.0'
api 'com.netflix.nebula:nebula-publishing-plugin:20.3.0'
api 'com.netflix.nebula:gradle-info-plugin:12.1.4'
api 'com.netflix.nebula:gradle-info-plugin:12.1.5'
api 'org.apache.rat:apache-rat:0.15'
api 'commons-io:commons-io:2.13.0'
api "net.java.dev.jna:jna:5.13.0"
Expand All @@ -125,16 +118,16 @@ dependencies {
api 'com.avast.gradle:gradle-docker-compose-plugin:0.16.12'
api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}"
api 'org.apache.maven:maven-model:3.9.3'
api 'com.networknt:json-schema-validator:1.0.85'
api 'com.networknt:json-schema-validator:1.0.86'
api 'org.jruby.jcodings:jcodings:1.0.58'
api 'org.jruby.joni:joni:2.2.1'
api "com.fasterxml.jackson.core:jackson-databind:${props.getProperty('jackson_databind')}"
api "org.ajoberstar.grgit:grgit-core:5.2.0"

testImplementation "junit:junit:${props.getProperty('junit')}"
testImplementation "com.carrotsearch.randomizedtesting:randomizedtesting-runner:${props.getProperty('randomizedrunner')}"
testRuntimeOnly gradleApi()
testRuntimeOnly gradleTestKit()
testFixturesApi "junit:junit:${props.getProperty('junit')}"
testFixturesApi "com.carrotsearch.randomizedtesting:randomizedtesting-runner:${props.getProperty('randomizedrunner')}"
testFixturesApi gradleApi()
testFixturesApi gradleTestKit()
testImplementation 'com.github.tomakehurst:wiremock-jre8-standalone:2.35.0'
testImplementation "org.mockito:mockito-core:${props.getProperty('mockito')}"
integTestImplementation('org.spockframework:spock-core:2.3-groovy-3.0') {
Expand Down Expand Up @@ -183,7 +176,7 @@ if (project != rootProject) {
// build-tools is not ready for primetime with these...
tasks.named("dependencyLicenses").configure { it.enabled = false }
dependenciesInfo.enabled = false
disableTasks('forbiddenApisMain', 'forbiddenApisTest', 'forbiddenApisIntegTest')
disableTasks('forbiddenApisMain', 'forbiddenApisTest', 'forbiddenApisIntegTest', 'forbiddenApisTestFixtures')
jarHell.enabled = false
thirdPartyAudit.enabled = false
if (org.opensearch.gradle.info.BuildParams.inFipsJvm) {
Expand Down Expand Up @@ -250,6 +243,12 @@ if (project != rootProject) {
}
}

// disable fail-on-warnings for this specific task which trips Java 11 bug
// https://bugs.openjdk.java.net/browse/JDK-8209058
tasks.named("compileTestFixturesJava").configure {
options.compilerArgs -= '-Werror'
}

tasks.register("integTest", Test) {
inputs.dir(file("src/testKit")).withPropertyName("testkit dir").withPathSensitivity(PathSensitivity.RELATIVE)
systemProperty 'test.version_under_test', version
Expand Down
2 changes: 1 addition & 1 deletion buildSrc/version.properties
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jettison = 1.5.4
woodstox = 6.4.0
kotlin = 1.7.10
antlr4 = 4.11.1
guava = 32.0.1-jre
guava = 32.1.1-jre
protobuf = 3.22.3
jakarta_annotation = 1.3.5

Expand Down
4 changes: 4 additions & 0 deletions distribution/src/config/jvm.options
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,7 @@ ${error.file}

# Explicitly allow security manager (https://bugs.openjdk.java.net/browse/JDK-8270380)
18-:-Djava.security.manager=allow

# JDK 20+ Incubating Vector Module for SIMD optimizations;
# disabling may reduce performance on vector optimized lucene
20:--add-modules=jdk.incubator.vector
4 changes: 2 additions & 2 deletions distribution/tools/keystore-cli/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ dependencies {
compileOnly project(":server")
compileOnly project(":libs:opensearch-cli")
testImplementation project(":test:framework")
testImplementation 'com.google.jimfs:jimfs:1.2'
testRuntimeOnly('com.google.guava:guava:32.1.1-jre') {
testImplementation 'com.google.jimfs:jimfs:1.3.0'
testRuntimeOnly("com.google.guava:guava:${versions.guava}") {
transitive = false
}
}
2 changes: 1 addition & 1 deletion distribution/tools/plugin-cli/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ dependencies {
api "org.bouncycastle:bc-fips:1.0.2.3"
testImplementation project(":test:framework")
testImplementation 'com.google.jimfs:jimfs:1.2'
testRuntimeOnly('com.google.guava:guava:32.1.1-jre') {
testRuntimeOnly("com.google.guava:guava:${versions.guava}") {
transitive = false
}

Expand Down
4 changes: 2 additions & 2 deletions distribution/tools/upgrade-cli/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ dependencies {
implementation "com.fasterxml.jackson.core:jackson-databind:${versions.jackson_databind}"
implementation "com.fasterxml.jackson.core:jackson-annotations:${versions.jackson}"
testImplementation project(":test:framework")
testImplementation 'com.google.jimfs:jimfs:1.2'
testRuntimeOnly('com.google.guava:guava:32.1.1-jre') {
testImplementation 'com.google.jimfs:jimfs:1.3.0'
testRuntimeOnly("com.google.guava:guava:${versions.guava}") {
transitive = false
}
}
Expand Down
4 changes: 2 additions & 2 deletions gradle/wrapper/gradle-wrapper.properties
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-8.2-all.zip
distributionUrl=https\://services.gradle.org/distributions/gradle-8.2.1-all.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
distributionSha256Sum=5022b0b25fe182b0e50867e77f484501dba44feeea88f5c1f13b6b4660463640
distributionSha256Sum=7c3ad722e9b0ce8205b91560fd6ce8296ac3eadf065672242fd73c06b8eeb6ee
Original file line number Diff line number Diff line change
Expand Up @@ -482,100 +482,3 @@
}]
- match: { error.root_cause.0.type: "illegal_argument_exception" }
- match: { error.root_cause.0.reason: "script score function must not produce negative scores, but got: [-9.0]"}

---

"Flat-object fields from within the scripting":
- skip:
version: " - 2.6.99"
reason: "flat_object is introduced in 2.7.0"

- do:
indices.create:
index: test
body:
mappings:
properties:
flat:
type : "flat_object"

# This document has 6 distinct parts in its flat_object field paths:
# - flat.field_1
# - flat.field_2
# - flat.field_3
# - flat.inner
# - flat.field_A
# - flat.field_B
- do:
index:
index: test
id: 1
body: {
"flat": {
"field_1": "John Doe",
"field_2": 33,
"field_3": false,
"inner": {
"field_A": ["foo", "bar"],
"field_B": false
}
}
}

- do:
index:
index: test
id: 2
body: {
"flat": {
"field_1": "Joe Public",
"field_2": 45
}
}

- do:
indices.refresh:
index: test

# It is possible to filter based on the number of distinct parts of flat_object field paths
- do:
search:
body: {
_source: true,
query: {
bool: {
filter: {
script: {
script: {
source: "doc['flat'].size() == 6",
lang: "painless"
}
}
}
}
}
}

- length: { hits.hits: 1 }
- match: { hits.hits.0._source.flat.field_1: "John Doe" }

- do:
search:
body: {
_source: true,
query: {
bool: {
filter: {
script: {
script: {
source: "doc['flat'].size() < 6",
lang: "painless"
}
}
}
}
}
}

- length: { hits.hits: 1 }
- match: { hits.hits.0._source.flat.field_1: "Joe Public" }

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ad575652d84153075dd41ec6177ccb15251262b2
2 changes: 1 addition & 1 deletion plugins/repository-azure/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ dependencies {
api "io.netty:netty-resolver-dns:${versions.netty}"
api "io.netty:netty-transport-native-unix-common:${versions.netty}"
implementation project(':modules:transport-netty4')
api 'com.azure:azure-storage-blob:12.22.2'
api 'com.azure:azure-storage-blob:12.22.3'
api 'org.reactivestreams:reactive-streams:1.0.4'
api 'io.projectreactor:reactor-core:3.5.6'
api 'io.projectreactor.netty:reactor-netty:1.1.8'
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0df12462c2eac3beaf25d283f707a0560853228b

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ad575652d84153075dd41ec6177ccb15251262b2
2 changes: 1 addition & 1 deletion plugins/repository-gcs/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ dependencies {
api "com.google.auth:google-auth-library-oauth2-http:${versions.google_auth}"

api 'com.google.cloud:google-cloud-core:2.5.10'
api 'com.google.cloud:google-cloud-core-http:2.17.0'
api 'com.google.cloud:google-cloud-core-http:2.21.0'
api 'com.google.cloud:google-cloud-storage:1.113.1'

api 'com.google.code.gson:gson:2.9.0'
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
07da4710ccdbcfee253672c0b9e00e7370626c26
1 change: 0 additions & 1 deletion plugins/repository-gcs/licenses/guava-32.0.1-jre.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions plugins/repository-gcs/licenses/guava-32.1.1-jre.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ad575652d84153075dd41ec6177ccb15251262b2
1 change: 0 additions & 1 deletion plugins/repository-hdfs/licenses/guava-32.0.1-jre.jar.sha1

This file was deleted.

1 change: 1 addition & 0 deletions plugins/repository-hdfs/licenses/guava-32.1.1-jre.jar.sha1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ad575652d84153075dd41ec6177ccb15251262b2

0 comments on commit c4519e0

Please sign in to comment.