Skip to content

Commit

Permalink
Merge branch 'main' into search-template-fix
Browse files Browse the repository at this point in the history
  • Loading branch information
cwperks committed Aug 8, 2023
2 parents 32194c3 + 6eb87b5 commit a6d5560
Show file tree
Hide file tree
Showing 44 changed files with 1,400 additions and 519 deletions.
22 changes: 16 additions & 6 deletions .github/workflows/precommit.yml
Original file line number Diff line number Diff line change
@@ -1,19 +1,29 @@
name: Gradle Precommit
name: Gradle Precommit and Asssemble
on: [pull_request]

jobs:
precommit:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [windows-latest, macos-latest] # precommit on ubuntu-latest is run as a part of the gradle-check workflow
os: [ubuntu-latest, windows-latest, macos-latest]
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Set up JDK 11
uses: actions/setup-java@v2
uses: actions/setup-java@v3
with:
java-version: 11
distribution: adopt
- name: Run Gradle
distribution: temurin
cache: gradle
- name: Run Gradle (precommit)
run: |
./gradlew javadoc precommit --parallel
- name: Setup docker (missing on MacOS)
if: runner.os == 'macos'
run: |
brew install docker
colima start
sudo ln -sf $HOME/.colima/default/docker.sock /var/run/docker.sock
- name: Run Gradle (assemble)
run: |
./gradlew assemble --parallel
9 changes: 8 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Add getter for path field in NestedQueryBuilder ([#4636](https://github.com/opensearch-project/OpenSearch/pull/4636))
- Allow mmap to use new JDK-19 preview APIs in Apache Lucene 9.4+ ([#5151](https://github.com/opensearch-project/OpenSearch/pull/5151))
- Add events correlation engine plugin ([#6854](https://github.com/opensearch-project/OpenSearch/issues/6854))
- Introduce new dynamic cluster setting to control slice computation for concurrent segment search ([#9107](https://github.com/opensearch-project/OpenSearch/pull/9107))

### Dependencies
- Bump `log4j-core` from 2.18.0 to 2.19.0
Expand Down Expand Up @@ -49,6 +50,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- [Remote Store] Add support to restore only unassigned shards of an index ([#8792](https://github.com/opensearch-project/OpenSearch/pull/8792))
- Replace the deprecated IndexReader APIs with new storedFields() & termVectors() ([#7792](https://github.com/opensearch-project/OpenSearch/pull/7792))
- [Remote Store] Restrict user override for remote store index level settings ([#8812](https://github.com/opensearch-project/OpenSearch/pull/8812))
- Removed blocking wait in TransportGetSnapshotsAction which was exhausting generic threadpool ([#8377](https://github.com/opensearch-project/OpenSearch/pull/8377))

### Deprecated

Expand Down Expand Up @@ -83,14 +85,15 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Add configuration for file cache size to max remote data ratio to prevent oversubscription of file cache ([#8606](https://github.com/opensearch-project/OpenSearch/pull/8606))
- Disallow compression level to be set for default and best_compression index codecs ([#8737]()https://github.com/opensearch-project/OpenSearch/pull/8737)
- Prioritize replica shard movement during shard relocation ([#8875](https://github.com/opensearch-project/OpenSearch/pull/8875))
- Introducing Default and Best Compression codecs as their algorithm name ([#9123]()https://github.com/opensearch-project/OpenSearch/pull/9123)
- Make SearchTemplateRequest implement IndicesRequest.Replaceable ([#9122]()https://github.com/opensearch-project/OpenSearch/pull/9122)

### Dependencies
- Bump `org.apache.logging.log4j:log4j-core` from 2.17.1 to 2.20.0 ([#8307](https://github.com/opensearch-project/OpenSearch/pull/8307))
- Bump `io.grpc:grpc-context` from 1.46.0 to 1.56.1 ([#8726](https://github.com/opensearch-project/OpenSearch/pull/8726))
- Bump `com.netflix.nebula:gradle-info-plugin` from 12.1.5 to 12.1.6 ([#8724](https://github.com/opensearch-project/OpenSearch/pull/8724))
- Bump `commons-codec:commons-codec` from 1.15 to 1.16.0 ([#8725](https://github.com/opensearch-project/OpenSearch/pull/8725))
- Bump `org.apache.zookeeper:zookeeper` from 3.8.1 to 3.8.2 ([#8844](https://github.com/opensearch-project/OpenSearch/pull/8844))
- Bump `org.apache.zookeeper:zookeeper` from 3.8.1 to 3.9.0 ([#8844](https://github.com/opensearch-project/OpenSearch/pull/8844), [#9146](https://github.com/opensearch-project/OpenSearch/pull/9146))
- Bump `org.gradle.test-retry` from 1.5.3 to 1.5.4 ([#8842](https://github.com/opensearch-project/OpenSearch/pull/8842))
- Bump `com.netflix.nebula.ospackage-base` from 11.3.0 to 11.4.0 ([#8838](https://github.com/opensearch-project/OpenSearch/pull/8838))
- Bump `com.google.http-client:google-http-client-gson` from 1.43.2 to 1.43.3 ([#8840](https://github.com/opensearch-project/OpenSearch/pull/8840))
Expand All @@ -102,17 +105,21 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Bump `com.maxmind.geoip2:geoip2` from 4.0.1 to 4.1.0 ([#8998](https://github.com/opensearch-project/OpenSearch/pull/8998))
- Bump `org.apache.commons:commons-lang3` from 3.12.0 to 3.13.0 in /plugins/repository-hdfs ([#8997](https://github.com/opensearch-project/OpenSearch/pull/8997))
- Bump `netty` from 4.1.94.Final to 4.1.96.Final ([#9030](https://github.com/opensearch-project/OpenSearch/pull/9030))
- Bump `io.projectreactor.netty:reactor-netty-http` from 1.1.8 to 1.1.9 ([#9147](https://github.com/opensearch-project/OpenSearch/pull/9147))
- Bump `org.apache.maven:maven-model` from 3.9.3 to 3.9.4 ([#9148](https://github.com/opensearch-project/OpenSearch/pull/9148))

### Changed
- Perform aggregation postCollection in ContextIndexSearcher after searching leaves ([#8303](https://github.com/opensearch-project/OpenSearch/pull/8303))
- Make Span exporter configurable ([#8620](https://github.com/opensearch-project/OpenSearch/issues/8620))
- Change InternalSignificantTerms to sum shard-level superset counts only in final reduce ([#8735](https://github.com/opensearch-project/OpenSearch/pull/8735))
- Exclude 'benchmarks' from codecov report ([#8805](https://github.com/opensearch-project/OpenSearch/pull/8805))
- Adds support for tracing runnable scenarios ([#8831](https://github.com/opensearch-project/OpenSearch/pull/8831))
- [Refactor] MediaTypeParser to MediaTypeParserRegistry ([#8636](https://github.com/opensearch-project/OpenSearch/pull/8636))
- Create separate SourceLookup instance per segment slice in SignificantTextAggregatorFactory ([#8807](https://github.com/opensearch-project/OpenSearch/pull/8807))
- Add support for aggregation profiler with concurrent aggregation ([#8801](https://github.com/opensearch-project/OpenSearch/pull/8801))
- [Remove] Deprecated Fractional ByteSizeValue support #9005 ([#9005](https://github.com/opensearch-project/OpenSearch/pull/9005))
- Make MultiBucketConsumerService thread safe to use across slices during search ([#9047](https://github.com/opensearch-project/OpenSearch/pull/9047))

### Deprecated

### Removed
Expand Down
6 changes: 3 additions & 3 deletions MAINTAINERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ This document contains a list of maintainers in this repo. See [opensearch-proje

| Maintainer | GitHub ID | Affiliation |
|--------------------------| ------------------------------------------------------- | ----------- |
| Abbas Hussain | [abbashus](https://github.com/abbashus) | Meta |
| Anas Alkouz | [anasalkouz](https://github.com/anasalkouz) | Amazon |
| Andrew Ross | [andrross](https://github.com/andrross) | Amazon |
| Andriy Redko | [reta](https://github.com/reta) | Aiven |
Expand All @@ -20,6 +21,7 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
| Marc Handalian | [mch2](https://github.com/mch2) | Amazon |
| Nick Knize | [nknize](https://github.com/nknize) | Amazon |
| Owais Kazi | [owaiskazi19](https://github.com/owaiskazi19) | Amazon |
| Rabi Panda | [adnapibar](https://github.com/adnapibar) | Independent |
| Rishikesh Pasham | [Rishikesh1159](https://github.com/Rishikesh1159) | Amazon |
| Ryan Bogan | [ryanbogan](https://github.com/ryanbogan) | Amazon |
| Sachin Kale | [sachinpkale](https://github.com/sachinpkale) | Amazon |
Expand All @@ -33,8 +35,6 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
## Emeritus

| Maintainer | GitHub ID | Affiliation |
|-------------------------|---------------------------------------------| ----------- |
| Abbas Hussain | [abbashus](https://github.com/abbashus) | Amazon |
|-------------------------|---------------------------------------------|-------------|
| Megha Sai Kavikondala | [meghasaik](https://github.com/meghasaik) | Amazon |
| Rabi Panda | [adnapibar](https://github.com/adnapibar) | Amazon |
| Xue Zhou | [xuezhou25](https://github.com/xuezhou25) | Amazon |
2 changes: 1 addition & 1 deletion buildSrc/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ dependencies {
api 'de.thetaphi:forbiddenapis:3.5.1'
api 'com.avast.gradle:gradle-docker-compose-plugin:0.16.12'
api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}"
api 'org.apache.maven:maven-model:3.9.3'
api 'org.apache.maven:maven-model:3.9.4'
api 'com.networknt:json-schema-validator:1.0.86'
api 'org.jruby.jcodings:jcodings:1.0.58'
api 'org.jruby.joni:joni:2.2.1'
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,9 @@ public abstract class DockerSupportService implements BuildService<DockerSupport
// Defines the possible locations of the Docker CLI. These will be searched in order.
private static String[] DOCKER_BINARIES_UNIX = { "/usr/bin/docker", "/usr/local/bin/docker" };

private static String[] DOCKER_BINARIES_WINDOWS = { System.getenv("PROGRAMFILES") + "\\Docker\\Docker\\resources\\bin\\docker.exe" };
private static String[] DOCKER_BINARIES_WINDOWS = {
System.getenv("PROGRAMFILES") + "\\Docker\\Docker\\resources\\bin\\docker.exe",
System.getenv("SystemRoot") + "\\System32\\docker.exe" /* Github Actions */ };

private static String[] DOCKER_BINARIES = Os.isFamily(Os.FAMILY_WINDOWS) ? DOCKER_BINARIES_WINDOWS : DOCKER_BINARIES_UNIX;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,17 @@ public DefaultTracer(TracingTelemetry tracingTelemetry, TracerContextStorage<Str

@Override
public SpanScope startSpan(String spanName) {
Span span = createSpan(spanName, getCurrentSpan());
return startSpan(spanName, null);
}

@Override
public SpanScope startSpan(String spanName, SpanContext parentSpan) {
Span span = null;
if (parentSpan != null) {
span = createSpan(spanName, parentSpan.getSpan());
} else {
span = createSpan(spanName, getCurrentSpanInternal());
}
setCurrentSpanInContext(span);
addDefaultAttributes(span);
return new DefaultSpanScope(span, (scopeSpan) -> endSpan(scopeSpan));
Expand All @@ -48,11 +58,15 @@ public void close() throws IOException {
((Closeable) tracingTelemetry).close();
}

// Visible for testing
Span getCurrentSpan() {
private Span getCurrentSpanInternal() {
return tracerContextStorage.get(TracerContextStorage.CURRENT_SPAN);
}

public SpanContext getCurrentSpan() {
final Span currentSpan = tracerContextStorage.get(TracerContextStorage.CURRENT_SPAN);
return (currentSpan == null) ? null : new SpanContext(currentSpan);
}

private void endSpan(Span span) {
if (span != null) {
span.endSpan();
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.telemetry.tracing;

/**
* Wrapped Span will be exposed to the code outside of tracing package for sharing the {@link Span} without having access to
* its properties.
*/
public final class SpanContext {
private final Span span;

/**
* Constructor.
* @param span span to be wrapped.
*/
public SpanContext(Span span) {
this.span = span;
}

Span getSpan() {
return span;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,17 @@ public interface Tracer extends Closeable {
*/
SpanScope startSpan(String spanName);

/**
* Started the {@link Span} with the given name and parent.
* @param spanName span name.
* @param parentSpan parent span.
* @return scope of the span, must be closed with explicit close or with try-with-resource
*/
SpanScope startSpan(String spanName, SpanContext parentSpan);

/**
* Returns the current span.
* @return current wrapped span.
*/
SpanContext getCurrentSpan();
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

import org.opensearch.telemetry.tracing.SpanScope;
import org.opensearch.telemetry.tracing.Tracer;
import org.opensearch.telemetry.tracing.SpanContext;

/**
* No-op implementation of Tracer
Expand All @@ -30,6 +31,16 @@ public SpanScope startSpan(String spanName) {
return SpanScope.NO_OP;
}

@Override
public SpanContext getCurrentSpan() {
return null;
}

@Override
public SpanScope startSpan(String spanName, SpanContext parentSpan) {
return SpanScope.NO_OP;
}

@Override
public void close() {

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.telemetry.tracing.runnable;

import org.opensearch.telemetry.tracing.SpanContext;
import org.opensearch.telemetry.tracing.SpanScope;
import org.opensearch.telemetry.tracing.Tracer;

/**
* Wraps the runnable and add instrumentation to trace the {@link Runnable}
*/
public class TraceableRunnable implements Runnable {
private final Runnable runnable;
private final SpanContext parent;
private final Tracer tracer;
private final String spanName;

/**
* Constructor.
* @param tracer tracer
* @param spanName spanName
* @param parent parent Span.
* @param runnable runnable.
*/
public TraceableRunnable(Tracer tracer, String spanName, SpanContext parent, Runnable runnable) {
this.tracer = tracer;
this.spanName = spanName;
this.parent = parent;
this.runnable = runnable;
}

@Override
public void run() {
try (SpanScope spanScope = tracer.startSpan(spanName, parent)) {
runnable.run();
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

/**
* Contains tracing related classes
*/
package org.opensearch.telemetry.tracing.runnable;
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,12 @@
package org.opensearch.telemetry.tracing;

import org.junit.Assert;
import org.opensearch.common.settings.Settings;
import org.opensearch.common.util.concurrent.ThreadContext;
import org.opensearch.test.OpenSearchTestCase;

import java.io.IOException;
import org.opensearch.test.telemetry.tracing.MockTracingTelemetry;

import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
Expand Down Expand Up @@ -40,7 +43,37 @@ public void testCreateSpan() {

defaultTracer.startSpan("span_name");

Assert.assertEquals("span_name", defaultTracer.getCurrentSpan().getSpanName());
Assert.assertEquals("span_name", defaultTracer.getCurrentSpan().getSpan().getSpanName());
}

public void testCreateSpanWithParent() {
TracingTelemetry tracingTelemetry = new MockTracingTelemetry();
DefaultTracer defaultTracer = new DefaultTracer(
tracingTelemetry,
new ThreadContextBasedTracerContextStorage(new ThreadContext(Settings.EMPTY), tracingTelemetry)
);

defaultTracer.startSpan("span_name", null);

SpanContext parentSpan = defaultTracer.getCurrentSpan();

defaultTracer.startSpan("span_name_1", parentSpan);

Assert.assertEquals("span_name_1", defaultTracer.getCurrentSpan().getSpan().getSpanName());
Assert.assertEquals(parentSpan.getSpan(), defaultTracer.getCurrentSpan().getSpan().getParentSpan());
}

public void testCreateSpanWithNullParent() {
TracingTelemetry tracingTelemetry = new MockTracingTelemetry();
DefaultTracer defaultTracer = new DefaultTracer(
tracingTelemetry,
new ThreadContextBasedTracerContextStorage(new ThreadContext(Settings.EMPTY), tracingTelemetry)
);

defaultTracer.startSpan("span_name", null);

Assert.assertEquals("span_name", defaultTracer.getCurrentSpan().getSpan().getSpanName());
Assert.assertEquals(null, defaultTracer.getCurrentSpan().getSpan().getParentSpan());
}

public void testEndSpanByClosingScope() {
Expand Down

0 comments on commit a6d5560

Please sign in to comment.