Skip to content

Commit

Permalink
Merge branch 'opensearch-project:main' into stats-rollup
Browse files Browse the repository at this point in the history
  • Loading branch information
shourya035 committed Aug 9, 2023
2 parents 33b4f86 + 6eb87b5 commit 6ce0cc2
Show file tree
Hide file tree
Showing 26 changed files with 777 additions and 462 deletions.
9 changes: 7 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,13 +85,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Add configuration for file cache size to max remote data ratio to prevent oversubscription of file cache ([#8606](https://github.com/opensearch-project/OpenSearch/pull/8606))
- Disallow compression level to be set for default and best_compression index codecs ([#8737]()https://github.com/opensearch-project/OpenSearch/pull/8737)
- Prioritize replica shard movement during shard relocation ([#8875](https://github.com/opensearch-project/OpenSearch/pull/8875))
- Introducing Default and Best Compression codecs as their algorithm name ([#9123]()https://github.com/opensearch-project/OpenSearch/pull/9123)

### Dependencies
- Bump `org.apache.logging.log4j:log4j-core` from 2.17.1 to 2.20.0 ([#8307](https://github.com/opensearch-project/OpenSearch/pull/8307))
- Bump `io.grpc:grpc-context` from 1.46.0 to 1.56.1 ([#8726](https://github.com/opensearch-project/OpenSearch/pull/8726))
- Bump `com.netflix.nebula:gradle-info-plugin` from 12.1.5 to 12.1.6 ([#8724](https://github.com/opensearch-project/OpenSearch/pull/8724))
- Bump `commons-codec:commons-codec` from 1.15 to 1.16.0 ([#8725](https://github.com/opensearch-project/OpenSearch/pull/8725))
- Bump `org.apache.zookeeper:zookeeper` from 3.8.1 to 3.8.2 ([#8844](https://github.com/opensearch-project/OpenSearch/pull/8844))
- Bump `org.apache.zookeeper:zookeeper` from 3.8.1 to 3.9.0 ([#8844](https://github.com/opensearch-project/OpenSearch/pull/8844), [#9146](https://github.com/opensearch-project/OpenSearch/pull/9146))
- Bump `org.gradle.test-retry` from 1.5.3 to 1.5.4 ([#8842](https://github.com/opensearch-project/OpenSearch/pull/8842))
- Bump `com.netflix.nebula.ospackage-base` from 11.3.0 to 11.4.0 ([#8838](https://github.com/opensearch-project/OpenSearch/pull/8838))
- Bump `com.google.http-client:google-http-client-gson` from 1.43.2 to 1.43.3 ([#8840](https://github.com/opensearch-project/OpenSearch/pull/8840))
Expand All @@ -103,17 +104,21 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Bump `com.maxmind.geoip2:geoip2` from 4.0.1 to 4.1.0 ([#8998](https://github.com/opensearch-project/OpenSearch/pull/8998))
- Bump `org.apache.commons:commons-lang3` from 3.12.0 to 3.13.0 in /plugins/repository-hdfs ([#8997](https://github.com/opensearch-project/OpenSearch/pull/8997))
- Bump `netty` from 4.1.94.Final to 4.1.96.Final ([#9030](https://github.com/opensearch-project/OpenSearch/pull/9030))
- Bump `io.projectreactor.netty:reactor-netty-http` from 1.1.8 to 1.1.9 ([#9147](https://github.com/opensearch-project/OpenSearch/pull/9147))
- Bump `org.apache.maven:maven-model` from 3.9.3 to 3.9.4 ([#9148](https://github.com/opensearch-project/OpenSearch/pull/9148))

### Changed
- Perform aggregation postCollection in ContextIndexSearcher after searching leaves ([#8303](https://github.com/opensearch-project/OpenSearch/pull/8303))
- Make Span exporter configurable ([#8620](https://github.com/opensearch-project/OpenSearch/issues/8620))
- Change InternalSignificantTerms to sum shard-level superset counts only in final reduce ([#8735](https://github.com/opensearch-project/OpenSearch/pull/8735))
- Exclude 'benchmarks' from codecov report ([#8805](https://github.com/opensearch-project/OpenSearch/pull/8805))
- Adds support for tracing runnable scenarios ([#8831](https://github.com/opensearch-project/OpenSearch/pull/8831))
- [Refactor] MediaTypeParser to MediaTypeParserRegistry ([#8636](https://github.com/opensearch-project/OpenSearch/pull/8636))
- Create separate SourceLookup instance per segment slice in SignificantTextAggregatorFactory ([#8807](https://github.com/opensearch-project/OpenSearch/pull/8807))
- Add support for aggregation profiler with concurrent aggregation ([#8801](https://github.com/opensearch-project/OpenSearch/pull/8801))
- [Remove] Deprecated Fractional ByteSizeValue support #9005 ([#9005](https://github.com/opensearch-project/OpenSearch/pull/9005))
- Make MultiBucketConsumerService thread safe to use across slices during search ([#9047](https://github.com/opensearch-project/OpenSearch/pull/9047))

### Deprecated

### Removed
Expand All @@ -124,4 +129,4 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
### Security

[Unreleased 3.0]: https://github.com/opensearch-project/OpenSearch/compare/2.x...HEAD
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.10...2.x
[Unreleased 2.x]: https://github.com/opensearch-project/OpenSearch/compare/2.10...2.x
6 changes: 3 additions & 3 deletions MAINTAINERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ This document contains a list of maintainers in this repo. See [opensearch-proje

| Maintainer | GitHub ID | Affiliation |
|--------------------------| ------------------------------------------------------- | ----------- |
| Abbas Hussain | [abbashus](https://github.com/abbashus) | Meta |
| Anas Alkouz | [anasalkouz](https://github.com/anasalkouz) | Amazon |
| Andrew Ross | [andrross](https://github.com/andrross) | Amazon |
| Andriy Redko | [reta](https://github.com/reta) | Aiven |
Expand All @@ -20,6 +21,7 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
| Marc Handalian | [mch2](https://github.com/mch2) | Amazon |
| Nick Knize | [nknize](https://github.com/nknize) | Amazon |
| Owais Kazi | [owaiskazi19](https://github.com/owaiskazi19) | Amazon |
| Rabi Panda | [adnapibar](https://github.com/adnapibar) | Independent |
| Rishikesh Pasham | [Rishikesh1159](https://github.com/Rishikesh1159) | Amazon |
| Ryan Bogan | [ryanbogan](https://github.com/ryanbogan) | Amazon |
| Sachin Kale | [sachinpkale](https://github.com/sachinpkale) | Amazon |
Expand All @@ -33,8 +35,6 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
## Emeritus

| Maintainer | GitHub ID | Affiliation |
|-------------------------|---------------------------------------------| ----------- |
| Abbas Hussain | [abbashus](https://github.com/abbashus) | Amazon |
|-------------------------|---------------------------------------------|-------------|
| Megha Sai Kavikondala | [meghasaik](https://github.com/meghasaik) | Amazon |
| Rabi Panda | [adnapibar](https://github.com/adnapibar) | Amazon |
| Xue Zhou | [xuezhou25](https://github.com/xuezhou25) | Amazon |
2 changes: 1 addition & 1 deletion buildSrc/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ dependencies {
api 'de.thetaphi:forbiddenapis:3.5.1'
api 'com.avast.gradle:gradle-docker-compose-plugin:0.16.12'
api "org.yaml:snakeyaml:${props.getProperty('snakeyaml')}"
api 'org.apache.maven:maven-model:3.9.3'
api 'org.apache.maven:maven-model:3.9.4'
api 'com.networknt:json-schema-validator:1.0.86'
api 'org.jruby.jcodings:jcodings:1.0.58'
api 'org.jruby.joni:joni:2.2.1'
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,17 @@ public DefaultTracer(TracingTelemetry tracingTelemetry, TracerContextStorage<Str

@Override
public SpanScope startSpan(String spanName) {
Span span = createSpan(spanName, getCurrentSpan());
return startSpan(spanName, null);
}

@Override
public SpanScope startSpan(String spanName, SpanContext parentSpan) {
Span span = null;
if (parentSpan != null) {
span = createSpan(spanName, parentSpan.getSpan());
} else {
span = createSpan(spanName, getCurrentSpanInternal());
}
setCurrentSpanInContext(span);
addDefaultAttributes(span);
return new DefaultSpanScope(span, (scopeSpan) -> endSpan(scopeSpan));
Expand All @@ -48,11 +58,15 @@ public void close() throws IOException {
((Closeable) tracingTelemetry).close();
}

// Visible for testing
Span getCurrentSpan() {
private Span getCurrentSpanInternal() {
return tracerContextStorage.get(TracerContextStorage.CURRENT_SPAN);
}

public SpanContext getCurrentSpan() {
final Span currentSpan = tracerContextStorage.get(TracerContextStorage.CURRENT_SPAN);
return (currentSpan == null) ? null : new SpanContext(currentSpan);
}

private void endSpan(Span span) {
if (span != null) {
span.endSpan();
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.telemetry.tracing;

/**
* Wrapped Span will be exposed to the code outside of tracing package for sharing the {@link Span} without having access to
* its properties.
*/
public final class SpanContext {
private final Span span;

/**
* Constructor.
* @param span span to be wrapped.
*/
public SpanContext(Span span) {
this.span = span;
}

Span getSpan() {
return span;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,17 @@ public interface Tracer extends Closeable {
*/
SpanScope startSpan(String spanName);

/**
* Started the {@link Span} with the given name and parent.
* @param spanName span name.
* @param parentSpan parent span.
* @return scope of the span, must be closed with explicit close or with try-with-resource
*/
SpanScope startSpan(String spanName, SpanContext parentSpan);

/**
* Returns the current span.
* @return current wrapped span.
*/
SpanContext getCurrentSpan();
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

import org.opensearch.telemetry.tracing.SpanScope;
import org.opensearch.telemetry.tracing.Tracer;
import org.opensearch.telemetry.tracing.SpanContext;

/**
* No-op implementation of Tracer
Expand All @@ -30,6 +31,16 @@ public SpanScope startSpan(String spanName) {
return SpanScope.NO_OP;
}

@Override
public SpanContext getCurrentSpan() {
return null;
}

@Override
public SpanScope startSpan(String spanName, SpanContext parentSpan) {
return SpanScope.NO_OP;
}

@Override
public void close() {

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.telemetry.tracing.runnable;

import org.opensearch.telemetry.tracing.SpanContext;
import org.opensearch.telemetry.tracing.SpanScope;
import org.opensearch.telemetry.tracing.Tracer;

/**
* Wraps the runnable and add instrumentation to trace the {@link Runnable}
*/
public class TraceableRunnable implements Runnable {
private final Runnable runnable;
private final SpanContext parent;
private final Tracer tracer;
private final String spanName;

/**
* Constructor.
* @param tracer tracer
* @param spanName spanName
* @param parent parent Span.
* @param runnable runnable.
*/
public TraceableRunnable(Tracer tracer, String spanName, SpanContext parent, Runnable runnable) {
this.tracer = tracer;
this.spanName = spanName;
this.parent = parent;
this.runnable = runnable;
}

@Override
public void run() {
try (SpanScope spanScope = tracer.startSpan(spanName, parent)) {
runnable.run();
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

/**
* Contains tracing related classes
*/
package org.opensearch.telemetry.tracing.runnable;
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,12 @@
package org.opensearch.telemetry.tracing;

import org.junit.Assert;
import org.opensearch.common.settings.Settings;
import org.opensearch.common.util.concurrent.ThreadContext;
import org.opensearch.test.OpenSearchTestCase;

import java.io.IOException;
import org.opensearch.test.telemetry.tracing.MockTracingTelemetry;

import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
Expand Down Expand Up @@ -40,7 +43,37 @@ public void testCreateSpan() {

defaultTracer.startSpan("span_name");

Assert.assertEquals("span_name", defaultTracer.getCurrentSpan().getSpanName());
Assert.assertEquals("span_name", defaultTracer.getCurrentSpan().getSpan().getSpanName());
}

public void testCreateSpanWithParent() {
TracingTelemetry tracingTelemetry = new MockTracingTelemetry();
DefaultTracer defaultTracer = new DefaultTracer(
tracingTelemetry,
new ThreadContextBasedTracerContextStorage(new ThreadContext(Settings.EMPTY), tracingTelemetry)
);

defaultTracer.startSpan("span_name", null);

SpanContext parentSpan = defaultTracer.getCurrentSpan();

defaultTracer.startSpan("span_name_1", parentSpan);

Assert.assertEquals("span_name_1", defaultTracer.getCurrentSpan().getSpan().getSpanName());
Assert.assertEquals(parentSpan.getSpan(), defaultTracer.getCurrentSpan().getSpan().getParentSpan());
}

public void testCreateSpanWithNullParent() {
TracingTelemetry tracingTelemetry = new MockTracingTelemetry();
DefaultTracer defaultTracer = new DefaultTracer(
tracingTelemetry,
new ThreadContextBasedTracerContextStorage(new ThreadContext(Settings.EMPTY), tracingTelemetry)
);

defaultTracer.startSpan("span_name", null);

Assert.assertEquals("span_name", defaultTracer.getCurrentSpan().getSpan().getSpanName());
Assert.assertEquals(null, defaultTracer.getCurrentSpan().getSpan().getParentSpan());
}

public void testEndSpanByClosingScope() {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

package org.opensearch.telemetry.tracing;

import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import org.opensearch.common.settings.Settings;
import org.opensearch.common.util.concurrent.ThreadContext;
import org.opensearch.telemetry.tracing.runnable.TraceableRunnable;
import org.opensearch.test.OpenSearchTestCase;
import org.opensearch.test.telemetry.tracing.MockTracingTelemetry;

public class TraceableRunnableTests extends OpenSearchTestCase {

private final ThreadContextBasedTracerContextStorage contextStorage = new ThreadContextBasedTracerContextStorage(
new ThreadContext(Settings.EMPTY),
new MockTracingTelemetry()
);

public void testRunnableWithNullParent() throws Exception {
String spanName = "testRunnable";
DefaultTracer defaultTracer = new DefaultTracer(new MockTracingTelemetry(), contextStorage);
final AtomicBoolean isRunnableCompleted = new AtomicBoolean(false);
TraceableRunnable traceableRunnable = new TraceableRunnable(
defaultTracer,
spanName,
null,
() -> { isRunnableCompleted.set(true); }
);
traceableRunnable.run();
assertTrue(isRunnableCompleted.get());
assertEquals(spanName, defaultTracer.getCurrentSpan().getSpan().getSpanName());
assertEquals(null, defaultTracer.getCurrentSpan().getSpan().getParentSpan());
}

public void testRunnableWithParent() throws Exception {
String spanName = "testRunnable";
String parentSpanName = "parentSpan";
DefaultTracer defaultTracer = new DefaultTracer(new MockTracingTelemetry(), contextStorage);
defaultTracer.startSpan(parentSpanName);
SpanContext parentSpan = defaultTracer.getCurrentSpan();
AtomicReference<SpanContext> currrntSpan = new AtomicReference<>(new SpanContext(null));
final AtomicBoolean isRunnableCompleted = new AtomicBoolean(false);
TraceableRunnable traceableRunnable = new TraceableRunnable(defaultTracer, spanName, parentSpan, () -> {
isRunnableCompleted.set(true);
currrntSpan.set(defaultTracer.getCurrentSpan());
});
traceableRunnable.run();
assertTrue(isRunnableCompleted.get());
assertEquals(spanName, currrntSpan.get().getSpan().getSpanName());
assertEquals(parentSpan.getSpan(), currrntSpan.get().getSpan().getParentSpan());
assertEquals(parentSpan.getSpan(), defaultTracer.getCurrentSpan().getSpan());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,15 @@ public void testReindexingMultipleCodecs() throws InterruptedException, Executio
Map<String, String> codecMap = Map.of(
"best_compression",
"BEST_COMPRESSION",
"zlib",
"BEST_COMPRESSION",
"zstd_no_dict",
"ZSTD_NO_DICT",
"zstd",
"ZSTD",
"default",
"BEST_SPEED",
"lz4",
"BEST_SPEED"
);

Expand Down
2 changes: 1 addition & 1 deletion plugins/repository-azure/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ dependencies {
api 'io.projectreactor:reactor-core:3.5.6'
api 'io.projectreactor.netty:reactor-netty:1.1.8'
api 'io.projectreactor.netty:reactor-netty-core:1.1.8'
api 'io.projectreactor.netty:reactor-netty-http:1.1.8'
api 'io.projectreactor.netty:reactor-netty-http:1.1.9'
api "org.slf4j:slf4j-api:${versions.slf4j}"
api "com.fasterxml.jackson.core:jackson-annotations:${versions.jackson}"
api "com.fasterxml.jackson.core:jackson-databind:${versions.jackson_databind}"
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
408b3037133f2e8ab0f195ccd3f807026be9b860

0 comments on commit 6ce0cc2

Please sign in to comment.