[DOC] Add new documentation for distributed tracing #4964
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vagimeli
requested review from
cwillum,
hdhalter,
kolchfa-aws,
Naarcha-AWS,
ananzh,
seanneumann,
AMoo-Miki and
natebower
as code owners
This was referenced Sep 5, 2023
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
|
@Gaganjuneja @reta The draft documentation is ready for technical review. Please make your suggestions/updates by end of week. Thank you :) |
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
hdhalter
added
3 - Tech review
reta
reviewed
Sep 8, 2023
reta
reviewed
Sep 8, 2023
reta
reviewed
Sep 8, 2023
reta
reviewed
Sep 8, 2023
reta
reviewed
Sep 8, 2023
| 2. **Exporters:** Exporters are responsible for persisting the data. OpenTelemetry provides several out-of-the-box exporters, and OpenSearch currently supports the following: | ||
| - `LoggingSpanExporter`: Exports spans to a log file, generating a separate file in the logs directory `_otel_traces.log`. Default is `telemetry.otel.tracer.span.exporter.class=io.opentelemetry.exporter.logging.LoggingSpanExporter`. | ||
| - `OtlpGrpcSpanExporter`: Exports spans by using [gRPC](https://grpc.io/). To use this exporter, you need to install the `otel-collector` on the node and specify the endpoint using the setting `telemetry.otel.tracer.exporter.endpoint`. By default, it writes to the endpoint `http://localhost:4317/`. If you want to configure it for HTTPS, follow the guidance in the [OpenTelemetry Configuration](https://opentelemetry.io/docs/collector/configuration/) documentation. The setting is as follows: | ||
| - `telemetry.otel.tracer.span.exporter.class=org.opensearch.telemetry.tracing.exporter.OtlpGrpcSpanExporterProvider` |
There was a problem hiding this comment.
Also not there yet, the pull request is not ready
There was a problem hiding this comment.
Yes, Only support for LoggingSpanExporter is there as of 2.10.
There was a problem hiding this comment.
We can update these parameters when they're available.
reta
reviewed
Sep 8, 2023
Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
vagimeli
added
5 - Editorial review
natebower
reviewed
Oct 12, 2023
|
|
||
| ## Get started | ||
|
|
||
| The distributed tracing feature is experimental as of OpenSearch 2.10. To begin using the distributed tracing feature, you need to first enable it using `opensearch.experimental.feature.telemetry.enabled` feature flag, and subsequently activate the tracer, using the dynamic setting `telemetry.tracer.enabled`. It's important to exercise caution when enabling this feature, as it can consume system resources. Detailed information on enabling and configuring distributed tracing, including on-demand troubleshooting and request sampling, is described in the following sections. |
There was a problem hiding this comment.
Second sentence: "...and subsequently activate the tracer by using..."?
|
|
||
| ### Sampling | ||
|
|
||
| Distributed tracing can generate numerous spans, consuming system resources unnecessarily. To reduce the number of traces, also called samples, you can configure different sampling thresholds. By default, sampling is configured to include only 1% of all HTTP requests. Sampling has the following types: |
There was a problem hiding this comment.
"Sampling includes" instead of "Sampling has"?
| 1. **Head sampling:** Sampling decisions are made before initiating the root span of a request. OpenSearch supports two head sampling methods: | ||
| - **Probabilistic:** A blanket limit on incoming requests, dynamically adjustable with the `telemetry.tracer.sampler.probability` setting. This setting ranges between 0 and 1. Default is 0.01, which indicates that 1% of incoming requests are sampled. | ||
| - **On-Demand:** For debugging specific requests, you can send the `trace=true` attribute as part of the HTTP headers, causing those requests to be sampled regardless of the probabilistic sampling setting. | ||
| 2. **Tail sampling:** To configure tail sampling, follow the [OpenTelemetry tail sampling documentation](https://opentelemetry.io/docs/concepts/sampling/#tail-sampling). Configuration depends on the type of collector you choose. Updates on ongoing work for OpenSearch are in the [RFC](https://github.com/opensearch-project/OpenSearch/issues/8918) on GitHub. |
There was a problem hiding this comment.
"follow the instructions in"? Can we delete the last sentence as unnecessary?
There was a problem hiding this comment.
Revised. Yes, deleted.
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
hdhalter
added
release-notes
vagimeli
added a commit
that referenced
this pull request
Oct 13, 2023
* Write new documentation about distributed tracing Signed-off-by: Melissa Vagi <vagimeli@amazon.com> --------- Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Andriy Redko <drreta@gmail.com> Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>
harshavamsi
pushed a commit
to harshavamsi/documentation-website
that referenced
this pull request
Oct 31, 2023
…ct#4964) * Write new documentation about distributed tracing Signed-off-by: Melissa Vagi <vagimeli@amazon.com> --------- Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Andriy Redko <drreta@gmail.com> Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
vagimeli
added a commit
that referenced
this pull request
Dec 21, 2023
* Write new documentation about distributed tracing Signed-off-by: Melissa Vagi <vagimeli@amazon.com> --------- Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Andriy Redko <drreta@gmail.com> Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add new documentation for distributed tracing
Issues Resolved
opensearch-project/OpenSearch#6750
Fixes #3951
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.