The OpenTelemetry Protocol with Apache Arrow project is an effort within OpenTelemetry to use Apache Arrow libraries for bulk data transport in OpenTelemetry collection pipelines. This repository is the home of the OpenTelemetry Protocol with Apache Arrow protocol and reference implementation.
Instructions for building an OpenTelemetry Collector with the modules in this repository are provided in BUILDING.md.
Examples for running the OpenTelemetry Collector with the modules in this repository are documented in collector/examples.
OpenTelemetry and Apache Arrow have similar charters, so it was natural to think about combining them. Both projects offer vendor-neutral interfaces with a cross-language interface specification, so that their implementation will feel familiar to users as they move between programming languages, and both specify a data model that is used throughout the project.
The OpenTelemetry project defines OTLP, the "OpenTeLemetry Prototcol" as the standard form of telemetry data in OpenTelemetry, being as similar as possible to the data model underlying the project. OTLP is defined in terms of Google protocol buffer definitions.
OTLP is a stateless protocol, where export requests map directly into the data model, nothing is omitted, and little is shared. OTLP export requests to not contain external or internal references, making the data relatively simple and easy to interpret. Because of this design, users of OTLP will typically configure network compression. In environments where telemetry data will be shipped to a service provider across a wide-area network, users would like more compression than can be achieved using a row-based data model and a stateless protocol.
This is organized in phases. Our initial aim is to facilitate traffic reduction between a pair of OpenTelemetry collectors as illustrated in the following diagram.
The collector provided in this repository implements a new Arrow Receiver and Exporter able to fallback on standard OTLP when needed. The following diagram is an overview of this integration. In this first phase, the internal representation of the telemetry data is still fundamentally row-oriented.
Ultimately, we believe that an end-to-end OpenTelemetry Protocol with Apache Arrow pipeline will enable telemetry pipelines with substantially lower overhead to be built. These are our future milestones for OpenTelemetry and Apache Arrow integration:
- Extend OpenTelemetry client SDKs to natively support the OpenTelemetry
Protocol with Apache Arrow Protocol - Extend the OpenTelemetry collector with direct support for OpenTelemetry Protocol with Apache Arrow pipelines
- Extend OpenTelemetry data model with native support for multi-variate metrics.
- Output OpenTelemetry data to the Parquet file format, part of the Apache Arrow ecosystem
The first general-purpose application for the project is traffic reduction. At a high-level, this protocol performs the following steps to compactly encode and transmit telemetry using Apache Arrow.
- Separate the OpenTelemetry Resource and Scope elements from the hierarchy, then encode and transmit each distinct entity once per stream lifetime.
- Calculate distinct attribute sets used by Resources, Scopes, Metrics, Logs, Spans, Span Events, and Span Links, then encode and transmit each distinct entity once per stream lifetime.
- Use Apache Arrow's built-in support for encoding dictionaries and leverage other purpose-built low-level facilities, such as delta-dictionaries and sorting, to encode structures compactly.
Here is a diagram showing how the protocol transforms OTLP Log Records into column-oriented data, which also makes the data more compressible.
The first phase of the project has entered the Beta stability level, as defined by the OpenTelemetry collector guidelines. We do not plan to make breaking changes in this protocol without first engineering an approach that ensures forwards and backwards-compatibility for existing and new users. We believe it is safe to begin using these components for production data, non-critical workloads.
We are pleased to release two new collector components, presently housed in this this repository.
We are working with the maintainers of the OpenTelemetry Collector-Contrib to merge these components into that repository. See our tracking issue.
The OpenTelemetry Protocol with Apache Arrow exporter and receiver components are drop-in compatible
with the core collector's OTLP exporter and receiver components.
Users with an established OTLP collection pipeline between two
OpenTelemetry Collectors can re-build their collectors with
otelarrow components, then simply replace the component name otlp
with otelarrow. The exporter and receiver both support falling back
to standard OTLP in case either side does not recognize the protocol,
so the upgrade should be painless. The OpenTelemetry Protocol with Apache Arrow receiver serves
both OpenTelemetry Protocol with Apache Arrow and OTLP on the standard port for OTLP gRPC (4317).
See the Exporter and Receiver documentation for details and sample configurations.
This package is a reference implementation of the OpenTelemetry Protocol with Apache Arrow protocol specified in this OTEP, which is currently the best source of information about OpenTelemetry Protocol with Apache Arrow. The Donation request describes how the project began.
Here are several more resources that are available to learn more about OpenTelemetry Protocol with Apache Arrow.
- Arrow Data Model - Mapping OTLP entities to Arrow Schemas.
- Benchmark results - Based on synthetic and production data.
- Validation process - Encoding/Decoding validation process.
- Articles describing some of the Arrow techniques used behind the scenes to optimize compression ratio and memory usage:
The following chart shows the compressed message size (in bytes) as a function of the batch size for metrics (univariate), logs, and traces. The bottom of the chart shows the reduction factor for both the standard OTLP protocol (with ZSTD compression) and the OpenTelemetry Protocol with Apache Arrow protocol (ZSTD) in comparison with an uncompressed OTLP protocol.
The next chart follows the same logic but shows the results for multivariate metrics (see left column).
The following heatmap represents, for different combinations of batch sizes and connection durations (expressed as the number of batches per stream), the additional percentage of compression gain between this new protocol and OTLP, both compressed with ZSTD. The data used here comes from a traffic of spans captured in a production environment. The gains are substantial in most cases. It is even interesting to note that these gains compared to OTLP+ZSTD are more significant for moderate-sized batches (e.g., 100 and 1000 spans per batch), which makes this protocol also interesting for scenarios where the additional latency introduced by batching must be minimized. There is hardly any scenario where micro-batches (e.g., 10 spans per batch) make the overhead of the Arrow schema prohibitive, and the advantage of a columnar representation becomes negligible. In other cases, this initial overhead is very quickly offset after just the first few batches. The columnar organization also lends itself better to compression. For very large batch sizes, ZSTD does an excellent job as long as the compression window is sufficiently large, but even in this case, the new protocol remains superior. As previously mentioned, these compression gains can be higher for traffic predominantly containing multivariate metrics.
For more details, see the following benchmark results page.
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. For more information, please read CONTRIBUTING.md.
OpenTelemetry Protocol with Apache Arrow Protocol Adapter is licensed under Apache 2.0.