NIFI-12115 Add ListenOTLP to collect OpenTelemetry#7830
NIFI-12115 Add ListenOTLP to collect OpenTelemetry#7830exceptionfactory wants to merge 4 commits intoapache:mainfrom
Conversation
- Added ListenOTLP Processor supporting OpenTelemetry OTLP 1.0.0 Specification with gRPC and HTTP - Updated nifi-event-transport to support configurable SSLParameters for configurable Cipher Suites
...ntelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/ListenOTLP.java
Outdated
Show resolved
Hide resolved
...main/java/org/apache/nifi/processors/opentelemetry/encoding/ByteStringFieldDeserializer.java
Outdated
Show resolved
Hide resolved
...rc/main/java/org/apache/nifi/processors/opentelemetry/encoding/JsonServiceRequestReader.java
Outdated
Show resolved
Hide resolved
.../main/java/org/apache/nifi/processors/opentelemetry/encoding/TelemetryMessageSerializer.java
Outdated
Show resolved
Hide resolved
.../main/java/org/apache/nifi/processors/opentelemetry/encoding/TelemetryMessageSerializer.java
Outdated
Show resolved
Hide resolved
...rc/main/java/org/apache/nifi/processors/opentelemetry/io/StandardRequestContentListener.java
Outdated
Show resolved
Hide resolved
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/pom.xml
Show resolved
Hide resolved
|
It is alive and working - very cool. Need to learn more about what this java agent exporter really does but the json output is interesting. I did note after running for a minute or two I received a couple of these outputs. I was making minor edits to the flow when it happened but not this processor so dont have the underlying data it choked on.
|
|
@exceptionfactory I should add I am not particularly convicted on the byte[]/buffer stuff. Just seems like an opportunity to avoid intermediate copies. My mental model is this stuff could be in some cases fairly high rate and thus we want to be as efficient as we can. I didn't comment on the one related to decompression but it too seemed to be subject to some improvements. I looked at what we're doing in other areas such as this and we have byte[] usage but less so are we jumping between byte buffers to then new byte[] etc.. Ultimately it probably isn't worth looking into now but if we see high utilization and profiling suggests we can do better then we deal with it then. The error I noted and stack trace above I could not get to happen again during testing last night. It suggests some funky state can be entered but I'm not sure of the nature of it. Might well be a problem with the agent itself. |
exceptionfactory
left a comment
exceptionfactory
left a comment
There was a problem hiding this comment.
Thanks for the helpful feedback @joewitt! I pushed several changes to optimize deserialization, avoiding byte array creation in several places. The input processing went through a few iterations, so it was a good opportunity to avoid unnecessary memory consumption.
I have not observed the Protobuf processing exceptions you noted in testing, but if you find a way to reproduce the issue, that would be helpful.
As one additional test, setting -Dotel.exporter.otlp.protocol=http/protobuf exercises the Protobuf over HTTP protocol versus the default gRPC protocol. Those are different exporters in the agent, so it might be interesting to see if they behave differently.
.../main/java/org/apache/nifi/processors/opentelemetry/encoding/TelemetryMessageSerializer.java
Outdated
Show resolved
Hide resolved
nifi-nar-bundles/nifi-opentelemetry-bundle/nifi-opentelemetry-processors/pom.xml
Show resolved
Hide resolved
.../main/java/org/apache/nifi/processors/opentelemetry/encoding/TelemetryMessageSerializer.java
Outdated
Show resolved
Hide resolved
...rc/main/java/org/apache/nifi/processors/opentelemetry/encoding/JsonServiceRequestReader.java
Outdated
Show resolved
Hide resolved
...ntelemetry-processors/src/main/java/org/apache/nifi/processors/opentelemetry/ListenOTLP.java
Outdated
Show resolved
Hide resolved
...rc/main/java/org/apache/nifi/processors/opentelemetry/io/StandardRequestContentListener.java
Outdated
Show resolved
Hide resolved
|
thanks - running now at rate with the latest commits and getting a series of buffer size failures and connection resets. 2023-10-03 11:45:16,798 WARN [ListenOTLP[f3e6ea42-018a-1000-d8d4-3a6b40e25474]-1-2] o.a.n.p.opentelemetry.ListenOTLP ListenOTLP[id=f3e6ea42-018a-1000-d8d4-3a6b40e25474] Communication Failed with Remote Address [/127.0.0.1:60346] |
- Set maximum input content length to 10 MB - Set default Queue Capacity to 1000 - Set default Batch Size to 100
|
Thanks for the testing and feedback @joewitt! After reviewing the request content buffering and the available Netty Handlers, I consolidated the HTTP processing to work with the Netty FullHttpRequest for both HTTP/1.1 and HTTP/2. This streamlined the implementation in Following these changes, running a continuous stream of JSON files through As part of streamlining the handler pipeline, I added a 10 MB maximum content length limitation. Various service providers set request limits for OpenTelemetry payloads to 4 MB or 5 MB. This could be changed to a configurable property, but larger sizes also introduce potential memory usage concerns. Along similar lines, the default Queue Capacity is now 1000 and the default Batch Size is now 100, also limiting potential memory issues in the default configuration. |
|
All looks good. And cruising around 9500 events/sec on default settings. No more errors of any kind and smooth sailing. Thanks! I'll merge to main. Can you please adjust and merge to support as desired. +1 on that |
- Added ListenOTLP Processor supporting OpenTelemetry OTLP 1.0.0 Specification with gRPC and HTTP - Updated nifi-event-transport to support configurable SSLParameters for configurable Cipher Suites This closes #7830 Signed-off-by: Joseph Witt <joewitt@apache.org> (cherry picked from commit 6394912)
2 participants
Summary
NIFI-12115 Adds a
ListenOTLPProcessor to collect OpenTelemetry information, supporting OTLP Specification 1.0.0.OTLP defines HTTP as the basic communication protocol and outlines the following transport encoding options:
OTLP also includes support for gzip compression over gRPC or HTTP.
The
ListenOTLPProcessor supports the standard transport encoding options and uses content negotiation based on the HTTPContent-Typeheader to select the appropriate parsing implementation.The Processor converts incoming requests to standard OpenTelemetry objects and writes objects using the standard JSON encoding, which includes special handling for hexadecimal-encoding identifiers. Using JSON as the standard output format allows subsequent Processors to select or filter as necessary. Output FlowFiles can contain a batch of one or more resource elements, depending on data volumes and batch size configuration.
ListenOTLPadds theclient.socket.addressandclient.socket.portattributes to theResourcedefinition of received requests to provide an additional level of source tracking.The Processor requires an
SSL Context Serviceproperty to enforce communication over HTTPS. The gRPC protocol requires HTTP/2, which uses Application Protocol Negotiation in the TLS handshake to select the HTTP version. With the potentially sensitive nature of telemetry information, the initial version ofListenOTLPrequires HTTPS, and can be configured for mTLS withClient Authenticationrequired.Additional changes include updates to the
nifi-event-transportlibrary supporting configurable TLS Cipher Suites through optionalSSLParameterssettings. HTTP/2 requires a restricted set of TLS Cipher Suites, so this adjustment supports that approach.The opentelemetry-proto and jackson-datatype-protobuf libraries are both licensed under Apache Software License Version 2 with no additional notice files.
OTLP 1.0.0 JSON examples are available in the opentelemetry-proto examples directory.
For testing with an instrumented system, the OpenTelemetry Java Agent can be configured using Java System properties in
bootstrap.conffor basic instrumentation of JVM memory and Jetty calls using the following properties:The
trusted.pemfile must contain the issuer of the server certificate configured forListenOTLP. The OpenTelemetry Java Agent defaults to using gRPC for transport encoding without compression, but additional options can be configured based on the OTLP exporter properties.This pull request targets the main branch, with the intent of backporting the addition following minor adjustments.
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking
Pull Request Tracking
NIFI-00000NIFI-00000Pull Request Formatting
mainbranchVerification
Please indicate the verification steps performed prior to pull request creation.
Build
mvn clean install -P contrib-checkLicensing
LICENSEandNOTICEfilesDocumentation