Added support for opentelemetry trace exporter

[pnerg]

Overview

This PR adds an exporter for exporting traces/spans according to OpenTelemetry Protocol (OTLP) as requested in #973.
The code has been tested against an instance of the OpenTelemetry Collector

What is not supported from the OTEL spec

• metrics (should be done at some point as a separate exporter in the same sbt sub-module)
• logs (spec still in draft and probably not for Kamon anyways)

What is not implemented in this first draft

Some things were left out for sake of getting something out there.

Support for custom trust store for managing TLS

The exporter supports HTTP (default) and HTTPS but there is no way to add custom trust store or keychains

No labels/tags/keyvalues are dropped

The spec allows for dropping meta-information in case if.

• the meta-data is too long
• there's too many instances of meta-data (e.g tags)

But dropping meta-information requires that the exporter notes how many were dropped.
So for sake of simplicity the exporter includes all environment/span tags

Assumptions made on the OTLP model

The data model for exporting spans looks like:

ExportServiceRequest - -> *  ResourceSpans
ResourceSpans -->* InstrumentationLibrarySpans
ResourceSpans -->1 Resource
InstrumentationLibrarySpans --> * Span
InstrumentationLibrarySpans --> 1 InstrumentationLibrary

1- is 1-1
*- is one to many

So one ExportServiceRequest can have multipleResourceSpanswhich in turn can have multiple InstrumentationLibrarySpanswhich finally has multiple Spans

I've assumed that always is a single resource (i.e. the application itself) and there is only one instrumentation library (kamon) and all spans belong in that chain.
So the meta-information in the Resource belonging to ResourceSpans comes largely from optional Kamon env tags.
The meta-information in the instrumentation library is info on Kamon.

Local testing

I've used the OTEL collector in a container and provided it with config to just print all traces it receives
otel-collector.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:55690

exporters:
  logging:
    loglevel: debug

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

Then just run this command (requires Docker)

docker run --rm -p 55690:55690 \
-v "${PWD}/otel-collector.yaml":/otel-local-config.yaml \
--name otelcol otel/opentelemetry-collector \
--config otel-local-config.yaml; \

Example printout from OTEL collector

Note the Resource labels as they're from the Kamon env tags apart from telemetry.sdk.* which we always add.
The InstrumentationLibrary kamon 2.1.14 is the InstrumentationLibrary meta-info provided

Resource labels:
     -> service.name: STRING(otlp-test-app)
     -> telemetry.sdk.name: STRING(kamon)
     -> telemetry.sdk.language: STRING(scala)
     -> telemetry.sdk.version: STRING(2.1.14)
     -> notset: STRING(/Users/peter.nerg)
     -> env: STRING(staging)
     -> service.instance.id: STRING(xxx-yyy)
     -> service.namespace: STRING(namespace-1)
     -> service.version: STRING(x.x.x)
InstrumentationLibrarySpans #0
InstrumentationLibrary kamon 2.1.14
Span #0
    Trace ID       : 4a7e482ebab331570cfbfc070dd5e70f
    Parent ID      : 
    ID             : 2c47181be9a542a2
    Name           : /purchase-item/{id}
    Kind           : SPAN_KIND_SERVER
    Start time     : 2021-03-30 12:48:37.244977523 +0000 UTC
    End time       : 2021-03-30 12:48:37.593127187 +0000 UTC
    Status code    : STATUS_CODE_OK
    Status message : 
Attributes:
     -> method: STRING(POST)
Span #1
    Trace ID       : 4a7e482ebab331570cfbfc070dd5e70f
    Parent ID      : 2c47181be9a542a2
    ID             : 4ccac7b8965abd99
    Name           : add-order
    Kind           : SPAN_KIND_CLIENT
    Start time     : 2021-03-30 12:48:37.288737377 +0000 UTC
    End time       : 2021-03-30 12:48:37.592803111 +0000 UTC
    Status code    : STATUS_CODE_OK
    Status message : 
Attributes:
     -> method: STRING(put)
Span #2
    Trace ID       : 4a7e482ebab331570cfbfc070dd5e70f
    Parent ID      : 2c47181be9a542a2
    ID             : c9c8937819d35d41
    Name           : check-credits
    Kind           : SPAN_KIND_CLIENT
    Start time     : 2021-03-30 12:48:37.258310723 +0000 UTC
    End time       : 2021-03-30 12:48:37.284974414 +0000 UTC
    Status code    : STATUS_CODE_OK
    Status message : 
Attributes:
     -> method: STRING(query)

[pnerg]

First test run was green, now something related to MongoDB broke.

[pnerg]

Hi guys, any progress/comments on this PR or did I DoS attack you?
Even if the PR is quite big it is at least isolated as I've added a new functionality and not touched the existing code.
So let's go brave and hit that "approve" button, you know you want to do that...:wink:
"There is no problem in the code, let it be approved"/Jedi mind trick

[SimunKaracic]

Don't worry, I'll take a hard look at everything you did tomorrow ;)

[SimunKaracic]

Looking great, no major complaints for me. I'll have to dive a bit deeper though, so on monday expect either smaller feedback or just a straight up merge

[SimunKaracic]

Other than those few small nitpicks, this is great. There's still work to do, but what is there is readable and well tested.
GJ!

[pnerg]

Yep, there's a few things that can be added in the continuation. That's why I added the notes in this PR on what is yet to be done. Also left a few TODOs in the code to mark where more features can be added.
IMO this makes for a decent first version it at least covers the most important parts of the OTLP spec.

[SimunKaracic]

Thanks for the contribution!
Will try to release this today!

[pnerg]

Any plans for when you'll do a new release?
#impatient

[SimunKaracic]

Now where would the fun be in just doing things on time?

Jokes aside, thanks for the patience 🎉