Propagate OTEL context between ZIO and non-ZIO code

[dmytr]

Motivation

At the moment OTEL context is not propagated between ZIO and non-ZIO code. Of course, there are functions from scopedEffect family provided by Tracing, but they expect an unsafe size-effecting code as their argument.

In case if you need to propagate context into libraries that wrap non-ZIO code into ZIO, but don't use tracing - there is no guarantee that underlying non-ZIO code will have access to the correct context.

And in cases when non-ZIO code invokes ZIO code (for example in zhttp, which starts fibers from Netty code) context is not (correctly) propagated as well.

This change makes it possible to use automatic instrumentation provided by OpenTelemetry JVM agent.

Related: #561

How this works

This solution uses a special ZIO supervisor that takes a snapshot of the current OpenTelemetry context before a new ZIO fiber is started and uses it as a parent context in the scope of that fiber.

Each time a fiber is suspended, supervisor takes a fresh snapshot of the current context and then restores the root context. When fiber is resumed, it restores the last context snapshot for that fiber.

When fiber is completed, supervisor restores the root context.

Probably, this is not the most correct or efficient implementation, but it does seem to work.

There is an end-to-end setup that uses this kind of supervisor here: https://github.com/dmytr/zio-otel-instrumentation.

[grouzen]

I guess it can be replaced with https://zio.dev/reference/sync/concurrentmap. As a consequence, you will have Supervisor[UIO[Unit]] and @nowarn annotation might be omitted.

[dmytr]

I'm afraid we'll have to stick to the regular ConcurrentHashMap because all callbacks in Supervisor are unsafe and return Unit. Unsafe side effects have to be executed within those callbacks for context to be properly propagated into non-ZIO code.

[dmytr]

@nowarn basically tells us that we are throwing away Scope that we get when we call Context.makeCurrent(). Ideally we shouldn't do this. I saw that other tracing libraries (like Datadog's tracing SDK) use stacks of scopes. So potentially we could change storage to be of type ConcurrentHashMap[FiberId, Stack[Scope]]. But this is kind of involved and apparently this all works even when we do throw scopes away.

[grouzen]

@dmytr Thanks for your contribution! Sorry for the delay, I honestly need some time to process this PR.

[grouzen]

Thanks again, @dmytr, for your contribution! It looks good to me.

[easel]

Very nice contribution. Given the interface surface area change is very small, I'd advocate for merging and releasing soon to make it easier to get more feedback. In general, I think the approach is the right one -- configure the zio runtime appropriately and context propagates as expected into and out-of the zio runtime.

[grouzen]

@dmytr Hey! Can we merge it?

[dmytr]

@grouzen yes, please. I don't have access to do this.

[dmytr]

I've updated my test harness to use the published snapshot version: dmytr/zio-otel-instrumentation#2
It works as expected in that specific setup 🥳

[grouzen]

I'll cut a release ASAP.
Update:
https://github.com/zio/zio-telemetry/releases/tag/v2.0.3