Add Trace.spanR #526

rossabaker · 2022-02-28T15:13:34Z

Adds a way to span a resource, from acquisition through release.

rossabaker · 2022-02-28T15:25:37Z

modules/core/shared/src/main/scala/Trace.scala

@@ -22,6 +22,9 @@ trait Trace[F[_]] {
   */
  def kernel: F[Kernel]

+  /** Creates a new span, and within it acquires and releases the spanR `r`. */
+  def spanR[A](name: String)(r: Resource[F, A]): Resource[F, A]


There are two competing designs here. Bayou defines a spanR(name: String): Resource[F, Unit].

Bayou's signature is biased toward a "stateful" trace. There is no Local implementation, including Kleisli[F, Span[F], *] implementation, because the resource span can't be threaded through to the resource effect.

This signature supports Local semantics (with the limitation that use does not see the resource span as the ambient span). But it does not support monad transformers T. The only way I've thought to implement them is to apply a natural transformation T ~> F to use Trace[F].spanR, but this does not exist for StateT, EitherT, OptionT, or Nested, which represent the compilation failures in this branch.

Is it possible to span a Stream with this design?

Hmm. Yes, sort of? This would cover the entire stream, but not be ambient. There's no .allocated on Stream that gives us similar hooks to what we have on Resource:

def stream[F[_]: Trace: MonadCancelThrow, A](name: String)(s: Stream[F, A]): Stream[F, A] = Stream.resource(Trace[F].spanR(name)(Resource.unit)) >> s

In other words, if we tried to span each value, all the "emit" spans would be siblings of the stream span:

def stream[F[_]: Trace: MonadCancelThrow, A](name: String)(s: Stream[F, A]): Stream[F, A] = Stream.resource(Trace[F].spanR(name)(Resource.unit)) >> s.translate(new (F ~> F) { def apply[B](fb: F[B]): F[B] = Trace[F].span("emit")(fb) })

This would not be true in your preferred definition of Trace[IO].

I guess that's not all that different from this definition for the transformer instances. It spans r, but is not ambient for its acquisition or release:

def spanR[A](name: String)(r: Resource[StateT[F, S, *], A]): Resource[StateT[F, S, *], A] = trace.spanR(name)(Resource.unit).mapK(StateT.liftK[F, S]) >> r

rossabaker · 2022-02-28T15:27:31Z

modules/core/shared/src/main/scala/Trace.scala

+        def spanR[A](name: String)(r: Resource[IO, A]): Resource[IO, A] =
+          Resource.eval(local.get).flatMap(parent =>
+            parent.span(name).flatMap { child =>
+              def inChild[B](io: IO[B]): IO[B] =
+                local.set(child).bracket(_ => io)(_ => local.set(parent))
+              Resource(inChild(r.allocated).map { case (a, release) =>
+                a -> inChild(release)
+              })
+            }
+          )
+


A simpler definition would just bracket the entire resource with child, and also make child the ambient span for use. But this definition is consistent with the Local semantics: child still spans the resource, and is ambient for acquire and release.

👍

I still think it would be nice to offer the alternative implementation as well. In practice it's what I'd want to use, and I don't care if its inconsistent with Kleisli!

rossabaker · 2022-02-28T15:28:23Z

modules/core/shared/src/main/scala/Trace.scala

@@ -140,7 +177,7 @@ object Trace {

  }

-  implicit def liftKleisli[F[_], E](implicit trace: Trace[F]): Trace[Kleisli[F, E, *]] =
+  implicit def liftKleisli[F[_]: MonadCancelThrow, E](implicit trace: Trace[F]): Trace[Kleisli[F, E, *]] =


Even if we can salvage the other monad transformers, they will probably all need a MonadCancelThrow constraint to deal with transforming the Resource.

I have a PR futzing with Resource#mapK constraints but I don't think it helps much here.

Relax Resource#mapK constraints cats-effect#2646

I think I encountered this working on Bayou, and I wondered if it was a good argument to have two Trace typeclasses after all.

I don't think that PR helps much here: you still need MonadCancelThrow[F], and we're going F ~> G and G ~> F unless someone is more clever than me. (Possible! Nay, probable!)

The base instances, other than noop, all already require MonadCancelThrow, so I don't think this limits which effects can be traced. But it could require a stronger constraint in a few places along the way.

armanbilge · 2022-03-01T02:16:01Z

modules/core/shared/src/main/scala/Trace.scala

+      def spanR[A](name: String)(r: Resource[StateT[F, S, *], A]): Resource[StateT[F, S, *], A] =
+        trace.spanR(name)(Resource.unit).mapK(StateT.liftK[F, S]) *> r


I haven't fully grasped the implications of these definitions, but is this much different than if we tried to implement spanR(name: String): Resource[F, Unit] instead?

I haven't tested this, but ...

If the transformed instance is "stateful" (parents acquire, use, and release), so are these, and would be nicer expressed as spanR(name: String): Resource[F, Unit].

If the transformed instance is "local" (parents acquire and release but not use), we still span the resource, but parent none of the three phases.

Local and stateful tracing are equivalent through the current Trace and the current abstraction works well until we introduce resources.

Resource tracing can be implemented either in a stateful subclass without local instances, or in the Trace constraint with an awkward signature and weird semantics for the transformers.

If we resort to a subclass, libraries would be choosing the tradeoff on behalf of apps.

I see, thanks, it took me a while to grasp this. IIUC, for a "stateful" instance if you have

def spanR[A](name: String)(r: Resource[F, A]): Resource[F, A]

then you can get "for free"

def spanR(name: String): Resource[F, Unit] = spanR(name)(Resource.unit)

with the exact same semantics as if Trace had been defined in terms of spanR(name: String): Resource[F, Unit] to begin with.

However, for a stateless instance there is a difference whether you use spanR[A](name: String)(r: Resource[F, A]) or spanR(name: String): Resource[F, Unit]. So you want to use the former whenever possible, and only use the latter when necessary e.g. Stream and transformer instances.

So in summary, defining Trace as in this PR doesn't compromise the power of the stateful IOLocal instance, it will work exactly like it does for a Bayou-style Trace.

What it does compromise however is the library experience: since we want spanR(name: String): Resource[F, Unit] to be the second-choice, we don't directly offer it on the typeclass, even though it's easily derived in practice.

Yes:

acquire release use

Stateful base ✅ ✅ ✅

Stateful transformed ✅ ✅ ✅

Stateless base 🤷 🤷 🚫

Stateless transformed 🚫 🚫 🚫

✅ - both spanR
🚫 - neither spanR
🤷 - spanR(name: String)(r: Resource[F, A]), but not spanR(name: String)

Cool, thanks. This PR seems good then: it doesn't compromise the stateful IOLocal instance, while still being at least somewhat compatible with Kleisli. On the library side, you can trace any Resource-like thing either directly or with trace.spanR(name)(Resource.unit). So everyone should be happy, right?

I'm still not convinced it's the right abstraction when we have a type class where a column mixes ✅ and 🚫.

I'll fix the other modules, unmark it as a draft, and we'll see if anyone else has any thoughts.

I'm still not convinced it's the right abstraction when we have a type class where a column mixes ✅ and 🚫.

That would leave us with "Stateless I" I believe? Where you can span a resource-like thing but can't parent any of the acquire/use/release phases.

FWIW I'd like Stateless I even more. Then we can have spanR(name: String): Resource[F, Unit]!

Under Stateless I, we can never add fields to any phase of a resource span. It is never ambient and can't be retrieved.

#527 is the most consistent and this has the broadest support. I think that's the tradeoff we're stuck with.

That's assuming that we can't have "inconsistent" implementations of Trace, which I think is fine.

Correct. This supports everything with one type class, with three distinct behaviors. #527 supports one behavior, but drops the instances that can't support it once we upgrade to a resource constraint.

#527 still feels more right to me. I'll undraft this and maybe we can find a mediator. 😆

rossabaker · 2022-03-10T03:22:54Z

I am in favor of continuing with #527 over this one, particularly if typelevel/fs2#2843 works out.

armanbilge · 2022-03-10T03:27:07Z

Is natchez okay to take on an fs2 dependency?

rossabaker · 2022-03-10T03:35:22Z

The repo already depends on it for xray. Not sure it's smart for core to take it on, but an fs2 module wouldn't hurt.

(Half-baked idea: as @zmccoy is fond of saying, tracing is most important around network calls. Having a natchez-fs2 that traced some of its io functions would be cool.)

rossabaker · 2022-03-11T03:29:40Z

It turns out a spanS is a lot easier than spanR for the Kleisli instance. translate gives us a hook to thread the state (here, just a String) through to the outputs:

    def spanS[F[_], A](name: String)(s: Stream[Kleisli[F, String, *], A]): Stream[Kleisli[F, String, *], A] =
      s.translate(
        new (Kleisli[F, String, *] ~> Kleisli[F, String, *]) {
          def apply[B](k: Kleisli[F, String, B]) = k.local(_ => name)
        }
      )

    val s = Stream.eval(Kleisli.ask[IO, String])
    (s ++ spanS("child")(s) ++ s)
      .compile
      .toList
      .run("root")
      .flatMap(IO.println)

prints List("root", "child", "root").

rossabaker · 2022-03-12T18:10:56Z

Tracing can be viewed as an exchange F ~> F of an untraced effect F with a traced effect F. This is witnessed by the polymorphic function that results from partially applying Trace.span.

If we redefine spanR as something that returns a resource of F ~> F, we decouple how we propagate the span from the structure of transformed effects. Thus, spanR supports a Trace[Resource[F, *], *], Trace[Stream[F, *], *], and any other "dynamically scoped" monad, all without introducing a second type class.

Expectation is that:

spanR(name).surround(fa) <-> span(name)(fa)

Caveats:

I haven't tested any of these.
The transformer instances still all require MonadCancelThrow.
Trace[Resource[F, *]] is still only ambient for acquisition and release, not use. That's the price of stateless instances.
Any Stream interruption hacks would have to be universal, because we're not defining a specific spanS.

rossabaker · 2022-03-12T23:51:16Z

The API for tracing a stream (example from armanbilge/bayou#1) ends up looking like

  def stream(implicit trace: Trace[IO]): Stream[IO, Unit] = {
    Trace[Stream[IO, *]].span("new_root")(
      Stream(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
        .covary[IO]
        .chunkN(3)
        .flatMap { chunk =>
          Trace[Stream[IO, *]].span("child")(
            Stream.eval(trace.put("vals" -> chunk.toList.mkString(","))) >>
            Stream.sleep[IO](1.second)
          )
        }
    )
  }

To weave in and out of Stream, we have to accept the simpler Trace[IO] and then summon a Trace[Stream[IO, *]]. We could just reference trace if we made spanS a derived method on Trace, but then we'd need a MonadCancelThrow member, and fs2 would be forced into natchez-core.

ryanmiville · 2022-06-09T21:30:19Z

Just figured I'd share that we've been using a custom Trace at work for the last few months that is essentially a combination of Bayou and the F ~> F implementation here and it's been working well for us.

I've open sourced it here. I don't intend to promote yet another tracing library. I'd much rather these concepts make it into Natchez.

mpilquist · 2022-11-19T14:10:39Z

Any thoughts on this design versus #527 now that some time has passed? @armanbilge @rossabaker

rossabaker · 2022-11-21T20:48:42Z

I moved onto otel4s and don't remember all the nuances, but I think the basic tradeoff was that this behaves similarly between stateless and stateful instances, and #527 was able to make the resource span the parent of use at the expense of stateless instances (and an extra type class.)

Thanks for picking it up.

WIP: add Trace.spanR

e530c72

rossabaker force-pushed the spanR branch from 80fed73 to e530c72 Compare February 28, 2022 15:14

rossabaker commented Feb 28, 2022

View reviewed changes

rossabaker mentioned this pull request Feb 28, 2022

Tracing resources #514

Closed

Core compiles, unsatisfyingly

a143ea0

armanbilge reviewed Mar 1, 2022

View reviewed changes

rossabaker mentioned this pull request Mar 1, 2022

Introduce a TraceResource type class #527

Closed

Compile noop and mtl

3cc5860

rossabaker marked this pull request as ready for review March 2, 2022 03:35

armanbilge mentioned this pull request Mar 10, 2022

Introduce Stream#noInterruptScope typelevel/fs2#2843

Closed

Redefine spanR as a natural transformation, add fs2

e2afc85

mpilquist changed the base branch from series/0.1 to main November 19, 2022 14:10

mpilquist added 4 commits November 19, 2022 09:26

Merge branch 'main' into spanR

ad3c1d9

Fix bad merge

d2845f5

Fix bad merge

e2bf950

Bump base version

613f544

mpilquist merged commit 73bf08b into typelevel:main Nov 19, 2022

rossabaker mentioned this pull request Jan 27, 2023

Add a way to get the current span typelevel/otel4s#101

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Trace.spanR #526

Add Trace.spanR #526

rossabaker commented Feb 28, 2022

rossabaker Feb 28, 2022

armanbilge Feb 28, 2022

rossabaker Feb 28, 2022 •

edited

rossabaker Feb 28, 2022

rossabaker Feb 28, 2022

armanbilge Feb 28, 2022

rossabaker Feb 28, 2022

armanbilge Feb 28, 2022

rossabaker Feb 28, 2022

armanbilge Mar 1, 2022

rossabaker Mar 1, 2022

armanbilge Mar 1, 2022 •

edited

rossabaker Mar 1, 2022

armanbilge Mar 1, 2022 •

edited

rossabaker Mar 2, 2022

armanbilge Mar 2, 2022 •

edited

rossabaker Mar 2, 2022

armanbilge Mar 2, 2022

rossabaker Mar 2, 2022

rossabaker commented Mar 10, 2022

armanbilge commented Mar 10, 2022

rossabaker commented Mar 10, 2022

rossabaker commented Mar 11, 2022

rossabaker commented Mar 12, 2022

rossabaker commented Mar 12, 2022

ryanmiville commented Jun 9, 2022

mpilquist commented Nov 19, 2022

rossabaker commented Nov 21, 2022

		def spanR[A](name: String)(r: Resource[StateT[F, S, ], A]): Resource[StateT[F, S, ], A] =
		trace.spanR(name)(Resource.unit).mapK(StateT.liftK[F, S]) *> r

	acquire	release	use
Stateful base	✅	✅	✅
Stateful transformed	✅	✅	✅
Stateless base	🤷	🤷	🚫
Stateless transformed	🚫	🚫	🚫

Add Trace.spanR #526

Add Trace.spanR #526

Conversation

rossabaker commented Feb 28, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rossabaker Feb 28, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

armanbilge Mar 1, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

armanbilge Mar 1, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

armanbilge Mar 2, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rossabaker commented Mar 10, 2022

armanbilge commented Mar 10, 2022

rossabaker commented Mar 10, 2022

rossabaker commented Mar 11, 2022

rossabaker commented Mar 12, 2022

rossabaker commented Mar 12, 2022

ryanmiville commented Jun 9, 2022

mpilquist commented Nov 19, 2022

rossabaker commented Nov 21, 2022

rossabaker Feb 28, 2022 •

edited

armanbilge Mar 1, 2022 •

edited

armanbilge Mar 1, 2022 •

edited

armanbilge Mar 2, 2022 •

edited