* All Progrock plumbing is now gone, though we may want to bring it back
  for compatibility. Removing it was a useful exercise to find the many
  places where we're relying on it.
* TUI now supports -v, -vv, -vvv (configurable verbosity). Global flags
  like --debug, -v, --silent, etc. are processed anywhere in the command
  string and respected.
* CLI forwards engine traces and logs to configured exporters, no need
  to configure engine-side (we already need this flow for the TUI)
* "Live" spans are emitted to TUI and cloud, filtered out before sending
  to a traditional (non-Live) otel stack
* Engine supports pub/sub for traces and logs, can be exposed in the
  future as a GraphQL subscription
* Refactor context.Background usage to context.WithoutCancel. We usually
  don't want a total reset, since that drops the span context and any
  other telemetry related things (loggers etc). Go 1.21 added
  context.WithoutCancel which is more precise.
* engine: don't include source in slogs. Added this prospectively and it
  doesn't seem worth the noise.
* idtui: DB can record multiple traces, polish
  * multi traces is mostly for dagviz, so i can run it with a single DB
  * add 'passthrough' UI flag which tells the UI to ignore a span and
    descend into its children
  * add 'ignore' UI flag, to be used sparingly for things whose
    signal:noise ratio is irredeemibly low (e.g. 'id' calls)
  * make loadFooFromID calls passthrough
  * make Buildkit gRPC calls passthrough
* Global Progrock rogs are theoretically replaced with
  tracing.GlobalLogger, but it has yet to be integrated into anything.
* Module functions are pure after all. They're already cached
  per-session, so this makes DagQL reflect that, avoiding duplicate
  Buildkit work that would be deduped at the Buildkit layer. Cleans up
  the telemetry since previously you'd see duplicate queries.

* TODO: ensure draining is airtight
* TODO: global logging to TUI
* TODO: batch forwarded engine spans instead of emitting them "live"
* TODO: fix dagger terminal

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

previously we would cancel all subscribers for a trace whenever a
client/derver went away. but for modules/nesting this meant the inner
call would cancel the whole trace early.

* TODO: looks like services still don't drain completely?

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

previously service spans would be left incomplete on exit. now we'll
detach from them on shutdown, which will only stop the service if we're
the last depender on it. end result _should_ be that services are always
completed through telemetry, but I've seen maybe 2 in 50 runs still
leave it running. still troubleshooting, but without this change there
is no hope at all.

fixes dagger#6493

Signed-off-by: Alex Suraci <alex@dagger.io>

Honestly not 100% confirmed, but seems right. I think the final solution
might be to get traces/logs out without going through a session in the
first place.

Signed-off-by: Alex Suraci <alex@dagger.io>

seeing a panic in ExportSpans/UploadTraces, this should help avoid
bringing whole server down - I think - or at least give us hope.

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

fetching the logs/traces over a session is really annoying with draining
because the session itself gets closed before things can be fully
flushed.

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

More than a 10x efficiency increase. Frontend still super easy to
implement.

Test:

  # in ~/src/bass
  $ with-dev dagger call -m ./ --src https://github.com/vito/bass unit --packages ./pkg/cli stdout --debug &> out
  $ rg measuring out | cut -d= -f2 | xargs | tr ' ' '+' | sed -e 's/0m//g' -e 's/[^0-9\+]//g' | cat -v | bc

Before:

  8524838 (~8.1 MiB)

After:

  727039 (~0.7 MiB)

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

kinda hacky, but it makes sense that we need to handle this, cause
loadFooFromID or generally anything can take an ID that's never been
seen by the server before, and the loadFooFromID span will come first.

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Rather than closing the telemetry connection and hoping the timing works
out, we keep track of which traces have active spans and wait for that
count to reach 0.

A bit more complicated but not seeing a simpler solution really. Without
this we can't ensure that the client sees the very outermost spans
complete.

Signed-off-by: Alex Suraci <alex@dagger.io>

hasn't really been useful, it's available in the full trace for devs, or
we can add a verbosity level.

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

otherwise we can get stuck waiting for child spans of a nested process
that got kill -9'd. not perfect but better than hanging on Ctrl+C which
is already an emergent situation where you're not likely that interested
in any remaining data if you already had a reason to interrupt.

in Cloud we'll clean up any orphaned spans based on keepalives anyway.

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

we don't need to try synchronizing here now that we just generically
wait for all spans to complete

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

no substitute for live console streaming, but easier to implement for
now, and probably easier to read in CI. probably needs more work, but
might get some tests passing.

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

these are expected to fail now

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

caught by TestDaggerUp/random

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

promoting these from _EXPERIMENTAL along the way, which has already been
done for _TOKEN, don't really see a strong reason to keep the
_EXPERIMENTAL prefix, but low conviction

Signed-off-by: Alex Suraci <alex@dagger.io>

originally aluzzardi/otel-in-flight@2fc011f

Signed-off-by: Alex Suraci <alex@dagger.io>

really helps with troubleshooting hanging tests!

Signed-off-by: Alex Suraci <alex@dagger.io>

otherwise the terminals resize a ton of times when a long string is
printed, absolutely tanking performance. would be nice if that were
fast, but no time for that now.

Signed-off-by: Alex Suraci <alex@dagger.io>

i thought this wouldn't break it, but ... ?

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

really not sure what's writing to stderr even with --silent but this is
just too brittle. redirect stderr to /dev/null instead.

Signed-off-by: Alex Suraci <alex@dagger.io>

fixes Views test; frontend must have been getting confused because there
were multiple "primary" spans

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

this was no longer needed with the change to wait for spans to finish,
not worth the review-time distraction

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Some clients presume this service is supported by the OTLP endpoint. So
we can just have a stub implementation for now.

Signed-off-by: Alex Suraci <alex@dagger.io>

TODO: set up otel for you

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

Previously we were just sending each individual update to the configured
exporters, which was very expensive and would even slow down the TUI.

When I originally tried to send it to span processors, nothing would be
sent out; turns out that was because the transform.Spans call we were
using didn't set the `Sampled` trace flag.

Now we forward engine traces and logs to all configured processors,
so their individual batching settings should be respected.

Signed-off-by: Alex Suraci <alex@dagger.io>

* fix detection for in flight spans; we need to check EndTime <
  StartTime since sometimes we end up with a 1754 timestamp
* when a span is already present in a batch, update it in-place rather
  than dropping it on the floor

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Helder Correia <174525+helderco@users.noreply.github.com>

more universally compatible than unix://

Signed-off-by: Alex Suraci <alex@dagger.io>

Signed-off-by: Alex Suraci <alex@dagger.io>

also start cleaning up "tasks" cruft nonsense, these can just be plain
old attributes on a single span i think

Signed-off-by: Alex Suraci <alex@dagger.io>

parse global flags in two passes, ensuring the same flags are installed
in both cases, and capturing the values before installing them into the
real flag set, since that clobbers the values

Signed-off-by: Alex Suraci <alex@dagger.io>