Switch from Progrock to OpenTelemetry

[vito]

Out with Progrock, in with OpenTelemetry

Now that OpenTelemetry has support for streaming logs, we are able to switch from Progrock and embrace an open standard and ecosystem instead.

This change should be mostly invisible to the user, unless they want to integrate with OpenTelemetry. In that case, they can set the standard OTEL_ environment variables client-side, and all telemetry will be sent to the configured OTLP exporter, without requiring any engine-side configuration.

How does it work?

1. We use a custom OpenTelemetry pipeline that is able to send "live" spans to a configured exporter; this is how the TUI (and soon Cloud) is able to show you what's happening rather than what happened.
2. To get telemetry data out of the engine and to the TUI, the engine implements an OpenTelemetry publish/subscribe pattern. The CLI subscribes to its own trace, and the engine publishes telemetry to the subscriber.
3. We originally needed this pub/sub just so the TUI can show engine activity, but since we already had to do the work anyway, we also re-process data received from the pub/sub and send finished spans to your configured exporter. This way you only need to configure OTEL_ on the client side!
4. DagQL instrumentation now creates an Otel span for each field resolution and adds base64-encoded callpbv1.Call as a span attribute. Not that this is a single Call and not a full DAG, which saves us ~10x (unpredictable) data sent over the wire, but implies the consumer needs to keep track of all Calls that it sees in order to grok inter-Call references.
5. Dagger Cloud will become an OTLP endpoint that is capable of receiving "live" spans.

That covers tracing. What about logging?

1. The Go OpenTelemetry SDK does not yet implement support for logging, so we have a package of our own, basically copy-pasta from the tracing stack with changes to support logging, since it's already supported everywhere else. This can be removed once Logs SDK prototype open-telemetry/opentelemetry-go#4955 lands.
2. Same as above, we have a pub/sub stack for logs, too. There's no "live" vs "complete" distinction for logs though.
3. Our shim detects the current trace and span, which Buildkit already propagates for us, and pipes the command's stdout/stderr to io.Writers that send the logs to OpenTelemetry. This should maybe be upstreamed to Buildkit once 4955 lands.
4. DagQL instrumentation automatically configures iotctx.Stdout and ioctx.Stderr to send to the resolver's span.
5. To replace Progrock's "global logging" feature, we can now send logs with a special annotation that we can detect client-side to show them globally, while still maintaining a relationship to an originating span. This part isn't fully baked yet, but seems viable.
6. Dagger Cloud will become an OTLP endpoint that is capable of receiving logs for spans.
7. Engine logs can also be sent to Dagger Cloud as before, but I need to resurrect this code and confirm it works.

Breaking changes

• The old TUIs have been removed. So both the dagger run DAG TUI, and the ancient interactive tree view TUI. These were just not worth translating to the new world, so this was finally the time to GC them. Apologies to the handful of users that still used the tree view style UI.
• The --focus flag is gone.
• The pipeline() APIs are now no-ops.

New features / improvements

• The CLI now supports curl style verbosity flags.
  • -v keeps things from disappearing
  • -vv reveals internal/encapsulated spans
  • -vvv reveals spans that evaluated very quickly
• dagger watch allows you to see all activity across the engine, which is pretty neat. I've already used it for troubleshooting integration tests. It works by just subscribing to "everything" via the pub/sub service mentioned above.
• A span can be given a special attribute that tells the UI to collapse its children by default: telemetry.Encapsulate().
• We now proactively shut down services on client close.
• We should no longer have to worry about missing the 'last few bits' of telemetry. The pub/sub system now waits for active spans to complete before detaching. So hopefully that means no more "runaway builds" or services left running forever.
• The TUI now supports a Ctrl+\ keybind and SIGQUITs itself, so if there's a hang for whatever reason, now we can more easily get a stack dump.
• The Engine now uses the same OpenTelemetry stack as the rest of Dagger, so hopefully we don't have to worry too much about dependency hell. (Knock on wood.)
• All the labels that used to be sent to Cloud as part of Pipelines are now associated to the CLI's OpenTelemetry resource instead.
• User-land code can directly hook in to the broader trace; we automatically configure OTEL_ as appropriate in the shim.

Regressions

• We no longer use Buildkit's progress stream, so we are at the mercy of their Otel integration, which is not as rich.
  • We no longer show CACHED status for Buildkit cache hits. :( Instead you'll see separate spans like cache request: ... and load cache: ....
  • We no longer show image pull/extract progress, so no more fancy 2d progress bars.
  • We no longer send the LLB op protobuf in the telemetry.
  • All of these can probably be upstream PRs.

Notable tweaks

• DagQL no longer marks module functions as 'impure', which was causing parallel calls to never be deduped, leading to duplicate spans.
  • Technically we already cache module functions within a session, and the DagQL server cache has the same lifetime as the session, so this should not be a behavior change, but we'll want to change this if/when we take a swing at module function caching and/or publishing recipes.
• dagger terminal no longer does the insane vterm emulation thing, which was originally done so we can support showing progress alongside your active terminal session.
  • Now you should just use dagger watch in a separate terminal. Implementing a terminal emulator is not our core competency. It was fun, but also endless toil. The new implementation is much simpler.
• OpenTelemetry is very dependent on ctx propagation, and context.Background() breaks that, so now we use the new context.WithoutCancel() API which is nearly always what we wanted instead, since it keeps all the juicy values but just drops the cancel-propagation aspect.
• We now initialize the TUI frontend super early on - literally before we even initialize telemetry. This is great for various reasons, but the trade-off is we need to do custom barebones flag parsing to handle -v/--debug and other global flags. We can't defer that to the usual flag parsing since that's tied to module loading and stuff (for e.g. dagger call).
• We currently have our own hokey OpenTelemetry Go SDK for logging under telemetry/env and telemetry/sdklog which should be nuked as soon as the official SDK lands (Logs SDK prototype open-telemetry/opentelemetry-go#4955).
• Deleted various stale commands under cmd/ that weren't worth maintaining.
  • otel-collector (this is just native now)
  • dagger-graph (this was coupled to Progrock/Buildkit progress stream)
  • upload-journal (replaced by OTLP)

[vito]

Update: ok, Go SDK support for custom spans is implemented: 31b303b

But honestly it's not a great reference for other SDK authors; it's a bit hairy for not great reasons. I'd check out the Python SDK (or the Elixir SDK 🎉 - thanks @wingyplus!) instead. You can totally reference the 'live span' code though, that part's just copy-pasta from telemetry/inflight/ and shows how to implement it.

@sipsma @jedevc WDYT of refactoring the Go SDK codegen <=> Go SDK runtime <=> Dagger engine code sharing?¹ Rough idea is to change the runtime to work via replace dagger.io/dagger => ./internal/dagger, the same way the engine works. That way we can share code more easily without worrying about import paths. (Motivation: #6835 (comment))

That would result in the easiest central place to share code being the Go SDK, which kind of tracks I guess, since we're already incentivized to keep its dependencies minimal, as we would with any code shared between all three.

Would save that for a future PR, this one's already way too big.

Also @sipsma I think all the feedback is addressed now, modulo recent changes. I didn't find time to double-check the Buildkit memory thing so I just made it send to io.Discard and tweaked the comment. But the flag parsing situation is way less gross now.

🙏

Footnotes

1. Having a hard time figuring out what to call these, lemme know if there are better terms. By "Go SDK codegen" I mean dagger.io/dagger, by "Go SDK runtime" I mean the module codegen'd package, and by the Dagger engine I mean... the Dagger engine. ↩

[wingyplus]

I found an issue when I'm working on #6962. I create a file to sdk/elixir/test.exs with the snippet below

Application.ensure_all_started(:inets)

Mix.install([{:dagger, path: "."}])

defmodule Test do
  def run(dag \\ Dagger.connect!()) do
    dag
    |> Dagger.Client.container()
    |> Dagger.Container.from("alpine")
    |> Dagger.Container.with_exec(~w"echo hello")
  |> Dagger.Sync.sync()
  after
    Dagger.close(dag)
  end
end

Test.run

The script cannot compile because I'm accidentally make the sdk compilation error. When run with with-dev dagger run elixir test.exs I found this error

wingyplus@WINGYMOMO:~/src/github.com/dagger/dagger-elixir-otel/sdk/elixir$ with-dev dagger run elixir test.exs
==> dagger
Compiling 2 files (.ex)

== Compilation error in file lib/dagger/core/graphql_client/httpc.ex ==
** (SyntaxError) invalid syntax found on lib/dagger/core/graphql_client/httpc.ex:23:7:
    error: syntax error before: '['
    │
 23 │       []
    │       ^
    │
    └─ lib/dagger/core/graphql_client/httpc.ex:23:7
    (elixir 1.16.2) lib/kernel/parallel_compiler.ex:428: anonymous fn/5 in Kernel.ParallelCompiler.spawn_workers/8
could not compile dependency :dagger, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile dagger --force", update it with "mi
x deps.update dagger" or clean it with "mix deps.clean dagger"
21:42:58 WRN failed to get repo HEAD err="reference not found"
exit status 1
wingyplus@WINGYMOMO:~/src/github.com/dagger/dagger-elixir-otel/sdk/elixir$

Let's ignore Elixir error for now. The issue is when error occurred, my shell display is now broken for some reason. :(

[vito]

Thanks for the approval. 🙏

Integration tests were failing for the Python SDK, related to the new opentelemetry dependency, so I've just reverted those for now so we can get this PR merged first. Each SDK can just be a separate PR.

Will merge once checks are green.

[vito]

Update: some Python related tests flaked due to a concurrent map write here:

https://github.com/dagger/dagger/blob/main/sdk/python/runtime/discovery.go#L250

This is unrelated, but I've piled a quick fix onto my PR (just added a mutex), since it failed twice in a row.