✨ [UX/DX] Dagger logs alarming "HTTP HEAD ERROR" for routine registry cache misses

[smolinari]

What are you trying to do?

I am running Dagger in a centralized "Build Service" where a CLI triggers remote builds. I connect to the Dagger Engine via the Go SDK and pipe the raw log stream directly back to the user's terminal using dagger.Connect(ctx, dagger.WithLogOutput(http.ResponseWriter), ...).

I want to present this log stream to developers without them seeing prominent red ERROR logs when Dagger essentially hits a "cache miss" (404) while checking for existing image layers in the registry.

Example of the current noisy output during a successful build:

13  : remotes.docker.resolver.HTTPRequest
14  : ┆ HTTP HEAD
14  : ┆ HTTP HEAD ERROR [0.3s]
13  : remotes.docker.resolver.HTTPRequest ERROR [0.3s]

Why is this important to you?

We prioritize a "No WTF moments" Developer Experience. Currently, when a developer sees a wall of red errors, their immediate emotional response is "The build failed." It causes confusion and unnecessary panic, making the logs look broken even when the operation is 100% successful.

More importantly, this behavior trains users to ignore ERROR logs, which is a dangerous habit. We want ERROR to mean "Action Required" or "Process Failed," not "Routine Check Returned False."

Here is an image of what it looks like for a dev:

How are you currently working around this?

We are currently forcing developers to tolerate the noise and explaining that "it's normal."

The only technical workaround would be to implement a brittle "Filtering Writer" in our application that intercepts the raw byte stream, parses for these specific error strings, and discards them before writing to the HTTP response. We would prefer not to maintain this kind of log-scraping logic downstream.

[Mjb141]

Seconding this, experiencing it with DockerBuild.Publish and it has caused confusion.

[marcosnils]

thx for reporting. This is a tricky one since those span errors are coming from within buildkit > containerd > otelhttp (https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/instrumentation/net/http/otelhttp/internal/semconv/client.go#L48-L56) where they flag any request with a status code >=400 as an error ERROR

A smilar thing happens also when pulling an image and the client initially tries to fetch the manifest and usually receives a 401 response indicating that a temporary token must be obtained first before requesting the operation. This also yields to an ERROR span within the trace.

When using dagger via the standard CLI, we annotate these spans with ui.encapsulated so they have a lower priority and are not displayed unless a reasonable high verbosity level is issued. In this particular case, since you're parsing the output of the CLI and you don't have this metadata, it's hard for you to accomplish this.

With all this said, I'm unsure what the "right" way to proceed here from the Dagger side is given that this doesn't represent an issue when using dagger call or dagger run as the TUI will not show those spans by default as they're intentionally encapsulated. On the other hand, adding specific conditions and/or filters in our side for this specific scenario doesn't also feel right given that we have the chance to overlook some real errors if we start adding specific conditions to handle this in a specific way.

I'm mentioning @vito in case he might have some other opinions about how to proceed here.

[smolinari]

@marcosnils

When using dagger via the standard CLI, we annotate these spans with ui.encapsulated so they have a lower priority and are not displayed unless a reasonable high verbosity level is issued. In this particular case, since you're parsing the output of the CLI and you don't have this metadata, it's hard for you to accomplish this.

That context about ui.encapsulated is super helpful!

Two questions on that:

• Could you point me to where that filtering logic lives in your codebase? I'd love to see how you determine what gets hidden.

• Does the Go SDK expose that metadata to the consumer? If I could implement a custom Logger that respects ui.encapsulated, I could replicate your CLI's behavior in my service. Currently WithLogOutput just seems to give me the flattened text stream.

[Mjb141]

Yeah realistically I'm not implementing a custom logger, and I don't think that's a scalable solution for everyone using DockerBuild.Publish.

as the TUI will not show those spans by default as they're intentionally encapsulated.

All Gitlab users will see these errors - the Gitlab terminal output is extremely basic and is equivalent to --progress=plain. I'd appreciate if Dagger could prevent these errors displaying so the user doesn't have to.

[verdverm]

I also advocate for doing something about this, I get pinged all the time for these loglines and have to remind them this is a HEAD call that is expected to fail and make our logs noisy

[nipuna-perera]

Have y'all tried --progress=dots? I just checked and I don't see those errors.

[vito]

This feels like a bug with the plain UI frontend, especially if those spans are already marked ui.encapsulated (don't have a trace handy to confirm).

[smolinari]

Have y'all tried --progress=dots? I just checked and I don't see those errors.

Unfortunately, dots hides basically everything. The process we are emulating is local iteration on a Dockerfile (from a remote dev environment). The developer needs the plain output to debug errors as quickly as possible. Having false-alarm errors mixed in with real process logs detracts significantly from that debugging loop.