🐞 WithServiceBinding + WithExec fails: dnsmasq addnhosts has IP entries with empty hostnames

[gtzo-anchorage]

What is the issue?

When a consumer container has any WithServiceBinding(alias, svc) and is then evaluated with WithExec(...).Stdout(ctx) (or .Sync(ctx)), the exec fails immediately at host-alias setup time with:

lookup <hash> for hosts file: lookup <hash> on 10.87.0.1:53: no such host
lookup <hash>.<session>.dagger.local on 10.87.0.1:53: no such host

This happens before the consumer's command runs at all — the consumer's command can be a harmless echo, never referencing the bound alias, and it still fails. Plain WithExec (no service binding) works. Starting the service in isolation (Service.Start(ctx) + Service.Hostname(ctx) + Service.Endpoint(ctx)) also works.

Direct inspection of the engine's dnsmasq hosts file while the service is running shows the bug's mechanism:

$ docker exec dagger-engine-v0.20.8 cat /var/run/containers/cni/dnsname/dagger/addnhosts
10.87.0.2	
10.87.0.3	
...
10.87.0.17	

Every entry has an IP but an empty hostname. The expected line for the service container (e.g. 10.87.0.18 <hash>) is never written. So when engine/buildkit/executor_spec.go does net.LookupIP(<hash>) to populate the consumer's /etc/hosts (around line 283), dnsmasq has no record and returns NXDOMAIN.

Looks related to #6951 (closed; same error string) and #13060 (open; different trigger). This reproduction does not use PRIVATE cache mounts or modules, so it's not #13060 specifically.

Dagger version

dagger v0.20.8 (image://registry.dagger.io/engine:v0.20.8) linux/amd64

Also reproduces on v0.18.19, where the failure was silent / indefinite hang instead of fast error — the upgrade made it diagnosable.

Steps to reproduce

Minimal Go test (go 1.26, dagger.io/dagger v0.20.8):

//go:build daggerE2ETest

package minrepro

import (
	"context"
	"testing"
	"time"

	"dagger.io/dagger"
	"github.com/stretchr/testify/require"
)

func TestServiceBindingDNS(t *testing.T) {
	ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
	defer cancel()
	dag, err := dagger.Connect(ctx, dagger.WithLogOutput(testWriter{t}))
	require.NoError(t, err)
	t.Cleanup(func() { _ = dag.Close() })

	svc := dag.Container().
		From("alpine:3.20").
		WithExposedPort(8888).
		AsService(dagger.ContainerAsServiceOpts{
			Args: []string{"sh", "-c", "echo serving; nc -lk -p 8888 -s 0.0.0.0"},
		})

	// Consumer binds the alias but its exec never references it.
	out, err := dag.Container().
		From("alpine:3.20").
		WithServiceBinding("served", svc).
		WithExec([]string{"sh", "-c", "echo 'consumer running'; cat /etc/hosts"}).
		Stdout(ctx)
	require.NoError(t, err, "bound-but-unused consumer must succeed")
	t.Logf("stdout:\n%s", out)
}

type testWriter struct{ t *testing.T }

func (w testWriter) Write(p []byte) (int, error) {
	w.t.Logf("[dagger] %s", string(p))
	return len(p), nil
}

Run with go test -tags daggerE2ETest -count=1 -v -run TestServiceBindingDNS ..

What we ruled out

• Test design / runtime DNS use: consumer's command never resolves the alias; the failure is at WithExec network setup time, not inside the container.
• Stale state: failure is identical on a fresh docker rm -f engine container.
• Contention: failure is identical on a near-idle host (loadavg ~5) and a loaded one.
• IPv4-only listener / readiness probe mismatch: explicit nc -s 0.0.0.0 (IPv4 wildcard) makes no difference.
• Cache key drift (regression: private cache mounts can split service endpoint and binding identity #13060 trigger): this repro uses no PRIVATE cache mounts.
• Start(ctx) anchoring: pre-anchoring the service identity via svc.Start(ctx) before binding to a consumer makes no difference.
• Custom hostname: Service.WithHostname("named") makes no difference — same error with the chosen name in place of the random hash.

Workaround that works

dag.Host().Tunnel(svc).Start(ctx) + dial the tunnel endpoint from the host process succeeds in ~1s on v0.20.8. (Tunnel was reportedly broken on v0.18.19 in similar environments.) This sidesteps the WithServiceBinding consumer setup path entirely.

Where it appears to break (engine side, code references)

• Lookup site: engine/buildkit/executor_spec.go:283 — net.LookupIP(qualified) returns NXDOMAIN because dnsmasq doesn't know the service hostname.
• Registration site: cmd/dnsname/files.go:14-35 writes IP\tpodname\t… to addnhosts. With podname == "", the empty-hostname entries we observe are produced.
• CNI args wiring: internal/buildkit/util/network/cniprovider/cni.go:269-277 — K8S_POD_NAME is only passed when the namespace is created with hostname != "". Pool slots (created via pool.provider.newNS(ctx, "")) intentionally have empty hostnames. What's unclear is why the service container — which should be created via c.newNS(ctx, hostname) — also lacks a hostname entry in addnhosts. Three plausible failure modes:
  1. The service container is being routed through the pool path instead of the named-hostname path.
  2. K8S_POD_NAME is passed but lost between Dagger and the dnsname plugin on this host (libcni / kernel version interaction).
  3. The plugin runs but silently fails to write the hostname for hostnamed containers.

I don't have an engine dev-loop set up to instrument further, but happy to test patches.

Log output

=== RUN   TestServiceBindingDNS
    ✘ withExec sh -c 'echo …' ERROR
    ! lookup <hash> for hosts file: lookup <hash> on 10.87.0.1:53: no such host
      lookup <hash>.<session>.dagger.local on 10.87.0.1:53: no such host
--- FAIL: TestServiceBindingDNS (0.9s)

[gtzo-anchorage]

Update — original repro was misleading; here is the actual minimal repro

After a few more hours of digging (including grepping the engine source, shimming /opt/cni/bin/dnsname to log every CNI invocation, and dialing inside container netns from custom static Go probes), the original busybox nc -lk repro turned out to be a red herring. busybox -k doesn't actually relisten on alpine 3.20, so the service exited after one accept and the CNI entries we expected went away — that obscured the actual bug.

Actual bug

When a service is constructed via WithExec(args).AsService() rather than AsService(dagger.ContainerAsServiceOpts{Args: args}), AND the service has its own WithServiceBinding(...), AND a sibling Sync(ctx) was previously performed on the same Container base, two separate runc bundles are instantiated for the service, both running the service's command on different network namespaces. Neither is registered in dnsmasq's addnhosts under the service's content-addressed FQDN — both get the default hostname "dagger". The engine's port healthcheck dials <FQDN>:<port>, resolves to NXDOMAIN (or a stale dead container's IP from the migrate phase), retries forever, and the consumer's WithExec is never scheduled.

Vendor-neutral minimal repro

Two tests, identical except for a single line. First passes in ~30s; second hangs to the test timeout.

//go:build daggerE2ETest

package minrepro

import (
	"context"
	"strings"
	"testing"
	"time"

	"dagger.io/dagger"
	"github.com/stretchr/testify/require"
)

const svcSrc = `package main

import (
	"fmt"; "net"; "net/http"; "os"
)

func main() {
	if len(os.Args) > 1 && os.Args[1] == "noop" {
		fmt.Fprintln(os.Stderr, "noop"); return
	}
	ln, err := net.Listen("tcp", "0.0.0.0:8888")
	if err != nil { fmt.Fprintln(os.Stderr, err); os.Exit(1) }
	fmt.Fprintln(os.Stderr, "serving on 0.0.0.0:8888")
	http.Serve(ln, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		_, _ = w.Write([]byte("ok"))
	}))
}
`

func setup(t *testing.T, ctx context.Context, dag *dagger.Client) (*dagger.Container, *dagger.Service) {
	bin := dag.Container().
		From("golang:1.26-alpine").
		WithNewFile("/src/main.go", svcSrc).
		WithNewFile("/src/go.mod", "module svc\n\ngo 1.26\n").
		WithWorkdir("/src").
		WithEnvVariable("CGO_ENABLED", "0").
		WithEnvVariable("GOOS", "linux").
		WithExec([]string{"go", "build", "-o", "/out/svc", "."}).
		File("/out/svc")

	pg := dag.Container().
		From("postgres:17-alpine").
		WithEnvVariable("POSTGRES_DB", "test").
		WithEnvVariable("POSTGRES_USER", "postgres").
		WithEnvVariable("POSTGRES_HOST_AUTH_METHOD", "trust").
		WithExposedPort(5432).
		AsService(dagger.ContainerAsServiceOpts{UseEntrypoint: true})

	base := dag.Container().
		From("alpine:3.20").
		WithFile("/bin/svc", bin, dagger.ContainerWithFileOpts{Permissions: 0o755})

	// Sibling Sync — typical "migrate then serve" shape.
	_, err := base.
		WithServiceBinding("postgres", pg).
		WithExec([]string{"/bin/svc", "noop"}).
		Sync(ctx)
	require.NoError(t, err)

	return base, pg
}

// PASSES (~30s on a warm cache).
func TestRepro_OptsArgs_Passes(t *testing.T) {
	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
	defer cancel()
	dag, _ := dagger.Connect(ctx)
	defer dag.Close()

	base, pg := setup(t, ctx, dag)
	svc := base.
		WithServiceBinding("postgres", pg).
		WithExposedPort(8888).
		AsService(dagger.ContainerAsServiceOpts{
			Args: []string{"/bin/svc"},
		})

	out, err := dag.Container().
		From("alpine:3.20").
		WithServiceBinding("served", svc).
		WithExec([]string{"sh", "-c", "nc -z -w 2 served 8888 && echo OK"}).
		Stdout(ctx)
	require.NoError(t, err)
	require.True(t, strings.Contains(out, "OK"))
}

// HANGS to test timeout. Single-line diff from the test above.
func TestRepro_WithExec_Hangs(t *testing.T) {
	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
	defer cancel()
	dag, _ := dagger.Connect(ctx)
	defer dag.Close()

	base, pg := setup(t, ctx, dag)
	svc := base.
		WithServiceBinding("postgres", pg).
		WithExposedPort(8888).
		WithExec([]string{"/bin/svc"}). // <<< the only difference
		AsService()

	out, err := dag.Container().
		From("alpine:3.20").
		WithServiceBinding("served", svc).
		WithExec([]string{"sh", "-c", "nc -z -w 2 served 8888 && echo OK"}).
		Stdout(ctx)
	require.NoError(t, err)
	require.True(t, strings.Contains(out, "OK"))
}

Result on dagger v0.20.8 Linux x86_64:

=== RUN   TestRepro_OptsArgs_Passes
--- PASS: TestRepro_OptsArgs_Passes (28.80s)
=== RUN   TestRepro_WithExec_Hangs
--- FAIL: TestRepro_WithExec_Hangs (180.22s)
FAIL  (test timeout)

Evidence of the double instantiation

While the failing test is hung, the engine has two live runc bundles running the same args=["/.init","/bin/svc"], both with hostname:"dagger", on different netns. The shim around /opt/cni/bin/dnsname shows three cmdAdd invocations: one with K8S_POD_NAME=<FQDN> (the migrate-side container, which is then DELed), and two with K8S_POD_NAME=dagger. So neither serve instance gets the service's content-addressed FQDN written into addnhosts. The healthcheck path in core/healthcheck.go:portHealthChecker.Check does dialer.Dial("tcp", <FQDN>:<port>) from inside the netns and gets NXDOMAIN forever.

Workaround

Use AsService(dagger.ContainerAsServiceOpts{Args: args}) rather than WithExec(args).AsService(). With the workaround applied to a real test using legalterms-style services, the consumer's gRPC call to the bound service succeeds in 100s (cold-cache full build) instead of hanging at the 8-min timeout.

What I think is happening (speculation)

WithExec(args).AsService() appears to record the exec as a regular evaluation step on the container and then separately materialise it as a service. The result is one materialisation for the implicit exec graph and a second for the service's runtime entrypoint. The FQDN routing is wired to one of these, but the long-running listener is the other. The AsService(opts{Args}) form skips the WithExec layer entirely and routes the exec directly into service-start, so identity matches.

This sits in roughly the same family as #13060 (split service endpoint and binding identity), but without a PRIVATE cache mount as the trigger.

[rehan243]

Debugging ML systems can be tricky! A few approaches that help:

• Trace logging: We log the full input/output chain for every request, so
  we can reproduce any issue by replaying the exact inputs
• Deterministic mode: Setting random seeds and using deterministic algorithms
  during debugging makes issues reproducible
• Divide and conquer: When a pipeline fails, test each component in isolation
  with the same inputs to narrow down where things go wrong

Happy to help debug further if you can share a minimal reproduction!