🐞 Dagger "close dagger session" fails when using _EXPERIMENTAL_DAGGER_CACHE_CONFIG

[cbochula]

What is the issue?

I have a simple module which builds an image and publishes it to ECR. I've also configured it to export the layer cache using _EXPERIMENTAL_DAGGER_CACHE_CONFIG. However, with this configured, it appears that the cache exporter exceeds the timeout window, causing the call to fail with:

ERR cleanup failed msg="close dagger session" err="shutdown: do shutdown: Post \"http://dagger/shutdown\": context deadline exceeded" duration=41.927581741s
Error: cleanup failed: "close dagger session": shutdown: do shutdown: Post "http://dagger/shutdown": context deadline exceeded

However, the engine logs indicate that the cache export succeeded:

time="2025-10-29T21:56:10Z" level=debug msg="finalized exporter" client_hostname=ip-192-168-129-205.ec2.internal client_id=ok6hevirad0rna055o310r3z8 session_id=4va5gib23c6dq54nb6xcf16jv spanID=aeaeb215ffcdec27 traceID=7e41954a60b711f98df1bcfe63d287c1
time="2025-10-29T21:56:10Z" level=debug msg="waited for cache export" client_hostname=ip-192-168-129-205.ec2.internal client_id=ok6hevirad0rna055o310r3z8 session_id=4va5gib23c6dq54nb6xcf16jv spanID=aeaeb215ffcdec27 traceID=7e41954a60b711f98df1bcfe63d287c1
time="2025-10-29T21:56:10Z" level=debug msg="done running cache export for client ok6hevirad0rna055o310r3z8" client_hostname=ip-192-168-129-205.ec2.internal client_id=ok6hevirad0rna055o310r3z8 session_id=4va5gib23c6dq54nb6xcf16jv spanID=aeaeb215ffcdec27 traceID=7e41954a60b711f98df1bcfe63d287c1

Note: I recently upgraded from 0.18.16, where this timeout did not occur and a long cache export did not cause a shutdown failure.

Dagger version

dagger v0.19.3 (image://registry.dagger.io/engine:v0.19.3) linux/amd64

Steps to reproduce

No response

Log output

No response

[cbochs]

I was just trying to understand what is going on here. It seems there is a fixed 10s deadline on the client:

func (c *Client) shutdownServer() error {
	// don't immediately cancel shutdown if we're shutting down because we were
	// canceled
	ctx := context.WithoutCancel(c.internalCtx)

	ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
	defer cancel()
...

And the /shutdown endpoint does wait until the cache export finishes: engine/server/session.go#L1205. Would a configuration option (client-side or engine-side) be considered for tuning tuning this timeout? Or, could the cache export run on the engine w/o it blocking client shutdown?

[marcosnils]

@cbochs I recall coming across this in the past and IIRC it was mostly a context race at the very end of the session shutdown which wasn't waiting for the cache exporter to finish. I might have some time tomorrow to take a look at this 🙏

[cbochs]

@marcosnils just checking in to see if there's any update on this?

[josephgoksuobrizum]

It would be great to know whether _EXPERIMENTAL_DAGGER_CACHE_CONFIG will be fully supported anytime soon.

cc @shykes @marcosnils