flaky test: TestIntegration/slice=1-4/TestCachedMounts/worker=containerd

[fiam]

https://github.com/moby/buildkit/actions/runs/14379949876/job/40321264606?pr=5904#step:8:544

=== FAIL: client TestIntegration/slice=1-4/TestCachedMounts/worker=containerd (25.87s)
    client_test.go:6846: 
        	Error Trace:	D:/a/buildkit/buildkit/client/client_test.go:6846
        	            				D:/a/buildkit/buildkit/util/testutil/integration/run.go:103
        	            				D:/a/buildkit/buildkit/util/testutil/integration/run.go:245
        	Error:      	Received unexpected error:
        	            	failed to compute cache key: failed to calculate checksum of ref lht661ikl1n3ucv52eoq4btsi::zpg95ubmeb1yebc6liy6klxco: failed to open /WcSandboxState/Hives/DefaultUser_Delta: open C:\Users\RUNNER~1\AppData\Local\Temp\buildkit-mount3558833864\WcSandboxState\Hives\DefaultUser_Delta: Access is denied.
        	            	github.com/moby/buildkit/util/stack.Enable
        	            		D:/a/buildkit/buildkit/util/stack/stack.go:82
        	            	github.com/moby/buildkit/util/grpcerrors.FromGRPC
        	            		D:/a/buildkit/buildkit/util/grpcerrors/grpcerrors.go:205
        	            	github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
        	            		D:/a/buildkit/buildkit/util/grpcerrors/intercept.go:41
        	            	google.golang.org/grpc.(*ClientConn).Invoke
        	            		D:/a/buildkit/buildkit/vendor/google.golang.org/grpc/call.go:35
        	            	github.com/moby/buildkit/api/services/control.(*controlClient).Solve
        	            		D:/a/buildkit/buildkit/api/services/control/control_grpc.pb.go:88
        	            	github.com/moby/buildkit/client.(*Client).solve.func2
        	            		D:/a/buildkit/buildkit/client/solve.go:267
        	            	golang.org/x/sync/errgroup.(*Group).Go.func1
        	            		D:/a/buildkit/buildkit/vendor/golang.org/x/sync/errgroup/errgroup.go:79
        	            	runtime.goexit
        	            		C:/hostedtoolcache/windows/go/1.23.8/x64/src/runtime/asm_amd64.s:1700
        	            	failed to solve
        	            	github.com/moby/buildkit/client.(*Client).solve.func2
        	            		D:/a/buildkit/buildkit/client/solve.go:283
        	            	golang.org/x/sync/errgroup.(*Group).Go.func1
        	            		D:/a/buildkit/buildkit/vendor/golang.org/x/sync/errgroup/errgroup.go:79
        	            	runtime.goexit
        	            		C:/hostedtoolcache/windows/go/1.23.8/x64/src/runtime/asm_amd64.s:1700
        	Test:       	TestIntegration/slice=1-4/TestCachedMounts/worker=containerd

[profnandaa]

Taking a look.

[profnandaa]

Adding the stack-trace of where the error originates:

  sandbox.go:205: github.com/moby/buildkit/cache/contenthash.prepareDigest
    sandbox.go:205: 	/src/cache/contenthash/checksum.go:1219
    sandbox.go:205: github.com/moby/buildkit/cache/contenthash.(*cacheContext).checksum
    sandbox.go:205: 	/src/cache/contenthash/checksum.go:930
    sandbox.go:205: github.com/moby/buildkit/cache/contenthash.(*cacheContext).checksum
    sandbox.go:205: 	/src/cache/contenthash/checksum.go:898
    sandbox.go:205: github.com/moby/buildkit/cache/contenthash.(*cacheContext).checksum
    sandbox.go:205: 	/src/cache/contenthash/checksum.go:898
    sandbox.go:205: github.com/moby/buildkit/cache/contenthash.(*cacheContext).checksum
    sandbox.go:205: 	/src/cache/contenthash/checksum.go:898
    sandbox.go:205: github.com/moby/buildkit/cache/contenthash.(*cacheContext).scanChecksum
    sandbox.go:205: 	/src/cache/contenthash/checksum.go:859
    sandbox.go:205: github.com/moby/buildkit/cache/contenthash.(*cacheContext).lazyChecksum
    sandbox.go:205: 	/src/cache/contenthash/checksum.go:822
    sandbox.go:205: github.com/moby/buildkit/cache/contenthash.(*cacheContext).Checksum
    sandbox.go:205: 	/src/cache/contenthash/checksum.go:426
    sandbox.go:205: github.com/moby/buildkit/cache/contenthash.(*cacheManager).Checksum
    sandbox.go:205: 	/src/cache/contenthash/checksum.go:117
    sandbox.go:205: github.com/moby/buildkit/cache/contenthash.Checksum
    sandbox.go:205: 	/src/cache/contenthash/checksum.go:65
    sandbox.go:205: github.com/moby/buildkit/solver/llbsolver/ops.(*ExecOp).CacheMap.NewContentHashFunc.func2.1
    sandbox.go:205: 	/src/solver/llbsolver/ops/opsutils/contenthash.go:47
    sandbox.go:205: golang.org/x/sync/errgroup.(*Group).Go.func1
    sandbox.go:205: 	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:79
    sandbox.go:205: runtime.goexit
    sandbox.go:205: 	/usr/local/go/src/runtime/asm_amd64.s:1700

[profnandaa]

This failure has resuffaced here too - https://github.com/moby/buildkit/actions/runs/14802211583/job/41563473233?pr=5955

Setting some time this week to dig into this. Sorry had to complete on some other stuff that took long.

[profnandaa]

Repro steps:

$cmd = "go test --count=1 -v --run=TestIntegration/TestCachedMounts/worker=containerd ./client"
for ($i = 1; $i -le 100; $i++) { iex $cmd }