Skip to content

session(auth): cancel blocked credential callbacks#6641

Merged
crazy-max merged 1 commit intomoby:masterfrom
crazy-max:session-auth-cancel
Apr 2, 2026
Merged

session(auth): cancel blocked credential callbacks#6641
crazy-max merged 1 commit intomoby:masterfrom
crazy-max:session-auth-cancel

Conversation

@crazy-max
Copy link
Copy Markdown
Member

@crazy-max crazy-max commented Apr 1, 2026 •
edited
Loading

Narrow fix for one concrete problem discussed in #6633. It's related to the broader auth deadlock discussion and the patch stack in #6630, but it only addresses the credential callback cancellation bug rather than the larger concurrency changes proposed there. It's also relevant to the repro effort in #6638, because it adds integration coverage for a real session auth cancellation path without taking the broader rewrite.

Tested integration test without fix and failed as expected: https://github.com/moby/buildkit/actions/runs/23849960618/job/69528134201?pr=6641#step:8:2972

=== Failed
=== FAIL: client TestClientGatewayIntegration/TestClientGatewayCanceledCredentialsCallbackReturns/worker=oci (300.73s)
    sandbox.go:147: sandbox timeout reached, stopping worker
    sandbox.go:187: buildkitd debug logs:
        goroutine 230 [running]:
        runtime/pprof.writeGoroutineStacks({0x251f020, 0x14735d1203c0})
        	/usr/local/go/src/runtime/pprof/pprof.go:819 +0x6b
        runtime/pprof.writeGoroutine({0x251f020?, 0x14735d1203c0?}, 0x14735d5b07b8?)
        	/usr/local/go/src/runtime/pprof/pprof.go:782 +0x25
        runtime/pprof.(*Profile).WriteTo(0x39e5d70?, {0x251f020?, 0x14735d1203c0?}, 0xc?)
        	/usr/local/go/src/runtime/pprof/pprof.go:408 +0x149
        net/http/pprof.handler.ServeHTTP({0x14735d526131, 0x9}, {0x253d4a0, 0x14735d1203c0}, 0x14735d06a640)
        	/usr/local/go/src/net/http/pprof/pprof.go:273 +0x52a
        net/http/pprof.Index({0x253d4a0, 0x14735d1203c0}, 0x14735d06a640?)
        	/usr/local/go/src/net/http/pprof/pprof.go:397 +0xda
        net/http.HandlerFunc.ServeHTTP(0x14735d3603c0?, {0x253d4a0?, 0x14735d1203c0?}, 0x767256?)
        	/usr/local/go/src/net/http/server.go:2286 +0x29
        net/http.(*ServeMux).ServeHTTP(0x48af59?, {0x253d4a0, 0x14735d1203c0}, 0x14735d06a640)
        	/usr/local/go/src/net/http/server.go:2828 +0x1c7
        net/http.serverHandler.ServeHTTP({0x14735d1d8e80?}, {0x253d4a0?, 0x14735d1203c0?}, 0x1?)
        	/usr/local/go/src/net/http/server.go:3311 +0x8e
        net/http.(*conn).serve(0x14735d3b4000, {0x2542998, 0x14735d0e0fc0})
        	/usr/local/go/src/net/http/server.go:2073 +0x650
        created by net/http.(*Server).Serve in goroutine 15
        	/usr/local/go/src/net/http/server.go:3464 +0x485
        
        goroutine 1 [select, 4 minutes, locked to thread]:
        main.main.func3(0x14735d3fc2c0)
        	/src/cmd/buildkitd/main.go:401 +0x17cf
        github.com/urfave/cli.HandleAction({0x1fa9360?, 0x14735d1886e0?}, 0x14735d0ae540?)
        	/src/vendor/github.com/urfave/cli/app.go:524 +0x9f
        github.com/urfave/cli.(*App).Run(0x14735d0ae540, {0x14735cf54000, 0xb, 0xb})
        	/src/vendor/github.com/urfave/cli/app.go:286 +0x77e
        main.main()
        	/src/cmd/buildkitd/main.go:431 +0x10b0
        
        goroutine 33 [select]:
        github.com/moby/buildkit/util/tracing/detect.(*TraceRecorder).gcLoop(0x14735d104150, {0x25429d0, 0x14735d2d60a0})
        	/src/util/tracing/detect/recorder.go:108 +0x186
        created by github.com/moby/buildkit/util/tracing/detect.NewTraceRecorder in goroutine 1
        	/src/util/tracing/detect/recorder.go:49 +0x17c
        
        goroutine 10 [syscall, 5 minutes]:
        os/signal.signal_recv()
        	/usr/local/go/src/runtime/sigqueue.go:152 +0x98
        os/signal.loop()
        	/usr/local/go/src/os/signal/signal_unix.go:23 +0x13
        created by os/signal.Notify.func1.1 in goroutine 1
        	/usr/local/go/src/os/signal/signal.go:152 +0x1f
        
        goroutine 11 [chan receive, 5 minutes]:
        github.com/moby/buildkit/util/appcontext.Context.func1.1()
        	/src/util/appcontext/appcontext.go:36 +0x65
        created by github.com/moby/buildkit/util/appcontext.Context.func1 in goroutine 1
        	/src/util/appcontext/appcontext.go:34 +0x194
        
        goroutine 15 [IO wait]:
        internal/poll.runtime_pollWait(0x7fac63036e00, 0x72)
        	/usr/local/go/src/runtime/netpoll.go:351 +0x85
        internal/poll.(*pollDesc).wait(0x14735d1f8700?, 0x20?, 0x0)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
        internal/poll.(*FD).Accept(0x14735d1f8700)
        	/usr/local/go/src/internal/poll/fd_unix.go:613 +0x28c
        net.(*netFD).accept(0x14735d1f8700)
        	/usr/local/go/src/net/fd_unix.go:150 +0x29
        net.(*UnixListener).accept(0x14735d507e48?)
        	/usr/local/go/src/net/unixsock_posix.go:172 +0x16
        net.(*UnixListener).Accept(0x14735d2cd740)
        	/usr/local/go/src/net/unixsock.go:267 +0x30
        net/http.(*Server).Serve(0x14735ce1a800, {0x253d9e0, 0x14735d2cd740})
        	/usr/local/go/src/net/http/server.go:3434 +0x30c
        main.setupDebugHandlers.func3()
        	/src/cmd/buildkitd/debug.go:76 +0x29
        created by main.setupDebugHandlers in goroutine 1
        	/src/cmd/buildkitd/debug.go:75 +0x510
        
        goroutine 171 [chan receive, 4 minutes]:
        github.com/moby/buildkit/session.(*Manager).handleConn(0x14735cf459b0, {0x25429d0?, 0x14735d08d810?}, {0x2555318, 0x14735d2d5340}, 0x14735d676990)
        	/src/session/manager.go:141 +0x469
        github.com/moby/buildkit/session.(*Manager).HandleConn(0x2542998?, {0x25429d0?, 0x14735d08d810?}, {0x2555318?, 0x14735d2d5340?}, 0x1?)
        	/src/session/manager.go:96 +0xb5
        github.com/moby/buildkit/control.(*Controller).Session(0x14735cf2c300, {0x2555148, 0x14735d54cf50})
        	/src/control/control.go:599 +0x1bd
        github.com/moby/buildkit/api/services/control._Control_Session_Handler({0x2379a40?, 0x14735cf2c300}, {0x254f118, 0x14735d429000})
        	/src/api/services/control/control_grpc.pb.go:304 +0xfb
        github.com/moby/buildkit/util/grpcerrors.StreamServerInterceptor({0x2379a40, 0x14735cf2c300}, {0x254f118, 0x14735d429000}, 0x1ef13a0?, 0x2509630)
        	/src/util/grpcerrors/intercept.go:33 +0x86
        google.golang.org/grpc.(*Server).processStreamingRPC(0x14735cedefc8, {0x2542998, 0x14735d6767e0}, 0x14735d646b60, 0x14735ced1080, 0x3a134c0, 0x0)
        	/src/vendor/google.golang.org/grpc/server.go:1722 +0x1151
        google.golang.org/grpc.(*Server).handleStream(0x14735cedefc8, {0x2544158, 0x14735d093860}, 0x14735d646b60)
        	/src/vendor/google.golang.org/grpc/server.go:1859 +0xb85
        google.golang.org/grpc.(*Server).serveStreams.func2.1()
        	/src/vendor/google.golang.org/grpc/server.go:1064 +0x7f
        created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 81
        	/src/vendor/google.golang.org/grpc/server.go:1075 +0x11d
        
        goroutine 49 [IO wait, 5 minutes]:
        internal/poll.runtime_pollWait(0x7fac63036c00, 0x72)
        	/usr/local/go/src/runtime/netpoll.go:351 +0x85
        internal/poll.(*pollDesc).wait(0x14735d1f8b80?, 0x909825?, 0x0)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
        internal/poll.(*FD).Accept(0x14735d1f8b80)
        	/usr/local/go/src/internal/poll/fd_unix.go:613 +0x28c
        net.(*netFD).accept(0x14735d1f8b80)
        	/usr/local/go/src/net/fd_unix.go:150 +0x29
        net.(*UnixListener).accept(0x14735cf696c8?)
        	/usr/local/go/src/net/unixsock_posix.go:172 +0x16
        net.(*UnixListener).Accept(0x14735ce4c840)
        	/usr/local/go/src/net/unixsock.go:267 +0x30
        google.golang.org/grpc.(*Server).Serve(0x14735cedf208, {0x253d9e0, 0x14735ce4c840})
        	/src/vendor/google.golang.org/grpc/server.go:920 +0x45c
        created by main.runTraceController in goroutine 1
        	/src/cmd/buildkitd/main.go:1024 +0x135
        
        goroutine 50 [IO wait, 5 minutes]:
        internal/poll.runtime_pollWait(0x7fac63036a00, 0x72)
        	/usr/local/go/src/runtime/netpoll.go:351 +0x85
        internal/poll.(*pollDesc).wait(0x14735d4b6660?, 0x14735d58fe53?, 0x1)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
        internal/poll.(*FD).Read(0x14735d4b6660, {0x14735d58fe53, 0x10000, 0x10000})
        	/usr/local/go/src/internal/poll/fd_unix.go:165 +0x2ae
        os.(*File).read(...)
        	/usr/local/go/src/os/file_posix.go:29
        os.(*File).Read(0x14735d3622b8, {0x14735d58fe53?, 0x0?, 0x0?})
        	/usr/local/go/src/os/file.go:144 +0x4f
        github.com/fsnotify/fsnotify.(*inotify).readEvents(0x14735d28e280)
        	/src/vendor/github.com/fsnotify/fsnotify/backend_inotify.go:357 +0xcf
        created by github.com/fsnotify/fsnotify.newBackend in goroutine 1
        	/src/vendor/github.com/fsnotify/fsnotify/backend_inotify.go:155 +0x1f6
        
        goroutine 51 [select, 5 minutes]:
        tags.cncf.io/container-device-interface/pkg/cdi.(*watch).watch(0x14735d2959b0, 0x14735ce8b1a0, 0x14735cf958b0, 0x14735d295d00, 0x14735ce4cf90)
        	/src/vendor/tags.cncf.io/container-device-interface/pkg/cdi/cache.go:541 +0xbb
        created by tags.cncf.io/container-device-interface/pkg/cdi.(*watch).start in goroutine 1
        	/src/vendor/tags.cncf.io/container-device-interface/pkg/cdi/cache.go:514 +0xb2
        
        goroutine 170 [sync.WaitGroup.Wait, 4 minutes]:
        sync.runtime_SemacquireWaitGroup(0x14735ce59890?, 0xa0?)
        	/usr/local/go/src/runtime/sema.go:114 +0x2e
        sync.(*WaitGroup).Wait(0x14735d059508)
        	/usr/local/go/src/sync/waitgroup.go:206 +0x85
        golang.org/x/sync/errgroup.(*Group).Wait(0x14735d059500)
        	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x1e
        github.com/moby/buildkit/control.(*Controller).Status(0x14735cf2c300, 0x14735d058580, {0x2552980, 0x14735d54ce10})
        	/src/control/control.go:584 +0x1f9
        github.com/moby/buildkit/api/services/control._Control_Status_Handler({0x2379a40, 0x14735cf2c300}, {0x254f118, 0x14735d428e00})
        	/src/api/services/control/control_grpc.pb.go:297 +0x145
        github.com/moby/buildkit/util/grpcerrors.StreamServerInterceptor({0x2379a40, 0x14735cf2c300}, {0x254f118, 0x14735d428e00}, 0x1ef13a0?, 0x2509640)
        	/src/util/grpcerrors/intercept.go:33 +0x86
        google.golang.org/grpc.(*Server).processStreamingRPC(0x14735cedefc8, {0x2542998, 0x14735d6764e0}, 0x14735d6469c0, 0x14735ced1080, 0x3a134a0, 0x0)
        	/src/vendor/google.golang.org/grpc/server.go:1722 +0x1151
        google.golang.org/grpc.(*Server).handleStream(0x14735cedefc8, {0x2544158, 0x14735d093860}, 0x14735d6469c0)
        	/src/vendor/google.golang.org/grpc/server.go:1859 +0xb85
        google.golang.org/grpc.(*Server).serveStreams.func2.1()
        	/src/vendor/google.golang.org/grpc/server.go:1064 +0x7f
        created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 81
        	/src/vendor/google.golang.org/grpc/server.go:1075 +0x11d
        
        goroutine 65 [IO wait, 4 minutes]:
        internal/poll.runtime_pollWait(0x7fac63037000, 0x72)
        	/usr/local/go/src/runtime/netpoll.go:351 +0x85
        internal/poll.(*pollDesc).wait(0x14735d1f8a00?, 0x20?, 0x0)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
        internal/poll.(*FD).Accept(0x14735d1f8a00)
        	/usr/local/go/src/internal/poll/fd_unix.go:613 +0x28c
        net.(*netFD).accept(0x14735d1f8a00)
        	/usr/local/go/src/net/fd_unix.go:150 +0x29
        net.(*UnixListener).accept(0x460f60?)
        	/usr/local/go/src/net/unixsock_posix.go:172 +0x16
        net.(*UnixListener).Accept(0x14735d28cb40)
        	/usr/local/go/src/net/unixsock.go:267 +0x30
        google.golang.org/grpc.(*Server).Serve(0x14735cedefc8, {0x253d9e0, 0x14735d28cb40})
        	/src/vendor/google.golang.org/grpc/server.go:920 +0x45c
        main.serveGRPC.serveGRPC.func1.func3()
        	/src/cmd/buildkitd/main.go:480 +0xd9
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:93 +0x50
        created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
        	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x95
        
        goroutine 282 [chan receive, 4 minutes]:
        github.com/moby/buildkit/util/progress.pipe.func1()
        	/src/util/progress/progress.go:197 +0x3a
        created by github.com/moby/buildkit/util/progress.pipe in goroutine 281
        	/src/util/progress/progress.go:196 +0x175
        
        goroutine 224 [chan receive, 4 minutes]:
        github.com/moby/buildkit/util/flightcontrol.(*call[...]).wait(0x2558280, {0x2542998, 0x14735d6148a0})
        	/src/util/flightcontrol/flightcontrol.go:173 +0x645
        github.com/moby/buildkit/util/flightcontrol.(*Group[...]).do(0x2558300, {0x2542998, 0x14735d6148a0}, {0x14735ce7dae0, 0x4d}, 0x14735d614b10)
        	/src/util/flightcontrol/flightcontrol.go:79 +0x293
        github.com/moby/buildkit/util/flightcontrol.(*Group[...]).Do(0x2558300, {0x2542998, 0x14735d6148a0}, {0x14735ce7dae0, 0x4d}, 0x14735d614b10)
        	/src/util/flightcontrol/flightcontrol.go:37 +0xd0
        github.com/moby/buildkit/source/containerimage.(*Source).ResolveImageMetadata(0x14735d413170, {0x2542998, 0x14735d6146f0}, 0x20ea4c0?, 0x14735d4e4880, 0x14735cf459b0, {0x251e320, 0x14735d4ef920})
        	/src/source/containerimage/source.go:189 +0x61c
        github.com/moby/buildkit/worker/base.(*Worker).ResolveSourceMetadata(0x14735cefd200, {0x2542998, 0x14735d6146f0}, 0x14735d4e4740, {{0x14735d443650, 0x6e}, {0x0, 0x0, 0x0}, 0x14735d4e4880, ...}, ...)
        	/src/worker/base/worker.go:446 +0x3bc
        github.com/moby/buildkit/solver/llbsolver.(*llbBridge).resolveSourceMetadata.func1({0x2542998?, 0x14735d6146f0?}, {0x253d660?, 0x14735d098870?})
        	/src/solver/llbsolver/bridge.go:275 +0xc2
        github.com/moby/buildkit/solver/llbsolver.inBuilderContext.func1({0x2542998, 0x14735d614630}, {0x253d660, 0x14735d098870})
        	/src/solver/llbsolver/solver.go:449 +0x1f0
        github.com/moby/buildkit/solver.(*Job).InContext(0x14735d098870, {0x2542998, 0x14735d614300}, 0x14735d4ef8c0)
        	/src/solver/jobs.go:869 +0xe3
        github.com/moby/buildkit/solver/llbsolver.inBuilderContext({0x2542998, 0x14735d614300}, {0x253dbc0, 0x14735d098870}, {0x14735d443650, 0x6e}, {0x14735d157020?, 0x0?}, 0x14735d645480)
        	/src/solver/llbsolver/solver.go:445 +0x1d8
        github.com/moby/buildkit/solver/llbsolver.(*llbBridge).resolveSourceMetadata(0x14735d2da480, {0x2542998, 0x14735d614300}, 0x14735d4e4740, {{0x14735d443650, 0x6e}, {0x0, 0x0, 0x0}, 0x14735d4e4880, ...}, ...)
        	/src/solver/llbsolver/bridge.go:274 +0xaea
        github.com/moby/buildkit/solver/llbsolver.(*llbBridge).ResolveSourceMetadata(...)
        	/src/solver/llbsolver/bridge.go:216
        github.com/moby/buildkit/solver/llbsolver.(*provenanceBridge).ResolveSourceMetadata(0x14735d47cae0, {0x2542998?, 0x14735d614300?}, 0x14735d4ef878?, {{0x0, 0x0}, {0x0, 0x0, 0x0}, 0x14735d4e4880, ...})
        	/src/solver/llbsolver/provenance.go:141 +0x105
        github.com/moby/buildkit/frontend/gateway.(*llbBridgeForwarder).ResolveSourceMeta(0x14735d692dd0, {0x2542998?, 0x14735d614300?}, 0x14735d626000)
        	/src/frontend/gateway/gateway.go:613 +0x674
        github.com/moby/buildkit/control/gateway.(*GatewayForwarder).ResolveSourceMeta(0x39efeb0?, {0x2542998, 0x14735d614300}, 0x14735d626000)
        	/src/control/gateway/gateway.go:105 +0xa6
        github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_ResolveSourceMeta_Handler.func1({0x2542998?, 0x14735d614300?}, {0x23ce6a0?, 0x14735d626000?})
        	/src/frontend/gateway/pb/gateway_grpc.pb.go:397 +0xda
        github.com/moby/buildkit/util/grpcerrors.UnaryServerInterceptor({0x2542998, 0x14735d614300}, {0x23ce6a0?, 0x14735d626000?}, 0x14735d4b6060?, 0x14735d4937a8?)
        	/src/util/grpcerrors/intercept.go:14 +0x69
        google.golang.org/grpc.getChainUnaryHandler.func1({0x2542998?, 0x14735d614300?}, {0x23ce6a0?, 0x14735d626000?})
        	/src/vendor/google.golang.org/grpc/server.go:1242 +0x153
        main.unaryInterceptor({0x2542998, 0x14735d614300}, {0x23ce6a0?, 0x14735d626000?}, 0x14735d61af60, 0x14735d4ef830?)
        	/src/cmd/buildkitd/main.go:734 +0xed
        google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x2542998, 0x14735d614300}, {0x23ce6a0, 0x14735d626000}, 0x14735d61af60, 0x78?)
        	/src/vendor/google.golang.org/grpc/server.go:1233 +0x7c
        github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_ResolveSourceMeta_Handler({0x23c88a0, 0x14735d283140}, {0x2542998, 0x14735d614300}, 0x14735d645400, 0x14735ce8ae20)
        	/src/frontend/gateway/pb/gateway_grpc.pb.go:399 +0x192
        google.golang.org/grpc.(*Server).processUnaryRPC(0x14735cedefc8, {0x2542998, 0x14735d6141e0}, 0x14735d35eb60, 0x14735ced1680, 0x3a17798, 0x0)
        	/src/vendor/google.golang.org/grpc/server.go:1429 +0x11d2
        google.golang.org/grpc.(*Server).handleStream(0x14735cedefc8, {0x2544158, 0x14735d093860}, 0x14735d35eb60)
        	/src/vendor/google.golang.org/grpc/server.go:1855 +0xbc6
        google.golang.org/grpc.(*Server).serveStreams.func2.1()
        	/src/vendor/google.golang.org/grpc/server.go:1064 +0x7f
        created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 81
        	/src/vendor/google.golang.org/grpc/server.go:1075 +0x11d
        
        goroutine 275 [select, 4 minutes]:
        github.com/moby/buildkit/util/progress.(*progressReader).Read.func1()
        	/src/util/progress/progress.go:124 +0xc8
        created by github.com/moby/buildkit/util/progress.(*progressReader).Read in goroutine 259
        	/src/util/progress/progress.go:121 +0x145
        
        goroutine 26 [chan receive, 4 minutes]:
        github.com/moby/buildkit/solver.(*scheduler).loop.func2()
        	/src/solver/scheduler.go:67 +0x30
        created by github.com/moby/buildkit/solver.(*scheduler).loop in goroutine 60
        	/src/solver/scheduler.go:66 +0xb6
        
        goroutine 58 [sleep, 2 minutes]:
        time.Sleep(0x1bf08eb000)
        	/usr/local/go/src/runtime/time.go:363 +0x165
        github.com/moby/buildkit/solver/llbsolver/history.NewQueue.func2()
        	/src/solver/llbsolver/history/buildhistory.go:135 +0x49
        created by github.com/moby/buildkit/solver/llbsolver/history.NewQueue in goroutine 1
        	/src/solver/llbsolver/history/buildhistory.go:131 +0x6d9
        
        goroutine 59 [chan receive, 4 minutes]:
        github.com/moby/buildkit/solver/llbsolver/history.NewQueue.func3()
        	/src/solver/llbsolver/history/buildhistory.go:140 +0x3c
        created by github.com/moby/buildkit/solver/llbsolver/history.NewQueue in goroutine 1
        	/src/solver/llbsolver/history/buildhistory.go:139 +0x725
        
        goroutine 60 [sync.Cond.Wait, 4 minutes]:
        sync.runtime_notifyListWait(0x14735d159d50, 0x5)
        	/usr/local/go/src/runtime/sema.go:617 +0x1b3
        sync.(*Cond).Wait(0x14735d5141c0?)
        	/usr/local/go/src/sync/cond.go:71 +0x73
        github.com/moby/buildkit/util/cond.(*StatefulCond).Wait(0x14735d283560)
        	/src/util/cond/cond.go:28 +0x98
        github.com/moby/buildkit/solver.(*scheduler).loop(0x14735d136380)
        	/src/solver/scheduler.go:92 +0x265
        created by github.com/moby/buildkit/solver.newScheduler in goroutine 1
        	/src/solver/scheduler.go:26 +0x25f
        
        goroutine 63 [select]:
        google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x14735cf19640, 0x1)
        	/src/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:420 +0x10a
        google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x14735d2d0d20)
        	/src/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:597 +0x78
+ ecode=1
+ go tool covdata textfmt '-i=/testreports/helpers' '-o=/testreports/helpers-report.txt'
        google.golang.org/grpc/internal/transport.NewServerTransport.func3()
        	/src/vendor/google.golang.org/grpc/internal/transport/http2_server.go:342 +0xdc
        created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 62
        	/src/vendor/google.golang.org/grpc/internal/transport/http2_server.go:340 +0x18e7
        
        goroutine 64 [select, 4 minutes]:
        google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0x14735d093860)
        	/src/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1199 +0x1e5
        created by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 62
        	/src/vendor/google.golang.org/grpc/internal/transport/http2_server.go:363 +0x1929
        
        goroutine 81 [IO wait]:
        internal/poll.runtime_pollWait(0x7fac63036400, 0x72)
        	/usr/local/go/src/runtime/netpoll.go:351 +0x85
        internal/poll.(*pollDesc).wait(0x14735d2da580?, 0x14735d1ea000?, 0x0)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
        internal/poll.(*FD).Read(0x14735d2da580, {0x14735d1ea000, 0x8000, 0x8000})
        	/usr/local/go/src/internal/poll/fd_unix.go:165 +0x2ae
        net.(*netFD).Read(0x14735d2da580, {0x14735d1ea000?, 0x4?, 0x14735d44fc20?})
        	/usr/local/go/src/net/fd_posix.go:68 +0x25
        net.(*conn).Read(0x14735d2fc0f8, {0x14735d1ea000?, 0x48bd0c?, 0x4a43b8?})
        	/usr/local/go/src/net/net.go:196 +0x45
        bufio.(*Reader).Read(0x14735cf52c60, {0x14735d210664, 0x9, 0x4f4efa?})
        	/usr/local/go/src/bufio/bufio.go:245 +0x197
        io.ReadAtLeast({0x251eb60, 0x14735cf52c60}, {0x14735d210664, 0x9, 0x9}, 0x9)
        	/usr/local/go/src/io/io.go:335 +0x8e
        io.ReadFull(...)
        	/usr/local/go/src/io/io.go:354
        golang.org/x/net/http2.readFrameHeader({0x14735d210664, 0x9, 0x9b18b2?}, {0x251eb60?, 0x14735cf52c60?})
        	/src/vendor/golang.org/x/net/http2/frame.go:250 +0x65
        golang.org/x/net/http2.(*Framer).ReadFrameHeader(0x14735d210620)
        	/src/vendor/golang.org/x/net/http2/frame.go:513 +0x6b
        google.golang.org/grpc/internal/transport.(*framer).readFrame(0x14735d2da780)
        	/src/vendor/google.golang.org/grpc/internal/transport/http_util.go:480 +0x45
        google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0x14735d093860, {0x2542998, 0x14735d2837d0}, 0x14735d283860)
        	/src/vendor/google.golang.org/grpc/internal/transport/http2_server.go:638 +0xff
        google.golang.org/grpc.(*Server).serveStreams(0x14735cedefc8, {0x2542498?, 0x3a58e00?}, {0x2544158, 0x14735d093860}, {0x2554180?, 0x14735d2fc0f8?})
        	/src/vendor/google.golang.org/grpc/server.go:1058 +0x368
        google.golang.org/grpc.(*Server).handleRawConn.func1()
        	/src/vendor/google.golang.org/grpc/server.go:992 +0x56
        created by google.golang.org/grpc.(*Server).handleRawConn in goroutine 62
        	/src/vendor/google.golang.org/grpc/server.go:991 +0x1cb
        
        goroutine 66 [sync.WaitGroup.Wait, 4 minutes]:
        sync.runtime_SemacquireWaitGroup(0x0?, 0x0?)
        	/usr/local/go/src/runtime/sema.go:114 +0x2e
        sync.(*WaitGroup).Wait(0x14735d2de008)
        	/usr/local/go/src/sync/waitgroup.go:206 +0x85
        golang.org/x/sync/errgroup.(*Group).Wait(0x14735d2de000)
        	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x1e
        main.serveGRPC.func2()
        	/src/cmd/buildkitd/main.go:485 +0x2a
        created by main.serveGRPC in goroutine 1
        	/src/cmd/buildkitd/main.go:484 +0x290
        
        goroutine 249 [sync.Cond.Wait, 4 minutes]:
        sync.runtime_notifyListWait(0x14735d059690, 0x2)
        	/usr/local/go/src/runtime/sema.go:617 +0x1b3
        sync.(*Cond).Wait(0x14735d4d9db8?)
        	/usr/local/go/src/sync/cond.go:71 +0x73
        github.com/moby/buildkit/util/progress.(*progressReader).Read(0x14735d676f60, {0x25424d0, 0x3a58e00})
        	/src/util/progress/progress.go:158 +0x1a5
        github.com/moby/buildkit/util/progress.(*MultiReader).handle(0x14735d47ca80)
        	/src/util/progress/multireader.go:112 +0x6f
        created by github.com/moby/buildkit/util/progress.(*MultiReader).Reader in goroutine 175
        	/src/util/progress/multireader.go:103 +0x2d3
        
        goroutine 42 [select]:
        github.com/moby/buildkit/executor/resources.(*Sampler[...]).run(0x2559e00)
        	/src/executor/resources/sampler.go:92 +0xce
        created by github.com/moby/buildkit/executor/resources.(*Sampler[...]).Record in goroutine 35
        	/src/executor/resources/sampler.go:83 +0x167
        
        goroutine 252 [select]:
        github.com/moby/buildkit/session.monitorHealth({0x25429d0, 0x14735d6a20f0}, 0x14735d498008, 0x14735d54d900?)
        	/src/session/grpc.go:85 +0x2f9
        created by github.com/moby/buildkit/session.grpcClientConn in goroutine 171
        	/src/session/grpc.go:66 +0x979
        
        goroutine 172 [sync.WaitGroup.Wait, 4 minutes]:
        sync.runtime_SemacquireWaitGroup(0x14735ce2f4a0?, 0xa0?)
        	/usr/local/go/src/runtime/sema.go:114 +0x2e
        sync.(*WaitGroup).Wait(0x14735cf198c8)
        	/usr/local/go/src/sync/waitgroup.go:206 +0x85
        golang.org/x/sync/errgroup.(*Group).Wait(0x14735cf198c0)
        	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x1e
        github.com/moby/buildkit/solver/llbsolver.(*Solver).recordBuildHistory.func1({0x2542a08, 0x14735d6d0670}, 0x14735d6d0520, {0x3a58e00, 0x0, 0x425874?}, {0x0, 0x0})
        	/src/solver/llbsolver/history.go:240 +0xaa8
        github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func2()
        	/src/solver/llbsolver/solver.go:229 +0xc3
        github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve(0x14735d1362a0, {0x2542998, 0x14735d676ea0}, {_, _}, {_, _}, {0x0, 0x0, {0x0, ...}, ...}, ...)
        	/src/solver/llbsolver/solver.go:388 +0x27b7
        github.com/moby/buildkit/control.(*Controller).Solve(0x14735cf2c300, {0x2542998, 0x14735d676ea0}, 0x14735d098690)
        	/src/control/control.go:535 +0x1e65
        github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1({0x2542998?, 0x14735d676ea0?}, {0x23f0f00?, 0x14735d098690?})
        	/src/api/services/control/control_grpc.pb.go:287 +0xda
        github.com/moby/buildkit/util/grpcerrors.UnaryServerInterceptor({0x2542998, 0x14735d676ea0}, {0x23f0f00?, 0x14735d098690?}, 0x14735d4b6060?, 0x14735d1b77a8?)
        	/src/util/grpcerrors/intercept.go:14 +0x69
        google.golang.org/grpc.getChainUnaryHandler.func1({0x2542998?, 0x14735d676ea0?}, {0x23f0f00?, 0x14735d098690?})
        	/src/vendor/google.golang.org/grpc/server.go:1242 +0x153
        main.unaryInterceptor({0x2542998, 0x14735d676ea0}, {0x23f0f00?, 0x14735d098690?}, 0x14735ce8bb40, 0x14735ce59a40?)
        	/src/cmd/buildkitd/main.go:734 +0xed
        google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x2542998, 0x14735d676ea0}, {0x23f0f00, 0x14735d098690}, 0x14735ce8bb40, 0x78?)
        	/src/vendor/google.golang.org/grpc/server.go:1233 +0x7c
        github.com/moby/buildkit/api/services/control._Control_Solve_Handler({0x2379a40, 0x14735cf2c300}, {0x2542998, 0x14735d676ea0}, 0x14735d2da400, 0x14735ce8ae20)
        	/src/api/services/control/control_grpc.pb.go:289 +0x192
        google.golang.org/grpc.(*Server).processUnaryRPC(0x14735cedefc8, {0x2542998, 0x14735d676d80}, 0x14735d646d00, 0x14735ced1080, 0x3a13118, 0x0)
        	/src/vendor/google.golang.org/grpc/server.go:1429 +0x11d2
        google.golang.org/grpc.(*Server).handleStream(0x14735cedefc8, {0x2544158, 0x14735d093860}, 0x14735d646d00)
        	/src/vendor/google.golang.org/grpc/server.go:1855 +0xbc6
        google.golang.org/grpc.(*Server).serveStreams.func2.1()
        	/src/vendor/google.golang.org/grpc/server.go:1064 +0x7f
        created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 81
        	/src/vendor/google.golang.org/grpc/server.go:1075 +0x11d
        
        goroutine 247 [chan receive, 4 minutes]:
        github.com/moby/buildkit/util/progress.pipe.func1()
        	/src/util/progress/progress.go:197 +0x3a
        created by github.com/moby/buildkit/util/progress.pipe in goroutine 175
        	/src/util/progress/progress.go:196 +0x175
        
        goroutine 175 [sync.Cond.Wait, 4 minutes]:
        sync.runtime_notifyListWait(0x14735d059e10, 0x1)
        	/usr/local/go/src/runtime/sema.go:617 +0x1b3
        sync.(*Cond).Wait(0x3?)
        	/usr/local/go/src/sync/cond.go:71 +0x73
        github.com/moby/buildkit/util/progress.(*progressReader).Read(0x14735d677650, {0x25429d0, 0x14735d08d770})
        	/src/util/progress/progress.go:158 +0x1a5
        github.com/moby/buildkit/solver.(*Job).Status(0x14735d098870, {0x25429d0, 0x14735d08d770}, 0x14735d2d5030)
        	/src/solver/progress.go:28 +0x24b
        github.com/moby/buildkit/solver/llbsolver.(*Solver).Status(0x14735d1362a0, {0x25429d0, 0x14735d08d770}, {0x14735ce78240, 0x19}, 0x14735d2d5030)
        	/src/solver/llbsolver/solver.go:413 +0x11c
        github.com/moby/buildkit/control.(*Controller).Status.func1()
        	/src/control/control.go:562 +0x45
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:93 +0x50
        created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 170
        	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x95
        
        goroutine 255 [select]:
        google.golang.org/grpc/internal/transport.(*recvBufferReader).readMessageHeader(0x14735d646c28, {0x14735d429028, 0x5, 0x5})
        	/src/vendor/google.golang.org/grpc/internal/transport/transport.go:177 +0xa5
        google.golang.org/grpc/internal/transport.(*recvBufferReader).ReadMessageHeader(0x14735d646c28, {0x14735d429028?, 0x14735d093860?, 0x14735d646b60?})
        	/src/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0x77
        google.golang.org/grpc/internal/transport.(*transportReader).ReadMessageHeader(0x14735d646c08, {0x14735d429028?, 0x14735d646b60?, 0xdb8b2a?})
        	/src/vendor/google.golang.org/grpc/internal/transport/transport.go:457 +0x25
        google.golang.org/grpc/internal/transport.(*Stream).ReadMessageHeader(0x14735d646b60, {0x14735d429028, 0x5, 0x5})
        	/src/vendor/google.golang.org/grpc/internal/transport/transport.go:366 +0xa9
        google.golang.org/grpc.(*parser).recvMsg(0x14735d429018, 0x1000000)
        	/src/vendor/google.golang.org/grpc/rpc_util.go:772 +0x3b
        google.golang.org/grpc.recvAndDecompress(0x14735d429018, {0x2520e60, 0x14735d646b60}, {0x0, 0x0}, 0x1000000, 0x14735d4abba0, {0x0, 0x0}, 0x1)
        	/src/vendor/google.golang.org/grpc/rpc_util.go:931 +0x97
        google.golang.org/grpc.recv(0x3a06d40?, {0x7fac1c464388, 0x3a58e00}, {0x2520e60?, 0x14735d646b60?}, {0x0?, 0x0?}, {0x237ea40, 0x14735d64e240}, 0x1000000, ...)
        	/src/vendor/google.golang.org/grpc/rpc_util.go:1014 +0xab
        google.golang.org/grpc.(*serverStream).RecvMsg(0x14735d429000, {0x237ea40, 0x14735d64e240})
        	/src/vendor/google.golang.org/grpc/stream.go:1823 +0x199
        github.com/moby/buildkit/session/grpchijack.(*conn).Read(0x14735d2d5340, {0x14735d6b2000, 0x8000, 0x14735d4abd48?})
        	/src/session/grpchijack/dial.go:72 +0x232
        bufio.(*Reader).Read(0x14735d47cd20, {0x14735d210744, 0x9, 0x2293a80?})
        	/usr/local/go/src/bufio/bufio.go:245 +0x197
        io.ReadAtLeast({0x251eb60, 0x14735d47cd20}, {0x14735d210744, 0x9, 0x9}, 0x9)
        	/usr/local/go/src/io/io.go:335 +0x8e
        io.ReadFull(...)
        	/usr/local/go/src/io/io.go:354
        golang.org/x/net/http2.readFrameHeader({0x14735d210744, 0x9, 0x147300000000?}, {0x251eb60?, 0x14735d47cd20?})
        	/src/vendor/golang.org/x/net/http2/frame.go:250 +0x65
        golang.org/x/net/http2.(*Framer).ReadFrameHeader(0x14735d210700)
        	/src/vendor/golang.org/x/net/http2/frame.go:513 +0x6b
        google.golang.org/grpc/internal/transport.(*framer).readFrame(0x14735d2db600)
        	/src/vendor/google.golang.org/grpc/internal/transport/http_util.go:480 +0x45
        google.golang.org/grpc/internal/transport.(*http2Client).reader(0x14735d4ced88, 0x14735d153960)
        	/src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1659 +0x1ba
        created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 251
        	/src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:411 +0x1e13
        
        goroutine 244 [chan receive, 4 minutes]:
        google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x14735d54d350, {0x25429d0, 0x14735d08dae0})
        	/src/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xe5
        created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 171
        	/src/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x11a
        
        goroutine 280 [chan receive, 4 minutes]:
        github.com/moby/buildkit/solver/llbsolver/history.(*Queue).ImportStatus(0x14735d167f40?, {0x25429d0, 0x14735d6a31d0}, 0x14735d515730)
        	/src/solver/llbsolver/history/buildhistory.go:709 +0x2da
        github.com/moby/buildkit/solver/llbsolver.(*Solver).recordBuildHistory.func1.5()
        	/src/solver/llbsolver/history.go:186 +0x50
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:93 +0x50
        created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 172
        	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x95
        
        goroutine 256 [select]:
        google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x14735d1595c0, 0x1)
        	/src/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:420 +0x10a
        google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x14735d12ad20)
        	/src/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:597 +0x78
        google.golang.org/grpc/internal/transport.NewHTTP2Client.func6()
        	/src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:469 +0xd2
        created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 251
        	/src/vendor/google.golang.org/grpc/internal/transport/http2_client.go:467 +0x23db
        
        goroutine 248 [select, 4 minutes]:
        github.com/moby/buildkit/util/progress.(*MultiReader).Reader.func1()
        	/src/util/progress/multireader.go:92 +0xeb
        created by github.com/moby/buildkit/util/progress.(*MultiReader).Reader in goroutine 175
        	/src/util/progress/multireader.go:49 +0x26f
        
        goroutine 243 [chan receive, 4 minutes]:
        google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x14735d54d320, {0x25429d0, 0x14735d08da90})
        	/src/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xe5
        created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 171
        	/src/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x11a
        
        goroutine 285 [select, 4 minutes]:
        github.com/moby/buildkit/util/progress.(*progressReader).Read.func1()
        	/src/util/progress/progress.go:124 +0xc8
        created by github.com/moby/buildkit/util/progress.(*progressReader).Read in goroutine 281
        	/src/util/progress/progress.go:121 +0x145
        
        goroutine 176 [chan receive, 4 minutes]:
        github.com/moby/buildkit/control.(*Controller).Status.func2()
        	/src/control/control.go:572 +0x9c
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:93 +0x50
        created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 170
        	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x95
        
        goroutine 241 [chan receive, 4 minutes]:
        github.com/moby/buildkit/control.(*Controller).Session.func1()
        	/src/control/control.go:595 +0x30
        created by github.com/moby/buildkit/control.(*Controller).Session in goroutine 171
        	/src/control/control.go:594 +0x189
        
        goroutine 242 [chan receive, 4 minutes]:
        google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x14735d54d2f0, {0x25429d0, 0x14735d08d8b0})
        	/src/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:88 +0xe5
        created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 171
        	/src/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x11a
        
        goroutine 274 [select, 4 minutes]:
        github.com/moby/buildkit/util/progress.(*progressReader).Read.func1()
        	/src/util/progress/progress.go:124 +0xc8
        created by github.com/moby/buildkit/util/progress.(*progressReader).Read in goroutine 175
        	/src/util/progress/progress.go:121 +0x145
        
        goroutine 283 [select, 4 minutes]:
        github.com/moby/buildkit/util/progress.(*MultiReader).Reader.func1()
        	/src/util/progress/multireader.go:92 +0xeb
        created by github.com/moby/buildkit/util/progress.(*MultiReader).Reader in goroutine 281
        	/src/util/progress/multireader.go:49 +0x26f
        
        goroutine 231 [runnable]:
        syscall.Syscall(0x0, 0x12, 0x14735d1d8ea1, 0x1)
        	/usr/local/go/src/syscall/syscall_linux.go:74 +0x25
        syscall.read(0x14735d644380?, {0x14735d1d8ea1?, 0x14735d31cf00?, 0x0?})
        	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:736 +0x38
        syscall.Read(...)
        	/usr/local/go/src/syscall/syscall_unix.go:183
        internal/poll.ignoringEINTRIO(...)
        	/usr/local/go/src/internal/poll/fd_unix.go:738
        internal/poll.(*FD).Read(0x14735d644380, {0x14735d1d8ea1, 0x1, 0x1})
        	/usr/local/go/src/internal/poll/fd_unix.go:161 +0x21d
        net.(*netFD).Read(0x14735d644380, {0x14735d1d8ea1?, 0x78cc00?, 0x14735d354ea0?})
        	/usr/local/go/src/net/fd_posix.go:68 +0x25
        net.(*conn).Read(0x14735ce20148, {0x14735d1d8ea1?, 0x14735d3269c0?, 0x14735d3269c0?})
        	/usr/local/go/src/net/net.go:196 +0x45
        net/http.(*connReader).backgroundRead(0x14735d1d8e80)
        	/usr/local/go/src/net/http/server.go:702 +0x33
        created by net/http.(*connReader).startBackgroundRead in goroutine 230
        	/usr/local/go/src/net/http/server.go:698 +0xb6
        
        goroutine 273 [select, 4 minutes]:
        github.com/moby/buildkit/util/progress.(*progressReader).Read.func1()
        	/src/util/progress/progress.go:124 +0xc8
        created by github.com/moby/buildkit/util/progress.(*progressReader).Read in goroutine 249
        	/src/util/progress/progress.go:121 +0x145
        
        goroutine 258 [chan receive, 4 minutes]:
        github.com/moby/buildkit/util/progress.pipe.func1()
        	/src/util/progress/progress.go:197 +0x3a
        created by github.com/moby/buildkit/util/progress.pipe in goroutine 224
        	/src/util/progress/progress.go:196 +0x175
        
        goroutine 259 [sync.Cond.Wait, 4 minutes]:
        sync.runtime_notifyListWait(0x14735d4e4a10, 0x0)
        	/usr/local/go/src/runtime/sema.go:617 +0x1b3
        sync.(*Cond).Wait(0x900735cf38fc0?)
        	/usr/local/go/src/sync/cond.go:71 +0x73
        github.com/moby/buildkit/util/progress.(*progressReader).Read(0x14735d614c30, {0x25424d0, 0x3a58e00})
        	/src/util/progress/progress.go:158 +0x1a5
        github.com/moby/buildkit/util/flightcontrol.(*progressState).run(0x14735d614ba0, {0x2522020, 0x14735d614c30})
        	/src/util/flightcontrol/flightcontrol.go:300 +0x75
        created by github.com/moby/buildkit/util/flightcontrol.newCall[...] in goroutine 224
        	/src/util/flightcontrol/flightcontrol.go:113 +0x28c
        
        goroutine 260 [chan receive, 4 minutes]:
        github.com/moby/buildkit/util/flightcontrol.(*Group[...]).do.func1()
        	/src/util/flightcontrol/flightcontrol.go:72 +0x54
        created by github.com/moby/buildkit/util/flightcontrol.(*Group[...]).do in goroutine 224
        	/src/util/flightcontrol/flightcontrol.go:70 +0x254
        
        goroutine 261 [select, 4 minutes]:
        google.golang.org/grpc/internal/transport.(*ClientStream).waitOnHeader(0x14735d647520)
        	/src/vendor/google.golang.org/grpc/internal/transport/client_stream.go:96 +0x66
        google.golang.org/grpc/internal/transport.(*ClientStream).RecvCompress(...)
        	/src/vendor/google.golang.org/grpc/internal/transport/client_stream.go:111
        google.golang.org/grpc.(*csAttempt).recvMsg(0x14735d098c30, {0x238e4e0, 0x14735d6a2820}, 0x1000002508490?)
        	/src/vendor/google.golang.org/grpc/stream.go:1155 +0x105
        google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x11?)
        	/src/vendor/google.golang.org/grpc/stream.go:1020 +0x1f
        google.golang.org/grpc.(*clientStream).withRetry(0x14735d2fa480, 0x14735d558828, 0x14735d558818)
        	/src/vendor/google.golang.org/grpc/stream.go:840 +0x12d
        google.golang.org/grpc.(*clientStream).RecvMsg(0x14735d2fa480, {0x238e4e0?, 0x14735d6a2820?})
        	/src/vendor/google.golang.org/grpc/stream.go:1019 +0x149
        google.golang.org/grpc.invoke({0x25424d0?, 0x3a58e00?}, {0x248473c?, 0x20e1580?}, {0x2380200, 0x14735cecd5c0}, {0x238e4e0, 0x14735d6a2820}, 0x10?, {0x14735d6cc2d0, ...})
        	/src/vendor/google.golang.org/grpc/call.go:73 +0xc4
        github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor({0x25424d0?, 0x3a58e00?}, {0x248473c?, 0x48bce5?}, {0x2380200?, 0x14735cecd5c0?}, {0x238e4e0?, 0x14735d6a2820?}, 0x14735d5589d0?, 0x2508480, ...)
        	/src/util/grpcerrors/intercept.go:41 +0xa7
        google.golang.org/grpc.(*ClientConn).Invoke(0x14735d498008, {0x25424d0?, 0x3a58e00?}, {0x248473c?, 0x22?}, {0x2380200?, 0x14735cecd5c0?}, {0x238e4e0?, 0x14735d6a2820?}, {0x14735d6d0100, ...})
        	/src/vendor/google.golang.org/grpc/call.go:35 +0x1f7
        github.com/moby/buildkit/session/auth.(*authClient).Credentials(0x14735d6d00f0, {0x25424d0, 0x3a58e00}, 0x14735cecd5c0, {0x0, 0x0, 0x14735d2db580?})
        	/src/session/auth/auth_grpc.pb.go:49 +0x191
        github.com/moby/buildkit/util/resolver.(*dockerAuthorizer).getCredentials.(*dockerAuthorizer).getCredentials.CredentialsFunc.func1.func2({0x25424d0, 0x3a58e00}, {0x14735ce786a0, 0x19}, {0x2543298?, 0x14735d2db580?})
        	/src/session/auth/auth.go:34 +0x102
        github.com/moby/buildkit/session.(*Manager).Any(0x14735cf459b0, {0x25424d0, 0x3a58e00}, {0x251e320?, 0x14735d4ef920?}, 0x14735d558c10)
        	/src/session/group.go:83 +0x305
        github.com/moby/buildkit/util/resolver.(*dockerAuthorizer).getCredentials.CredentialsFunc.func1(...)
        	/src/session/auth/auth.go:31
        github.com/moby/buildkit/util/resolver.(*dockerAuthorizer).getCredentials(0x14735d6cc000?, {0x14735d157147?, 0x14735d615560?})
        	/src/util/resolver/authorizer.go:144 +0xef
        github.com/moby/buildkit/util/resolver.(*dockerAuthorizer).AddResponses(0x14735d615470, {0x2542998, 0x14735d615560}, {0x14735d3624f0, 0x1, 0x1})
        	/src/util/resolver/authorizer.go:208 +0x1fe
        github.com/containerd/containerd/v2/core/remotes/docker.(*request).retryRequest(0x14735d626750, {0x2542998, 0x14735d615560}, {0x14735d3624f0, 0x1, 0x1}, 0x0?)
        	/src/vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go:744 +0x1f9
        github.com/containerd/containerd/v2/core/remotes/docker.(*request).doWithRetriesInner(0x14735d626750, {0x2542998, 0x14735d615560}, {0x0, 0x0, 0x0}, 0x1)
        	/src/vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go:723 +0xfd
        github.com/containerd/containerd/v2/core/remotes/docker.(*request).doWithRetries(0x14735d626750, {0x2542998?, 0x14735d615560?}, 0x6?, {0x0, 0x0, 0x4?})
        	/src/vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go:698 +0x69
        github.com/containerd/containerd/v2/core/remotes/docker.(*dockerResolver).Resolve(0x14735d4e49c0, {0x2542998, 0x14735d615260}, {0x14735ce7db80, 0x46})
        	/src/vendor/github.com/containerd/containerd/v2/core/remotes/docker/resolver.go:308 +0xc0f
        github.com/moby/buildkit/util/resolver.(*Resolver).Resolve(0x14735d141c20, {0x2542998, 0x14735d615260}, {0x14735ce7db80, 0x46})
        	/src/util/resolver/pool.go:250 +0x290
        github.com/moby/buildkit/util/imageutil.Config({0x25429d0, 0x14735d628190}, {0x14735ce7d8b0, 0x46}, {0x253d090, 0x14735d141c20}, {0x7fac1c63be20, 0x14735d2abb20}, {0x254f280, 0x14735d2abc20}, ...)
        	/src/util/imageutil/config.go:107 +0x689
        github.com/moby/buildkit/source/containerimage.(*Source).ResolveImageMetadata.func2({0x25429d0?, 0x14735d628190?})
        	/src/source/containerimage/source.go:190 +0x9b
        github.com/moby/buildkit/util/flightcontrol.(*call[...]).run(0x2558280)
        	/src/util/flightcontrol/flightcontrol.go:122 +0x179
        sync.(*Once).doSlow(0x14735d155b80?, 0x25590a0?)
        	/usr/local/go/src/sync/once.go:78 +0xac
        sync.(*Once).Do(...)
        	/usr/local/go/src/sync/once.go:69
        created by github.com/moby/buildkit/util/flightcontrol.(*call[...]).wait in goroutine 224
        	/src/util/flightcontrol/flightcontrol.go:166 +0x4ef
        
        goroutine 281 [sync.Cond.Wait, 4 minutes]:
        sync.runtime_notifyListWait(0x14735cf19990, 0x1)
        	/usr/local/go/src/runtime/sema.go:617 +0x1b3
        sync.(*Cond).Wait(0x0?)
        	/usr/local/go/src/sync/cond.go:71 +0x73
        github.com/moby/buildkit/util/progress.(*progressReader).Read(0x14735d6cd770, {0x25429d0, 0x14735d6a31d0})
        	/src/util/progress/progress.go:158 +0x1a5
        github.com/moby/buildkit/solver.(*Job).Status(0x14735d098870, {0x25429d0, 0x14735d6a31d0}, 0x14735d515730)
        	/src/solver/progress.go:28 +0x24b
        github.com/moby/buildkit/solver/llbsolver.(*Solver).recordBuildHistory.func1.6()
        	/src/solver/llbsolver/history.go:205 +0x2d
        golang.org/x/sync/errgroup.(*Group).Go.func1()
        	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:93 +0x50
        created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 172
        	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x95
        
        goroutine 229 [sync.Mutex.Lock]:
        internal/sync.runtime_SemacquireMutex(0x413685?, 0x88?, 0x14735d3b8750?)
        	/usr/local/go/src/runtime/sema.go:95 +0x25
        internal/sync.(*Mutex).lockSlow(0x14735d4e4950)
        	/usr/local/go/src/internal/sync/mutex.go:149 +0x15d
        internal/sync.(*Mutex).Lock(...)
        	/usr/local/go/src/internal/sync/mutex.go:70
        sync.(*Mutex).Lock(...)
        	/usr/local/go/src/sync/mutex.go:46
        github.com/moby/buildkit/util/resolver.(*Pool).gc(0x14735d188490)
        	/src/util/resolver/pool.go:49 +0x14e
        created by time.goFunc
        	/usr/local/go/src/time/sleep.go:215 +0x2d
    build_test.go:2405: 
        	Error Trace:	/src/client/build_test.go:2405
        	            				/src/util/testutil/integration/run.go:105
        	            				/src/util/testutil/integration/run.go:253
        	Error:      	Received unexpected error:
        	            	Canceled: context canceled
        	            	github.com/moby/buildkit/util/stack.Enable
        	            		/src/util/stack/stack.go:82
        	            	github.com/moby/buildkit/util/grpcerrors.FromGRPC
        	            		/src/util/grpcerrors/grpcerrors.go:264
        	            	github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
        	            		/src/util/grpcerrors/intercept.go:41
        	            	google.golang.org/grpc.(*ClientConn).Invoke
        	            		/src/vendor/google.golang.org/grpc/call.go:35
        	            	github.com/moby/buildkit/api/services/control.(*controlClient).Solve
        	            		/src/api/services/control/control_grpc.pb.go:88
        	            	github.com/moby/buildkit/client.(*Client).solve.func2
        	            		/src/client/solve.go:320
        	            	golang.org/x/sync/errgroup.(*Group).Go.func1
        	            		/src/vendor/golang.org/x/sync/errgroup/errgroup.go:93
        	            	runtime.goexit
        	            		/usr/local/go/src/runtime/asm_amd64.s:1771
        	            	failed to solve
        	            	github.com/moby/buildkit/client.(*Client).solve.func2
        	            		/src/client/solve.go:322
        	            	golang.org/x/sync/errgroup.(*Group).Go.func1
        	            		/src/vendor/golang.org/x/sync/errgroup/errgroup.go:93
        	            	runtime.goexit
        	            		/usr/local/go/src/runtime/asm_amd64.s:1771
        	Test:       	TestClientGatewayIntegration/TestClientGatewayCanceledCredentialsCallbackReturns/worker=oci

@crazy-max crazy-max force-pushed the session-auth-cancel branch 2 times, most recently from db35f19 to b0818f7 Compare April 1, 2026 09:57
@crazy-max crazy-max force-pushed the session-auth-cancel branch from 0037029 to 29e2331 Compare April 1, 2026 13:02
@crazy-max
session(auth): cancel blocked credential callbacks
d6d3c4d 
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
@crazy-max crazy-max force-pushed the session-auth-cancel branch from 29e2331 to d6d3c4d Compare April 1, 2026 13:25
@crazy-max crazy-max requested a review from tonistiigi April 1, 2026 14:10
@crazy-max crazy-max marked this pull request as ready for review April 1, 2026 14:12
tonistiigi
tonistiigi approved these changes Apr 2, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@tonistiigi tonistiigi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems ok but it is only affecting the manual cancellation case, so probably more fixes needed to handle the issues in the related discussion.

@crazy-max crazy-max added this to the v0.30.0 milestone Apr 2, 2026
@crazy-max crazy-max merged commit 283f4cc into moby:master Apr 2, 2026
191 checks passed
@crazy-max crazy-max deleted the session-auth-cancel branch April 2, 2026 08:28
@crazy-max crazy-max mentioned this pull request Apr 2, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tonistiigi tonistiigi tonistiigi approved these changes

Assignees

@crazy-max crazy-max

Labels

area/client area/session area/testing area/util

Projects

None yet

Milestone

v0.30.0

Development

Successfully merging this pull request may close these issues.

2 participants

@crazy-max @tonistiigi