Deadlock on cache mounts

[cpuguy83]

I am experiencing deadlocks related to preparing cache mounts.

Here is a stack trace from one of my build agents:
https://gist.github.com/cpuguy83/0e79cd121c780df71eb1cbbcbcc8f1e1

This has wedged most of my build agents (when they try to lookup a particular cache mount) and seems to happen rather easily.

I have setup my builds to generate stack dumps any time we end up cancelling (e.g. due to timeout) the build so I can get more of these pretty easily.

[cpuguy83]

Comments from @tonistiigi on Slack:

Seems like a loop in https://gist.github.com/cpuguy83/0e79cd121c780df71eb1cbbcbcc8f1e1#file-dockerd-goroutines-txt-L2206-L2210

there’s a double clone going on where cacheRefShare’s Mutable ref is another CacheRef. Not sure why it doesn’t happen more frequently though. not sure if just need to handle that case in release or should do some refactoring to avoid case like this completely

[cpuguy83]

I know where my concurrency is coming from.
I end up building several go binaries (all the moby things) simultaneously as part of the build because of stage parallelization.