build failures under load due to deadline exceeded from recordBuildHistory

[coryb]

I have been trying to track down some random deadline exceeded errors from buildkit that start happening after many hours of load testing. With Go 1.21rc2 I updated all the WithTimeout to WithTimeoutCause and annotated the deadline error with the function/line it came from and found all of my deadline exceeded errors came from recordBuildHistory

DeadlineExceeded: recordBuildHistory: context deadline exceeded
1224004 v0.0.0+unknown buildkitd
github.com/moby/buildkit/solver/llbsolver.(*Solver).recordBuildHistory.func1
        /src/solver/llbsolver/solver.go:188
github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func2
        /src/solver/llbsolver/solver.go:460
github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve
        /src/solver/llbsolver/solver.go:576
github.com/moby/buildkit/control.(*Controller).Solve
        /src/control/control.go:431

This is likely related to #3743

I am wondering if we should have a shorter timeout and instead of returning the context error, just log that we failed to record build history. It doesn't seem right to return an error for the solve when the solve actually succeeds and we just failed to record it.