Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update buildkit past ca71a44 #6374

Closed
jedevc opened this issue Jan 8, 2024 · 1 comment · Fixed by #6428 or #6481
Closed

Update buildkit past ca71a44 #6374

jedevc opened this issue Jan 8, 2024 · 1 comment · Fixed by #6428 or #6481
Assignees

Comments

@jedevc
Copy link
Member

jedevc commented Jan 8, 2024

Currently, we are stuck on the current version of buildkit, since moby/buildkit#4347 (comment) seems to have introduced a deadlock.

To reproduce: update buildkit in go.mod to ca71a446d9f4e31ca7c4fa30fb8fb8861b8b0117, and run a large number of integration tests. The engine will lock up after some time - after sending SIGUIT to the engine, you can observe the stack trace:

goroutine 441 [sync.Mutex.Lock, 4 minutes]:
runtime.gopark(0xc01d5ebbf0?, 0xc00aef48e1?, 0xc0?, 0xf?, 0xc017641870?)
	/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc017641790 sp=0xc017641770 pc=0x43e42e
runtime.goparkunlock(...)
	/usr/local/go/src/runtime/proc.go:404
runtime.semacquire1(0xc01d77eb74, 0x0?, 0x3, 0x1, 0x21?)
	/usr/local/go/src/runtime/sema.go:160 +0x218 fp=0xc0176417f8 sp=0xc017641790 pc=0x44f918
sync.runtime_SemacquireMutex(0xffffffffffffffff?, 0x0?, 0xc00ed7c4b0?)
	/usr/local/go/src/runtime/sema.go:77 +0x25 fp=0xc017641830 sp=0xc0176417f8 pc=0x46d545
sync.(*Mutex).lockSlow(0xc01d77eb70)
	/usr/local/go/src/sync/mutex.go:171 +0x15d fp=0xc017641880 sp=0xc017641830 pc=0x488f1d
sync.(*Mutex).Lock(...)
	/usr/local/go/src/sync/mutex.go:90
github.com/moby/buildkit/util/progress.(*MultiWriter).writeRawProgress(0xc01d77eb70, 0xc004bd6e00)
	/go/pkg/mod/github.com/moby/buildkit@v0.13.0-beta1.0.20231107172820-ca71a446d9f4/util/progress/multiwriter.go:91 +0x5c fp=0xc017641960 sp=0xc017641880 pc=0xcb537c
github.com/moby/buildkit/util/progress.(*MultiWriter).WriteRawProgress(0xc01d77eb70?, 0xc004bd6e00?)
	/go/pkg/mod/github.com/moby/buildkit@v0.13.0-beta1.0.20231107172820-ca71a446d9f4/util/progress/multiwriter.go:87 +0xac fp=0xc017641a28 sp=0xc017641960 pc=0xcb516c
github.com/moby/buildkit/util/progress.(*MultiWriter).writeRawProgress(0xc01dfe5da0, 0xc004bd6e00)
	/go/pkg/mod/github.com/moby/buildkit@v0.13.0-beta1.0.20231107172820-ca71a446d9f4/util/progress/multiwriter.go:95 +0x17e fp=0xc017641b08 sp=0xc017641a28 pc=0xcb549e
github.com/moby/buildkit/util/progress.(*MultiWriter).WriteRawProgress(0xc01dfe5da0?, 0xc004bd6e00?)
	/go/pkg/mod/github.com/moby/buildkit@v0.13.0-beta1.0.20231107172820-ca71a446d9f4/util/progress/multiwriter.go:87 +0xac fp=0xc017641bd0 sp=0xc017641b08 pc=0xcb516c
github.com/moby/buildkit/util/progress.(*MultiWriter).Add(0xc01d77eb70, {0x1f7e8b0?, 0xc01dfe5da0?})
	/go/pkg/mod/github.com/moby/buildkit@v0.13.0-beta1.0.20231107172820-ca71a446d9f4/util/progress/multiwriter.go:46 +0x190 fp=0xc017641c60 sp=0xc017641bd0 pc=0xcb4d70
github.com/moby/buildkit/solver.(*state).setEdge(0xc01dfe3440, 0x2, 0xc003975a40, 0xc0165fcb40)
	/go/pkg/mod/github.com/moby/buildkit@v0.13.0-beta1.0.20231107172820-ca71a446d9f4/solver/jobs.go:178 +0x370 fp=0xc017641ce8 sp=0xc017641c60 pc=0xdef3f0
github.com/moby/buildkit/solver.(*Solver).setEdge(0xc0007071a0, {0xc003975a40?, {0x1f92360?, 0xc004bd6480?}}, 0xc003975a40)
	/go/pkg/mod/github.com/moby/buildkit@v0.13.0-beta1.0.20231107172820-ca71a446d9f4/solver/jobs.go:295 +0x125 fp=0xc017641d58 sp=0xc017641ce8 pc=0xdf0265
github.com/moby/buildkit/solver.(*scheduler).dispatch(0xc0002fc770, 0xc01356a140)
	/go/pkg/mod/github.com/moby/buildkit@v0.13.0-beta1.0.20231107172820-ca71a446d9f4/solver/scheduler.go:177 +0xbdb fp=0xc017641f68 sp=0xc017641d58 pc=0xdfe87b
github.com/moby/buildkit/solver.(*scheduler).loop(0xc0002fc770)
	/go/pkg/mod/github.com/moby/buildkit@v0.13.0-beta1.0.20231107172820-ca71a446d9f4/solver/scheduler.go:104 +0x1d4 fp=0xc017641fc8 sp=0xc017641f68 pc=0xdfdb34
github.com/moby/buildkit/solver.newScheduler.func1()
	/go/pkg/mod/github.com/moby/buildkit@v0.13.0-beta1.0.20231107172820-ca71a446d9f4/solver/scheduler.go:35 +0x25 fp=0xc017641fe0 sp=0xc017641fc8 pc=0xdfd865
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc017641fe8 sp=0xc017641fe0 pc=0x4715e1
created by github.com/moby/buildkit/solver.newScheduler in goroutine 1
	/go/pkg/mod/github.com/moby/buildkit@v0.13.0-beta1.0.20231107172820-ca71a446d9f4/solver/scheduler.go:35 +0x22d

See #6169 for more information on when this was originally discovered.

Next steps:

  • Wait until upstream PRs are merged 🎉
@jedevc jedevc changed the title Update buildkit past ca71a446d9f4e31ca7c4fa30fb8fb8861b8b0117 Update buildkit past ca71a44 Jan 8, 2024
@jedevc jedevc added the kind/upstream Related to upstream project (eg. Buildkit) label Jan 8, 2024
@jedevc jedevc self-assigned this Jan 8, 2024
@jedevc jedevc reopened this Jan 19, 2024
@jedevc
Copy link
Member Author

jedevc commented Jan 19, 2024

The fix for this was reverted in #6458.

@samalba samalba removed the kind/upstream Related to upstream project (eg. Buildkit) label May 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants