vendor: update buildkit to master@c4f191410a41

[jedevc]

Split out from #6134 (note - all we actually need here is the buildkit update. Buildkit master already uses a more recent version of Docker which should not be affected by https://github.com/dagger/dagger/security/dependabot/386).

There are two significant changes in upstream (both made by me) since we last updated:

• Git URL parsing has changed to return a custom type, to bridge over the differences between standard URLs and SCP-style URLs. This actually simplifies our parsing code for the command line git directory args, but means we need to update our tests.
• fsutil has updated to split the FS interface more, so that now we need to manually create a FilterFS based on a standard FS to define filtering operations (like include, exclude, map, etc).

This vendoring update also bumps github.com/docker/docker to v25.0.0-beta.1, which should prevent GHSA-jq35-85cj-fj4p.

[jedevc]

cc @sipsma, this also removes our OTEL hack to avoid panics with conflicting versions of schemas (this was fixed in moby/buildkit#4318).

[gerhard]

All changes look good to me! WDYT @sipsma @vito?

I am suspicious of the failing checks in CI, doing a local run now.

We shouldn't merge until all checks are green.

---

Current status: dagger run ./hack/make engine:test

[gerhard]

Yes, 189 tests are failing for me locally too:

Are they failing for you too @jedevc?

[jedevc]

Yup, I missed these, will dig into them now :)

[jedevc]

Ok, it looks like at some point buildkit has introduced a regression which causes us to deadlock - looks like moby/buildkit#4347 is the possible culprit, so I've taken the commit right before it was merged (which thankfully has the fsutil update, so we can pull in docker 25).

I'll dive into the deadlock later, but we don't need to block this PR on it.

[gerhard]

I just merged the other PR which introduced some conflicts in the go.mod & go.sum.

Would you mind rebasing @jedevc? If I do it, I will appear as a co-author on all commits which doesn't seem right.

If all checks pass after the rebase, all good to merge from my POV. Would feel more comfortable if this gets @sipsma 👍 too

[sipsma]

LGTM, but

cc @sipsma, this also removes our OTEL hack to avoid panics with conflicting versions of schemas (this was fixed in moby/buildkit#4318).

@jedevc did we have to back out of this due to the other deadlock issue? Just curious, not blocking

Ok, it looks like at some point buildkit has introduced a regression which causes us to deadlock - looks like moby/buildkit#4347 is the possible culprit, so I've taken the commit right before it was merged (which thankfully has the fsutil update, so we can pull in docker 25).

Good find on that too btw; very curious what we're doing in our integ tests that buildkit isn't (or if this is somehow particular our dagger-engine flavored buildkit, though it's hard to immediately imagine how since that commit changes things several abstraction layers away from anything we diverge from w/ vanilla buildkit)

[jedevc]

I just merged the other PR which introduced some conflicts in the go.mod & go.sum.

Would you mind rebasing @jedevc? If I do it, I will appear as a co-author on all commits which doesn't seem right.

Done, should be good to go 🤞

@jedevc did we have to back out of this due to the other deadlock issue? Just curious, not blocking

Yup 😢 I just merged moby/buildkit#4318, so it appears in the history after the deadlock we hit from moby/buildkit#4347. Once we find the fix for the deadlock, we should be able to pick that commit over.

Good find on that too btw; very curious what we're doing in our integ tests that buildkit isn't (or if this is somehow particular our dagger-engine flavored buildkit, though it's hard to immediately imagine how since that commit changes things several abstraction layers away from anything we diverge from w/ vanilla buildkit)

Yeah I'm not actually 100% sure on this 🤔 I wonder if maybe we only hit this when doing tons of builds - I think it's specifically triggered by lots of edge merging. Upstream buildkit tests each create their very own buildkit instance (I don't think there's any optimization or anything to share these), so they're all isolated from each other - this has the benefit of isolating each test in it's own sandbox, but means that we don't really have any large scale tests where they can all run into each other.

Since we use only a single dagger backend for all our tests, I think it makes edge merges a lot more likely, especially if we're running loads of tests in parallel (which would explain why this doesn't reproduce when only running a small subset of tests).

I think it's more likely that this is an upstream buildkit issue instead of something we're doing specifically in dagger, but I think some more investigation is needed with a reproducer before we can report upstream - will look at seeing if I can get this to reliably reproduce.

[gerhard]

💪 🚀