Run integration tests against dockerd

[SamWhited]

Adds a dockerd worker to integration tests when an environment variable is set.

[GordonTheTurtle]

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "test_dockerd" git@github.com:SamWhited/buildkit.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

[tonistiigi]

Buildkitd should not start dockerd. Dockerd sandbox implementation starts dockerd and gets a client through Docker API instead of calling runBuildkitd.

[SamWhited]

Buildkitd should not start dockerd. Dockerd sandbox implementation starts dockerd and gets a client through Docker API instead of calling runBuildkitd.

@tonistiigi we still have to run buildkitd, no? As far as I can tell buildkitdclient.New doesn't launch buildkitd itself which is why everything else calls runbuildkitd.

[tonistiigi]

we still have to run buildkitd, no?

No. Need to run docker daemon, get docker client for that daemon, upgrade that client to buildkit client like https://github.com/docker/buildx/blob/master/driver/docker/driver.go#L42-L44 and run tests on that buildkit client.

[SamWhited]

Okay, I've been digging around all day to figure out how best to do this and I think we'll still need a bigger restructure than the sandbox/backend changes that have already been made, but we'll see. I'll have to tweak sandbox to include some client options or a new client method since right now the tests themselves each create their buildkit clients. Since we create the sandbox implementation with newSandbox, we can add whatever we need there but it will end up being a bigger change than I thought was necessary. I may also have to figure out a place to move some things to avoid import loops, but we'll see. I'm a bit nervous moving things around since I still don't really understand what everything in here does or why it's necessary, as I discover that I may start documenting them as well.

[tonistiigi]

Shouldn't be a need for big refactorings afaics and lets make sure to avoid changes to existing test (instead we can listen on a temporary socket for example). It is expected that not all tests will work this way(and not all features are actually supported in docker yet) and some need to be skipped.

[AkihiroSuda]

Is the dockerd worker expected to be used only for testing? Or will it replace Moby's builder-next pkg?

[SamWhited]

@AkihiroSuda this is just to run the integration tests.

[SamWhited]

Updated docker dependency again. PTAL.

[tonistiigi]

@SamWhited Please post a log of what a test run result looks with the new worker if this is ready for review. I don't see any skips in tests and I'm sure some of them can't pass with moby atm.

[SamWhited]

@tonistiigi the dockerd worker isn't initiated unless TEST_DOCKERD environment variable isn't set to "1". This will be done on the docker side as soon as I can get these tests running (I'm trying to figure out where the best place to clone buildkit and run these with the environment variable in docker/docker would be right now). For this PR no tests will be run because nothing is setting TEST_DOCKERD.

[tonistiigi]

Yes, I know buildkit CI does not invoke this code. That's why I asked to just post the logs of the manual run so we can see if this is doing the right things and how far along it is.

Eg. for the possible test skips, they still need to be in this PR, even if it is invoked in moby CI later.

[SamWhited]

Yes, I know buildkit CI does not invoke this code. That's why I asked to just post the logs of the manual run so we can see if this is doing the right things and how far along it is.

Ah, I see what you mean. I'll try to get the tests running locally so that I can actually set the environment variable.

Eg. for the possible test skips, they still need to be in this PR, even if it is invoked in moby CI later.

I'm not sure what this means, what are possible test skips?

EDIT: Oh, do you mean to suggest that some tests won't work with dockerd specifically? I guess I'll figure out what when I get them running. Will advise.

[SamWhited]

Digging through the failures now; most of them are just that no dockerd binary exists, which makes sense since this is being run in the buildkit container and not the dockerd one. If any of them are failing for other reasons that aren't actual failures I'll add skips for them.

[tonistiigi]

most of them are just that no dockerd binary exists, which makes sense since this is being run in the buildkit container and not the dockerd one.

I'm not sure what you mean by that. TEST_DOCKERD has a hard requirement on dockerd.

[SamWhited]

I'm not sure what you mean by that. TEST_DOCKERD has a hard requirement on dockerd.

yes, but I'm just running these with make test which uses a container that doesn't have dockerd. I haven't figured out how to get these running in the actual dockerd container yet where this won't happen.

[SamWhited]

Trying to get this building in moby/moby#40023, once I do I'll update it to manually checkout this PR so we can verify that it works and have a log.

[SamWhited]

Finally got an actual clean run of these tests (https://ci.docker.com/public/job/moby/job/PR-40023/107/execution/node/248/log/), working on cleaning up things that need to be skipped and tracking down networking issues now.

[SamWhited]

Once #1245 is merged this PR should also be ready, then I can update moby/moby#40023 to stop using my fork.

[SamWhited]

💚 tests; PTAL, thanks! A complete run with green tests has also been run on the docker side of things.

[AkihiroSuda]

@tonistiigi PTAL

[SamWhited]

Ping

[AkihiroSuda]

needs rebase

[SamWhited]

Rebased!

[AkihiroSuda]

@tonistiigi @tiborvass PTAL

[AkihiroSuda]

ping @tonistiigi

[tonistiigi]

Merged in #1434

@SamWhited Thanks for working on this and sorry that it got stuck.