As a Developer I want to be able to build Windows Containers with BuildKit

[bplasmeijer]

The windows BuildKit is only supported on Linux

[jsturtevant]

It is possible to use buildx (which uses buildkit) if there are no RUN commands in the docker file:

docker buildx create --name img-builder --use
docker buildx build --no-cache --pull --push \
  --platform "windows/amd64" -t $imagetag -f $dockerfile .
docker buildx rm img-builder

ARG nanoserverTag="mcr.microsoft.com/windows/nanoserver:1809-amd64"
# othertags list: curl -L https://mcr.microsoft.com/v2/windows/nanoserver/tags/list

FROM ${nanoserverTag}
COPY some.exe "/Windows/System32/some.exe"

It would be interesting to see if could use some windows emulator connected to Binfmt_misc as QEMU is used for other architectures

[jorgearteiro]

There is an issue logged on BuildKit repo moby/buildkit#616

[TBBle]

@jsturtevant If you wanted to go down that path, I believe Wine can be used with binfmt_misc, although I'm not sure how well that would work with namespaces since there's a WineServer and configuration directory to create, and you probably don't want that in the final image...

It would be easier to prototype that on a raw Linux box than via the Linux support in Docker on Windows, I think.

[ghost]

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

[TBBle]

Current status of my work on moby/buildkit#616 is that it's stalled waiting on a pile of 4 containerd PRs to be reviewed and advanced. The top of the stack is containerd/containerd#4419, but there's some outstanding design questions on the top two PRs regarding HCS behaviours, see microsoft/hcsshim#853, but the bottom two are just pending containerd maintainer review to merge or rework as appropriate.

And after that, will come fixing path-related issues in the BuildKit LLB layer, for which I'd only scratched the surface of in moby/buildkit#1621, exploring the solution space, and which needs to be reworked per maintainer feedback.

[ghost]

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

[TBBle]

Status unchanged, still waiting on merging or feedback on containerd support for Windows container creation/export/mount support, and then continuing the work on BuildKit itself, starting from moby/buildkit#1621

[immuzz]

@kevpar Do you have any information on the containerd PRs mentioned above?

[ghost]

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

[TBBle]

Still working through containerd PRs per above. One of four has landed, and one needs to be rewritten or at least iterated upon further. I've also opened PR on hcsshim again to remove some duplication of code with the containerd PRs.

That said, Docker is looking at supporting containerd on Windows for 21.x (moby/moby#41455), which might mean we get BuildKit on Docker+containerd even without that containerd PR stack, as the parts I was trying to implement in containerd (WCOW layer management) are provided by Docker when Docker uses containerd, as far as I know.

I haven't looked closely at how BuildKit integrates with Docker though, it's possible it'll still need fixes in containerd, and either way, BuildKit itself has higher-level issues to be resolved, e.g., moby/buildkit#1621.

[ghost]

This issue has been open for 30 days with no updates.
@immuzz, please provide an update or close this issue.

[TBBle]

Not much has changed since December. The containerd PRs are now in a reasonable state, two are ready-to-go and one (containerd/containerd#4419) is functional but possibly needs some discussion about the test suite changes, and depends on a hcsshim feature PR (microsoft/hcsshim#901).

Some recent testing suggests that that hcsshim PR might have issues with Windows 10 20H2 that don't appear on Windows Server LTSC2019, but I haven't definitely proved that as I have only tested the latter via CI pipelines so far. Edit: Still using CI pipelines, I confirmed that the PR works on RS5 on AppVeyor (VS 2019 image), but fails on my personal Windows 10 20H2 box, suggesting that some part of the reverse engineering used to build that PR was incorrect or incomplete.

[immuzz]

@weijuans-msft can you provide clarity here as part of your effort to provide containerd parity with Docker developer experience.

[ghost]

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

[TBBle]

Compared to January, I've identified and have a fix for the issue that microsoft/hcsshim#901 shows on Windows 10 20H2, but that PR is still pending merge, and then the fix will be up for PR.

The containerd PRs (containerd/containerd#4399, containerd/containerd#4415, and containerd/containerd#4419) should all be in a reasonable state to flow through, although the latter depends on microsoft/hcsshim#901, often triggers a now-known containerd issue (containerd/containerd#4924) in its test suite, and also frequently triggers an unknown (hcsshim?) issue that I've yet been unable to replicate outside containerd.

It's possible that the Docker/containerd integration for Windows (moby/moby#41455 is the best place to track that) will mean some or all of those PRs stop being critical-path for BuildKit on Windows, and then the BuildKit-local platform issues (i.e. filesystem handling and similar) can be addressed.

[weijuans-msft]

@TBBle great work here! I changed this to "In Progress" as I am seeing you and lots of others in the moby and containerd community working on this. Thank you.

[ghost]

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

[ghost]

This issue has been open for 30 days with no updates.
@weijuans-msft, please provide an update or close this issue.

[ntrappe-msft]

Ignore the microsoft-github-policy-service messages, this is still in progress.

[viceice]

don't you have a label to block it? 😞

[ntrappe-msft]

I wish we did, I've talked to the team that creates the bot about being able to customize it (e.g. not run on Q/A or enhancement requests) but there's no support for that right now.

[profnandaa]

Hi folks! We are getting close to releasing the first experimental buildkitd.exe binary with the next Buildkit 0.13.0 RC release, I'll update here once the release is out. We will be releasing this even as we work on stabilizing the product through the integration tests and taking in feedback from the early adopters.

[ntrappe-msft]

@profnandaa This is huge! Thank you + the team for all your hard work.

[profnandaa]

Update: Good news! Buildkit v0.13.0-rc2 is now out with buildkitd.exe binary ready for experimental use. We'll be updating the quick start guide with Windows instructions shortly.

[profnandaa]

// Update: Buildkit 0.13.0 is now released with windows binaries - Release v0.13.0 · moby/buildkit (github.com) 🎉
Thank you all who have continued to test the releases, WCOW support is still experimental but we are on the way to stabilization. We will appreciate any feedback; this will aid in our prioritization especially on closing the Linux/Windows feature-parity gap. Feel free to open the issue at moby/buildkit.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@profnandaa, @iankingori, @lucillex, please provide an update or close this issue.

[slonopotamus]

There's still a lack of support in moby/moby.

[pl4nty]

GitHub Actions hosted Windows runners are pinned to Docker 24.0.7, so I wrote some steps to setup BuildKit manually:

- name: Setup containerd
  run: |
    $version = "1.7.15"
    curl.exe -L https://github.com/containerd/containerd/releases/download/v$version/containerd-$version-windows-amd64.tar.gz -o containerd.tar.gz
    tar.exe xvf containerd.tar.gz

    .\bin\containerd.exe config default | Out-File .\bin\config.toml -Encoding ascii
    .\bin\containerd.exe --register-service
    Start-Service containerd

- name: Setup BuildKit
  run: |
    $version = "v0.13.1"
    curl.exe -L https://github.com/moby/buildkit/releases/download/$version/buildkit-$version.windows-amd64.tar.gz -o buildkit.tar.gz
    tar.exe xvf buildkit.tar.gz
    
    .\bin\buildkitd.exe --register-service --debug --containerd-worker=true --service-name buildkitd
    Start-Service buildkitd

- name: Setup Docker Buildx
  run: |
    $version = "v0.13.1"
    curl.exe -L https://github.com/docker/buildx/releases/download/$version/buildx-$version.windows-amd64.exe -o $env:ProgramData\Docker\cli-plugins\docker-buildx.exe

- uses: docker/setup-buildx-action@v3.2.0
  with:
    driver: remote
    endpoint: npipe:////./pipe/buildkitd

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@profnandaa, @iankingori, @lucillex, please provide an update or close this issue.

[ntrappe-msft]

@pl4nty This is great, thank you! If you want, you can submit a PR for your script under our helpful_tools folder so we can add more visibility.

[microsoft-github-policy-service]

This issue has been open for 30 days with no updates.
@profnandaa, @iankingori, @lucillex, please provide an update or close this issue.

[profnandaa]

// Update: Buildkit 0.13.0 is now released with windows binaries - Release v0.13.0 · moby/buildkit (github.com) 🎉 Thank you all who have continued to test the releases, WCOW support is still experimental but we are on the way to stabilization. We will appreciate any feedback; this will aid in our prioritization especially on closing the Linux/Windows feature-parity gap. Feel free to open the issue at moby/buildkit.

// UPDATE: Stabilization efforts still going on. We have had several releases since v0.13.0, now at v0.14.1. Aiming for GA by September 2024. Feel free to give it a try and open any issues on moby/buildkit.

[FrankRichterAnsys]

If you want to try out HyperV Isolation, BuildKit master branch (as of moby/buildkit#4364) can pass through the necessary options from its own buildkit.toml config file. I haven't tested this, but it may be as simple as passing the SandboxIsolation=1 flag, similar to what containerd/cri does:

[worker.containerd.runtime.options]
  SandboxIsolation = "1"
  ScaleCpuLimitsToSandbox = true

I was trying that, but it didn't work for me.
Trying to build an image on a Windows 11 machine "FROM" the servercore:ltsc2019-amd64 image always ends in The container operating system does not match the host operating system, even with quite recent containerd (1.7.18) and buildkit (v0.14.1) versions.
I'd be grateful for suggestions for other things to try...

[TBBle]

Drat, so either that change wasn't correct, wasn't sufficient, or it didn't flow through into hcsshim for some reason. I'm not sure if containerd will have any useful logs for this and I'm not sure if the relevant options are ever written to disk before being passed to hcsshim.

[slonopotamus]

See instructions for easy experimental way to set up building & running Windows containers using nerdctl and BuildKit.

[FrankRichterAnsys]

See instructions for easy experimental way to set up building & running Windows containers using nerdctl and BuildKit.

These work well enough if the host Windows matches the container Windows.
In my case, however, I want to use different Windows versions - which requires HyperV isolation.
The comment I quoted is explicitly about that, and the contained suggestion was what "it didn't work for me" is referring to.

[slonopotamus]

I'm talking about the topic in general - the way to use BuildKit for building containers on Windows.