Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Epic: Linux Containers on Windows (LCOW) #33850

Closed
PatrickLang opened this issue Jun 27, 2017 · 57 comments
Closed

Epic: Linux Containers on Windows (LCOW) #33850

PatrickLang opened this issue Jun 27, 2017 · 57 comments

Comments

@PatrickLang
Copy link

@PatrickLang PatrickLang commented Jun 27, 2017

Microsoft is hard at work adding support to run both Windows & Linux containers side by side on the same node with a single Docker daemon. As Linux containers are launched, Hyper-V will be used to boot and run a Linux kernel which will then be used to host the container natively. Because many areas of Docker will require changes to handle running multiple platforms side by side, we'll be using this epic to track the list of PRs and proposals in a single place.

Preview documentation

https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/linux-containers

Progress

Required for Docker for Windows - Developer Scenarios

Status Description
Merged Store Linux images on Windows & launch Linux containers on Window. Right now, platform switching is done with a system-wide environment variable. Side by side support needs the next item.
Merged Support Linux filesystem operations in single-use VMs instead of reusable ones. This can provide protection in the case where a file parsing bug could lead to a container polluting the filesystem of another container.
Merged Use a single tag store so both Windows & Linux containers can be stored and managed by the same engine
Merged Remote filesystem support needed for docker commit, docker cp, and Dockerfile ADD commands
Merged Support resizing ext4 sandbox VHD instead of defaulting to 20Gb only.
Merged Bind mount support.
Merged New --platform parameter for docker run, docker pull, docker import and docker build to choose between platforms and order of precedence when multiple are supported (Windows & Linux) on the same node. See Proposal #34617 .
Merged Coalesce daemon stores - needed to manage both Windows & Linux containers with the same daemon
In progress Add docker kill -s for signals != SIGKILL.
Merged Support WORKDIR in running existing images and building new ones
Not started Add platform filters to docker search and docker images
Merged Memory & CPU settings - see: microsoft/opengcs#145
Not started LCOW: docker run --user seems to be ignored #36469

Required for Kubernetes

Status Description
Not started Update docker stats for multiple platforms
Proposal soon Update docker info & daemon configuration to show what platforms each node is capable of, and restrict to single platforms if needed

Required for Swarm

Status Description
Not started Add --platform flag support to docker-compose, and docker stack deploy
Not started Update docker stats for multiple platforms
Not started Support multiple platforms for docker volume commands
Not started Adjust Swarm mode placement to handle running multiple platforms instead of just the node's native platform. Until this is done, use docker service create --no-resolve-image as a workaround

Backlog, need to set priority

Status Description
Not started docker top
Not started docker run flags. Feedback welcome on what's needed vs not
Not started docker export, docker import, docker save need platform flags
Not started UID & GID flag support - see: microsoft/opengcs#146
No plans to support Cross-platform multi stage builds, support for COPY command in Dockerfile

Glossary & References

  • LCOW - Linux Containers on Windows
  • GCS - guest compute service. Receives commands from the Hyper-V host services (DockerD via HCS) and invokes processes for specific tasks, creating namespaces, and creating containers using runC. (Repo link coming soon)
  • HCS - host compute service. Windows-specific service used to manage containers and VMs (doc link coming soon)

There's a brief description of gcs & hcs from a Dockercon 2016 presentation

FAQs

What's needed to fully test this?

  • As of #39108, Windows 10 or Windows Server version 1809 (build 17763), or a later Windows Insider build
  • A build of moby/moby with the PR's above applied
  • A bootable UEFI-based Linux image with the right modules & gcs included - see https://github.com/microsoft/opengcs
  • At least 4GB of RAM and Intel VT or AMD-V instruction support
    • Note: other apps using these instructions (VMWare, Virtualbox, ...) cannot be run concurrently
@AkihiroSuda
Copy link
Member

@AkihiroSuda AkihiroSuda commented Jun 28, 2017

What would be a usecase of "Cross-platform multi stage builds"?

@PatrickLang
Copy link
Author

@PatrickLang PatrickLang commented Jun 28, 2017

@AkihiroSuda It would be great for cross platform builds. For example, if you had a collection of bash scripts and a language with cross-compilation support like golang, then you could actually build your code in Linux then copy the binaries into a microsoft/nanoserver image to run it in a Windows native container or vice versa. The same would be feasible for projects based on JVM, .Net Core, or Mono.

@MZap
Copy link

@MZap MZap commented Jun 20, 2019

Any status on the platform flag for Compose?

@lowenna
Copy link
Contributor

@lowenna lowenna commented Jun 20, 2019

@MZap probably a question to ask over at https://github.com/docker/compose

@oldium
Copy link

@oldium oldium commented Jun 20, 2019

Is the mentioned v2 API related to WSL 2 announced recently for Windows Insiders and to Docker preview, which shall be available in July?

@lowenna
Copy link
Contributor

@lowenna lowenna commented Jun 20, 2019

WSL2 uses some of the HCS v2 APIs. But no, there's no relation as such between LCOW in moby and WSL2.

@oldium
Copy link

@oldium oldium commented Jun 20, 2019

So Linux containers support in Docker preview (which is not yet available) is something different to experimental LCOW in moby from this Epic?

@lowenna
Copy link
Contributor

@lowenna lowenna commented Jun 20, 2019

If you're referring to Docker for Windows, currently it can use this LCOW rather than moby VM as an experimental feature in D4W, but I understand "they" (folks at Docker Inc) are working on an alternate solution using WSL2.

@ggirard07
Copy link

@ggirard07 ggirard07 commented Jun 20, 2019

@jhowardmsft regarding Docker announcement for WSL2, does it mean LCOW is going to die and be replaced by WSL2 support if conclusive?

@oldium
Copy link

@oldium oldium commented Jun 20, 2019

@jhowardmsft Yes, I am referring to https://engineering.docker.com/2019/06/docker-hearts-wsl-2/, which is announced by Docker Inc folks. Anyway, using either solution (MS LCOW or Docker WSL2) will be great improvement over Samba connection to local files (in sense of reliability).

@krshubham
Copy link

@krshubham krshubham commented Aug 8, 2019

Hi, is there a time line for a stable GA of LCOW?

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Aug 8, 2019

LCOW support is still experimental, and the current implementation is not recommended for serious work as there's still many features missing, and quite some known bugs/limitations; the current will be removed at some point (Active work is ongoing to re-implement LCOW at the runtime level through containerd).

If your use-case is to run Linux containers on a Windows machine, you might be interested in the tech preview of Docker Desktop for WSL2;

@kojoru
Copy link

@kojoru kojoru commented Aug 21, 2019

@thaJeztah do you know if there is any alternative (in beta, planned or otherwise) to manage Windows and Linux containers simultaneously without engine switching? WSL2 builds do not allow this at the moment and behave similarly to "Use Linux containers" mode.

That's the main use case for us at the moment, and it's unclear how if at all this will be supported in the future.

@AmitHole
Copy link

@AmitHole AmitHole commented Aug 21, 2019

@thaJeztah do you know how to run Linux container on Windows (LCOW), with --network=transparent.
Because I have linux container which I want to directly run on the host using the transparent driver. So I am running the linux container in the LCOW mode with transparent driver as network. With the below mentioned command.

docker run -d --network=Mytransparent --platform=linux --name dynamicui dynamicui:dev

But ran into the below mentioned issue, after running the above command -

docker run -d --network TeamedNet --platform linux --name dynamicui dynamicui:dev
47cf6329ea2fe7487245547f3f816a3a12b7d3901e4b65358814ae0f3cd36e1a
docker: Error response from daemon: container 47cf6329ea2fe7487245547f3f816a3a12b7d3901e4b65358814ae0f3cd36e1a encountered an error during CreateProcess: failure in a Windows system call: Unspecified error (0x80004005)
[Event Detail: failed to link resolv.conf file for adapter de3394a3-4b6d-4cb9-9e19-04307b13390b: link /etc/resolv.conf /tmp/base/etc/resolv.conf: invalid cross-device link Stack Trace:
github.com/Microsoft/opengcs/service/gcs/oslayer/realos.(*realOS).Link
        /go/src/github.com/Microsoft/opengcs/service/gcs/oslayer/realos/realos.go:232
github.com/Microsoft/opengcs/service/gcs/core/gcs.(*gcsCore).configureAdapterInNamespace
        /go/src/github.com/Microsoft/opengcs/service/gcs/core/gcs/networking.go:57
github.com/Microsoft/opengcs/service/gcs/core/gcs.(*gcsCore).ExecProcess
        /go/src/github.com/Microsoft/opengcs/service/gcs/core/gcs/gcs.go:369
github.com/Microsoft/opengcs/service/gcs/bridge.(*Bridge).execProcess
        /go/src/github.com/Microsoft/opengcs/service/gcs/bridge/bridge.go:582
github.com/Microsoft/opengcs/service/gcs/bridge.(*Bridge).(github.com/Microsoft/opengcs/service/gcs/bridge.execProcess)-fm
        /go/src/github.com/Microsoft/opengcs/service/gcs/bridge/bridge.go:236
github.com/Microsoft/opengcs/service/gcs/bridge.HandlerFunc.ServeMsg
        /go/src/github.com/Microsoft/opengcs/service/gcs/bridge/bridge.go:68
github.com/Microsoft/opengcs/service/gcs/bridge.(*Mux).ServeMsg
        /go/src/github.com/Microsoft/opengcs/service/gcs/bridge/bridge.go:139
github.com/Microsoft/opengcs/service/gcs/bridge.(*Bridge).ListenAndServe.func2.1
        /go/src/github.com/Microsoft/opengcs/service/gcs/bridge/bridge.go:315
runtime.goexit
        /usr/lib/go/src/runtime/asm_amd64.s:2361 Provider: 00000000-0000-0000-0000-000000000000] extra info: {"CommandArgs":["/bin/sh","-c","./Cognex.DynamicUIManager.Server"],"WorkingDirectory":"/DynamicUIManager","Environment":{"ASPNETCORE_URLS":"http://+:80","ASPNETCORE_VERSION":"2.2.6","DOTNET_RUNNING_IN_CONTAINER":"true","HOSTNAME":"47cf6329ea2f","PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"},"CreateStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0],"OCISpecification":{"ociVersion":"1.0.1-dev","process":{"user":{"uid":0,"gid":0},"args":["/bin/sh","-c","./Cognex.DynamicUIManager.Server"],"env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","HOSTNAME=47cf6329ea2f","ASPNETCORE_URLS=http://+:80","DOTNET_RUNNING_IN_CONTAINER=true","ASPNETCORE_VERSION=2.2.6"],"cwd":"/DynamicUIManager","capabilities":{"bounding":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FSETID","CAP_FOWNER","CAP_MKNOD","CAP_NET_RAW","CAP_SETGID","CAP_SETUID","CAP_SETFCAP","CAP_SETPCAP","CAP_NET_BIND_SERVICE","CAP_SYS_CHROOT","CAP_KILL","CAP_AUDIT_WRITE"],"effective":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FSETID","CAP_FOWNER","CAP_MKNOD","CAP_NET_RAW","CAP_SETGID","CAP_SETUID","CAP_SETFCAP","CAP_SETPCAP","CAP_NET_BIND_SERVICE","CAP_SYS_CHROOT","CAP_KILL","CAP_AUDIT_WRITE"],"inheritable":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FSETID","CAP_FOWNER","CAP_MKNOD","CAP_NET_RAW","CAP_SETGID","CAP_SETUID","CAP_SETFCAP","CAP_SETPCAP","CAP_NET_BIND_SERVICE","CAP_SYS_CHROOT","CAP_KILL","CAP_AUDIT_WRITE"],"permitted":["CAP_CHOWN","CAP_DAC_OVERRIDE","CAP_FSETID","CAP_FOWNER","CAP_MKNOD","CAP_NET_RAW","CAP_SETGID","CAP_SETUID","CAP_SETFCAP","CAP_SETPCAP","CAP_NET_BIND_SERVICE","CAP_SYS_CHROOT","CAP_KILL","CAP_AUDIT_WRITE"]}},"root":{"path":"rootfs"},"hostname":"47cf6329ea2f","mounts":[{"destination":"/proc","type":"proc","source":"proc","options":["nosuid","noexec","nodev"]},{"destination":"/dev","type":"tmpfs","source":"tmpfs","options":["nosuid","strictatime","mode=755","size=65536k"]},{"destination":"/dev/pts","type":"devpts","source":"devpts","options":["nosuid","noexec","newinstance","ptmxmode=0666","mode=0620","gid=5"]},{"destination":"/sys","type":"sysfs","source":"sysfs","options":["nosuid","noexec","nodev","ro"]},{"destination":"/sys/fs/cgroup","type":"cgroup","source":"cgroup","options":["ro","nosuid","noexec","nodev"]},{"destination":"/dev/mqueue","type":"mqueue","source":"mqueue","options":["nosuid","noexec","nodev"]},{"destination":"/dev/shm","type":"tmpfs","source":"shm","options":["nosuid","noexec","nodev","mode=1777"]}],"linux":{"resources":{},"namespaces":[{"type":"mount"},{"type":"network"},{"type":"uts"},{"type":"pid"},{"type":"ipc"}],"maskedPaths":["/proc/asound","/proc/acpi","/proc/kcore","/proc/keys","/proc/latency_stats","/proc/timer_list","/proc/timer_stats","/proc/sched_debug","/proc/scsi","/sys/firmware"],"readonlyPaths":["/proc/bus","/proc/fs","/proc/irq","/proc/sys","/proc/sysrq-trigger"]},"windows":{"layerFolders":["C:\\ProgramData\\Docker\\lcow\\ce5656471f1972d1fb065863f34ca3d631c431f3fad97d77ff6fa6fd4bfb5f9b","C:\\ProgramData\\Docker\\lcow\\73c2b07e7168f0701ed45af4933f92ce9888a0b556014fd267d2d8f473618ed2","C:\\ProgramData\\Docker\\lcow\\7cb5da704e42d7ad33560322d8a69a2f59811501f288e5c7c238be792678a205","C:\\ProgramData\\Docker\\lcow\\284964a9ea03c7b4c044df2c54737000596d144a32c200a5dcf1d6a520abfbc5","C:\\ProgramData\\Docker\\lcow\\2657b3efd2cebb278566ff8a523aa8c9d7b83a841de626f461b5513a55b66092","C:\\ProgramData\\Docker\\lcow\\b1b031fdab0ab0279f303a8540a7eb04cf27bae2a08b1a8d82640b2c3601e79c","C:\\ProgramData\\Docker\\lcow\\5b8011f4e4693c51cd42000cc6afc6e127173f63a1746ee6c942487fe634d605","C:\\ProgramData\\Docker\\lcow\\2500e1d0c12cc21339dd2ae544337a08c38581afe160e0f7dd1bd4693f7378c3","C:\\ProgramData\\Docker\\lcow\\47cf6329ea2fe7487245547f3f816a3a12b7d3901e4b65358814ae0f3cd36e1a"],"hyperv":{},"network":{"endpointList":["de3394a3-4b6d-4cb9-9e19-04307b13390b"]}}}}.
@S93rastogi
Copy link

@S93rastogi S93rastogi commented Oct 3, 2019

I am quite confused with the strategy of providing the Linux container support on Windows. I have Heard alot about LCOW and WSL to run linux container but I am still not sure why they are not providing the support of Linux Container using MOBY VM as in WIN 10 till things are not stable ( ie till LCOW is no more an experimental thing). There seems to be no fix strategy for the support of Linux containers on server 2016 and 2019. In server 2019 I have no other way except running linux container as LCOW and that is experimental. On server 2016 there is another issue if I run docker for win on it I don't get the support of the latest Docker builds as the new versions of Docker are not supported.
As there no solid way to run LINUX container on server 2016 and server 2019 that always works with no issues.

@Kralizek
Copy link

@Kralizek Kralizek commented Oct 3, 2019

I am quite confused with the strategy of providing the Linux container support on Windows. I have Heard alot about LCOW and WSL to run linux container but I am still not sure why they are not providing the support of Linux Container using MOBY VM as in WIN 10 till things are not stable ( ie till LCOW is no more an experimental thing). There seems to be no fix strategy for the support of Linux containers on server 2016 and 2019. In server 2019 I have no other way except running linux container as LCOW and that is experimental. On server 2016 there is another issue if I run docker for win on it I don't get the support of the latest Docker builds as the new versions of Docker are not supported.
As there no solid way to run LINUX container on server 2016 and server 2019 that always works with no issues.

I would dare to say that the general perception is to use linux servers for linux containers (in production). LCOW and Moby are more developer tools than production tools.

@S93rastogi
Copy link

@S93rastogi S93rastogi commented Oct 3, 2019

I am quite confused with the strategy of providing the Linux container support on Windows. I have Heard alot about LCOW and WSL to run linux container but I am still not sure why they are not providing the support of Linux Container using MOBY VM as in WIN 10 till things are not stable ( ie till LCOW is no more an experimental thing). There seems to be no fix strategy for the support of Linux containers on server 2016 and 2019. In server 2019 I have no other way except running linux container as LCOW and that is experimental. On server 2016 there is another issue if I run docker for win on it I don't get the support of the latest Docker builds as the new versions of Docker are not supported.
As there no solid way to run LINUX container on server 2016 and server 2019 that always works with no issues.

I would dare to say that the general perception is to use linux servers for linux containers (in production). LCOW and Moby are more developer tools than production tools.

One could not directly say that running Linux containers on Windows Environment is a wrong Strategy. Windows itself is putting efforts in this direction but the problem is that things are not clear. We also have client constraints that puts us to Windows Route

@BretFisher
Copy link

@BretFisher BretFisher commented Oct 3, 2019

@PatrickLang
Copy link
Author

@PatrickLang PatrickLang commented Oct 15, 2019

just a quick note - I haven't been tracking status of this since I changed teams early last year and am focusing on Kubernetes work. I'm looking for someone to give an update on what's next.

Personally I'm quite happy using WSL2 for development since I can build, test and run Linux code there including Moby, ContainerD and Kubernetes.

@thaJeztah thaJeztah changed the title Epic: Linux Containers on Windows Epic: Linux Containers on Windows (LCOW) Oct 24, 2019
@oldium
Copy link

@oldium oldium commented Nov 3, 2019

There was no update to vendor.conf (opengcs, hcsshim) for 7 months now. Is there anything to show to public, please?

@PatrickLang
Copy link
Author

@PatrickLang PatrickLang commented Feb 26, 2020

Linux Containers on Windows development moved to ContainerD, and no more work is happening in the Moby codebase. I don't think we need to keep this epic open any longer since there's no progress tracked here.

cc @mamezgeb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.