Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows port of Docker Client #9113

Merged
merged 10 commits into from Nov 15, 2014
Merged

Windows port of Docker Client #9113

merged 10 commits into from Nov 15, 2014

Conversation

@ahmetb
Copy link
Contributor

@ahmetb ahmetb commented Nov 12, 2014

Docker CLI for Windows

Introduction

This pull request contains the set of changes to compile and get the Docker client working on Microsoft Windows.

Changes were mostly around pkg/term to get the docker attach and other TTY-involving parts working on Windows terminals, pkg/archive and utils which rely on Unix-specfic calls or structures like stat and other things like home directory or path separators.

We haven't gotten the CLI integration tests running on Windows, yet, therefore functionality is mostly verified manually (more about this below). There are known bugs, which we are looking forward to fix very soon.

Since some of the client code uses packages which contains daemon code (and therefore closely coupled to Linux), we had to provide alternative implementations to those using Go build tags. Those functions mostly return errors when they are called because they shouldn't be called on client code paths.

Running make cross now produces 386/amd64 versions of docker.exe!

Known issues

  • Integration-CLI tests not executed: Most of this test suite assumes
    the daemon is running on the same machine as the client (not possible on Windows, yet)
    and uses Unix sockets to verify the results. These need to be skipped, where
    applicable, or these tests should be converted to use the Remote API over HTTP
    instead of using Unix sockets.
  • Path issues: Due to the difference between Unix and Windows path separators,
    functionality involving Dockerfile and .dockerignore (e.g. docker build) are not
    yet working properly.
  • ANSI.SYS dependency for TTY: ANSI escape sequences used by TTY terminal code
    (e.g. docker attach) are not natively working in the Windows Command Line (cmd.exe)
    or PowerShell. The reason is, ANSI.SYS has not been shipping since Windows 7.
    However, other terminal emulators like ConEmu
    or ANSI.SYS alternatives like ansicon
    solve this problem. We will be searching for solutions that work out-of-the box.

Signed-off-by: Ahmet Alp Balkan ahmetb@microsoft.com
Signed-off-by: John Gossman johngos@microsoft.com

@tiborvass
Copy link
Collaborator

@tiborvass tiborvass commented Nov 12, 2014

@ahmetalpbalkan ❤️ !

)

const SIGCHLD = syscall.SIGCHLD
const SIGWINCH = syscall.SIGWINCH
Copy link
Collaborator

@tiborvass tiborvass Nov 12, 2014

these files should go to pkg/signal

Copy link
Contributor Author

@ahmetb ahmetb Nov 12, 2014

@tiborvass pkg/signal is used mostly for looking up signals by name and forwarding signals. In api/client case (monitorTtySize & exclusion of SIGCHLD in forwardAllSignals), we just need those specific signals. And in Windows case, these two are not available on Windows, therefore we provide invalid signals on _windows implementation. Should it still go there?

Copy link
Collaborator

@tiborvass tiborvass Nov 13, 2014

@ahmetalpbalkan Yes it should still go to pkg/signal. We're already importing it anyways in api/client.

@ahmetb ahmetb closed this Nov 12, 2014
@ahmetb ahmetb deleted the windows-client branch Nov 12, 2014
@ahmetb ahmetb reopened this Nov 12, 2014
@SvenDowideit
Copy link
Contributor

@SvenDowideit SvenDowideit commented Nov 12, 2014

hey! 🎂 🎆 🎉

@ahmetb
Copy link
Contributor Author

@ahmetb ahmetb commented Nov 12, 2014

@tiborvass @SvenDowideit any ideas about this specific failure https://ci.dockerproject.com/github.com/docker/docker/commit/6f0d5b1e8dca2e4411858907a274cff7c7cd0468?branch=windows-client this randomly happens 50% of time we trigger a make test-unit (e.g. I just got a test run passing locally on my machine).

EDIT: turns out this is a bug we introduced, debugging...

@jessfraz
Copy link
Contributor

@jessfraz jessfraz commented Nov 12, 2014

Wow

@huslage
Copy link
Contributor

@huslage huslage commented Nov 12, 2014

Happy to have you guys!

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Nov 12, 2014

Wow, just wow! Really awesome job

Next step...native fig and my colleagues will be very happy!

@mgpatel75
Copy link

@mgpatel75 mgpatel75 commented Nov 12, 2014

WoW. Awesome.

@pmario
Copy link

@pmario pmario commented Nov 12, 2014

So the help text "A self-suffictient runtime for linux containers" will need to be more generic soon :)

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Nov 12, 2014

So the help text "A self-suffictient runtime for linux containers" will need to be more generic soon :)

Good catch! I was already expecting SvenDowideit starting with "and where are the docs?" 😉

@afolarin
Copy link

@afolarin afolarin commented Nov 12, 2014

Cool. This seems to be moving very quickly

@crosbymichael
Copy link
Contributor

@crosbymichael crosbymichael commented Nov 12, 2014

It looks like there are test errors in the unit tests with this PR. Can you please look into those?

@ahmetb
Copy link
Contributor Author

@ahmetb ahmetb commented Nov 12, 2014

@crosbymichael working on that right now, error happens doesn't repro all the time, has something either with granularity of mtim on OS/FS or flushing changes to fs.

@unclejack
Copy link
Contributor

@unclejack unclejack commented Nov 13, 2014

@ahmetalpbalkan You'll see some comments on the code from me. These can be addressed after we sort out the archive issue.

@@ -4,6 +4,7 @@
.vagrant*
bin
docker/docker
docker/docker.exe
Copy link
Contributor

@unclejack unclejack Nov 13, 2014

Is this actually needed? I think just adding docker.exe or *.exe without any prefix would be better.

Copy link
Contributor Author

@ahmetb ahmetb Nov 13, 2014

docker/docker here is specifically intended for ignoring the binary product of go build ran on docker/ directory. Same output will be docker.exe on Windows. however *.exe might be checked in to somewhere at a later point, I don't want to ignore that. It is better to keep the intention same by extending this specific ignore rule.

"syscall"
)

func FromStat_t(s *syscall.Stat_t) *stat {
Copy link
Contributor

@unclejack unclejack Nov 13, 2014

This should be FromStatT.

Copy link
Contributor Author

@ahmetb ahmetb Nov 13, 2014

fixed as fromStatT

@unclejack
Copy link
Contributor

@unclejack unclejack commented Nov 13, 2014

@ahmetalpbalkan Can you please make https://github.com/Azure/docker/commit/6cf258e53479849b90081dd0f7e6277b9f8c35f0 an individual PR and add the explicit build tags in a new commit?

The changes to utils should also be in their own separate PR with explicit build tags. There should be 2 commits in that PR: one for tmpdir and one for TreeSize.

Doing this would help us speed up the process of reviewing and merging this PR. Reviewing smaller PRs is easier.

@ahmetb
Copy link
Contributor Author

@ahmetb ahmetb commented Nov 13, 2014

@triajoern same as darwin (OS X), arm is most likely not a valid use case for Windows. (830b5c0) Windows ARM support is for Surface RTs and Windows Phones and these neither runs arbitrary executables like this nor used as development devices. Do you have a need for windows/arm?

@ahmetb
Copy link
Contributor Author

@ahmetb ahmetb commented Nov 13, 2014

@triajoern this PR is not merged yet, therefore the repo you're cloning won't have the Windows client. Please checkout the branch this PR is sent from and then run "make cross".

@unclejack
Copy link
Contributor

@unclejack unclejack commented Nov 13, 2014

@ahmetalpbalkan As the co-maintainer of pkg/archive, this PR gets NOT LGTM from me for now. I've tried to debug the archive bug introduced by this commit https://github.com/Azure/docker/commit/e732b79376a163fda94619531d13791fafa94038, but tracking it down to help fix this PR has proven to be too complicated because so many unrelated changes are bundled together.

@ahmetb
Copy link
Contributor Author

@ahmetb ahmetb commented Nov 13, 2014

@unclejack we fixed the bug that was breaking the unit tests later tonight and git push -f'ed here. Would separating it into different PRs help?

@unclejack
Copy link
Contributor

@unclejack unclejack commented Nov 13, 2014

@triajoern Please wait until this gets merged. We don't provide support for trying out code from pull requests.

@unclejack
Copy link
Contributor

@unclejack unclejack commented Nov 13, 2014

@ahmetalpbalkan Please take a look at https://github.com/unclejack/docker/commits/split_up_pr_9113.
This is what has been done on that branch based on this PR:

  • 929e700206750153e73d0a0838a6ae16a47f7783 "Consolidate tmpdir implementations" > squashed into e732b79376a163fda94619531d13791fafa94038 "Refactor packages to compile client on Windows" so that we avoid making a change which gets reverted in a new commit
  • go fmt fixes were squashed into e732b79376a163fda94619531d13791fafa94038 (for utils) and 26f9c49daaa2b294489e112be46762c3981d1f54 (for pkg/term) so that we avoid changing the same code multiple times

This is what still needs to be done:

  • stat and lstat need to go into pkg/system as a separate commit, not into pkg/archive
  • the archive changes need to be in their own commit which comes after the pkg/system lstat+stat commit to make archive switch to the new stat
  • api/client/signals needs to move to pkg/signal and this needs to be a separate commit
  • the utils tmpdir change should be in its own commit
  • the utils TreeSize change should be in its own commit
  • explicit build tags need to be added
  • the changes I've made to my branch need to be included in this PR as well

edit: The branch I've mentioned is only meant to be an example, it serves no other purpose beyond that. All of the changes need to be made to this PR.

@ahmetb
Copy link
Contributor Author

@ahmetb ahmetb commented Nov 13, 2014

@unclejack This sounds good, thanks for helping. I will fix those and split into multiple PRs as you said.

@rghv
Copy link
Contributor

@rghv rghv commented Nov 19, 2014

👍 👏

Hopefully, in years to come, windows would by default have a posix compliant shell as well! (though there's an addon , it doesn't seem to be too popular)

@subfuzion
Copy link

@subfuzion subfuzion commented Nov 21, 2014

This is great and will improve the developer experience for our Window devs who chafe at having to log into a linux shell to use the CLI. I imagine using the CLI with a boot2docker host on the developer machine will be a common use case.

We're running into the following problem, but before I submit an actual issue, I was just wondering if anyone else has seen this and if there is a straightforward workaround. Has anyone been able to successfully connect the client to a boot2docker host on Windows? I'm getting an error responsse from the daemon: client and server don't have the same version (client : 1.16, server: 1.15).

@ahmetb
Copy link
Contributor Author

@ahmetb ahmetb commented Nov 21, 2014

@tonypujals regardless of boot2docker, you are running a more recent version than Docker 1.3.0 (because you're compiling from master branch) and its API version is bumped to "v1.16". You should either edit that string and recompile, or set up docker host side by compiling binaries from master branch. You can find more about this and ask questions here.

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Nov 21, 2014

client and server don't have the same version (client : 1.16, server: 1.15)

It means that the version of docker running inside the boot2docker VM is older than the Windows client. So you'll have to install the "nightly" build inside the boot2docker VM.

Hope this helps.

@tiborvass
Copy link
Collaborator

@tiborvass tiborvass commented Nov 21, 2014

Guys, I know everybody is excited about this PR being merged and the direction this is going, but can we please continue this conversation on IRC or the mailing list? Let's keep in mind, that every comment results in 1,358 mails being sent. Thanks!

EDIT: For any new Windows fans who are joining now: you are welcome! and please join on the freenode IRC #docker if you're a Docker user and #docker-dev if you want to contribute or know more about the ongoing development. The mailing lists are at https://groups.google.com/forum/#!forum/docker-user for users and https://groups.google.com/forum/#!forum/docker-dev for docker development related things.

@markl17
Copy link

@markl17 markl17 commented Dec 2, 2014

how do i resolve this problem

C:\Go\src\pkg\github.com\docker\docker>go build

github.com/docker/docker/pkg/system

pkg\system\meminfo_unsupported.go:6: undefined: ErrNotSupportedPlatform
pkg\system\stat_unsupported.go:7: undefined: syscall.Stat_t
pkg\system\stat_unsupported.go:12: undefined: syscall.Stat_t
pkg\system\stat_unsupported.go:15: undefined: syscall.Stat_t
pkg\system\utimes_unsupported.go:8: undefined: ErrNotSupportedPlatform
pkg\system\utimes_unsupported.go:12: undefined: ErrNotSupportedPlatform
pkg\system\xattrs_unsupported.go:6: undefined: ErrNotSupportedPlatform
pkg\system\xattrs_unsupported.go:10: undefined: ErrNotSupportedPlatform

github.com/docker/docker/pkg/term

pkg\term\term.go:16: undefined: Termios

github.com/docker/libcontainer/devices

..\libcontainer\devices\devices.go:73: undefined: syscall.Stat_t

@mattn
Copy link

@mattn mattn commented Dec 26, 2014

Hi, if you want, I can contribute ansi escape sequence for docker client on windows.
For example below is a code that works as ttyrec/ttyplay on windows.

https://github.com/mattn/ttyrec4windows/blob/master/ttyplay/ttyplay.go

But currently I don't know how to do for working docker.exe on my PC.

@markl17
Copy link

@markl17 markl17 commented Dec 26, 2014

if you want i can email you a working docker.exe

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Dec 26, 2014

If you need a windows exe, the official builds of master can be found here;
https://master.dockerproject.com

@mattn
Copy link

@mattn mattn commented Dec 27, 2014

Ah, Sorry. I had mistake to write comment. I could build it already. I don't know how to use docker.exe on local PC.

@so0k
Copy link

@so0k so0k commented Jan 6, 2015

Do volume mounts work from windows client?

on my CoreOS - Hyper-V machine:

~$ docker version
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8-dirty
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8-dirty

from windows 8.1 64bit, using ansicon and with latest docker build from https://master.dockerproject.com/

D:\prj\drfquickstart>ansicon
Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

D:\prj\drfquickstart>docker version
Client version: 1.4.1-dev
Client API version: 1.16
Go version (client): go1.4
Git commit (client): 7eb5262
OS/Arch (client): windows/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8-dirty

trying to use volume mount with Windows path notation:

D:\prj\drfquickstart>docker run -it --rm -v D:\prj\drfquickstart:/usr/src/app/ python:3 bash
invalid value "D:\\prj\\drfquickstart:/usr/src/app/" for flag -v: \prj\drfquickstart:/usr/src/app/ is not an absolute path

trying to use volume mount with unix path notation:

D:\prj\drfquickstart>dir /B
db.sqlite3
manage.py
pip-cache
quickstart.sublime-project
quickstart.sublime-workspace
tutorial

D:\prj\drfquickstart>docker run -it --rm -v /D/prj/drfquickstart/:/usr/src/app/ python:3 bash
root@006dc57efb28:/usr/src/python# ls -l /usr/src/app/
total 0

if I run a privileged container - share the windows folder & install cifs-utils in my dev box - I can mount the windows folder... so I can work around, but... any plans on volume mounts?

I'm not using virtualbox or guest additions to mount host folders, I've tried running a containerized samba server to share folders from the VM - at which point I have to either store the files within a VM or run folder sync tools between my host and my VM. I'm not a fan of any of these approaches

@SvenDowideit
Copy link
Contributor

@SvenDowideit SvenDowideit commented Jan 6, 2015

no, volume mounts are only local to the Docker daemon host - and using the windows docker client will not create any links to do this for you.

in the boot2docker management tool, we create a vbox share of c:\Users that is mounted into the Docker host, and thus can be used - you'd need to create something similar manually in hyper-v, and then bind mount using unix paths.

using containerised network mounts won't work, as they're not recusivly bind mounted into other containers via the volumes (I tried to make it work, but failed - i'm not sure that you can tbh)

the most reliable solution I've had, is to mount the windows dir into the host, and then use that bind mount. And I'm probably going to do more work in b2d to try out a few things we can benchmark, and then chose what goes into docker-machine.

@so0k
Copy link

@so0k so0k commented Jan 6, 2015

Ok, thank you for your reply.

in that case, I'm not sure if the docker windows client can give me any advantage over putty for containerized development of Linux applications? Until there is a windows docker deamon for windows application development.

given the different environment, I assume it is also not possible to do "docker in docker" by mounting docker bin & socket from windows client

as for my work around: I currently have docker images with cifs-utils, while still manually mounting the windows shares, I will try to automate it similar to this: http://chapeau.freevariable.com/2014/08/docker-uid.html

I prefer not to install cifs-utils on the host server (which is running the docker deamon) and as such keep cifs-utils within a docker container itself...

@SvenDowideit
Copy link
Contributor

@SvenDowideit SvenDowideit commented Jan 6, 2015

note that a windows docker daemon will only be useful for containerising native windows applications.

I personally find the docker native osx and windows clients very useful - as they allow me to run a docker build directly from my local host, which pushes all the source code from windows to the container, and then I run the container on the linux box.

Note that my workflow does not bind-mount (the -v /hostdir:/containerdir) at all, as that won't work in a performant way for any remote Docker daemon - and some of mine are out in the cloud.

I don't understand why you'd need to install cifs-tools - last time I worked on this, I thought mount -t cifs ... worked with no changes on boot2docker (and tbh, thats the most likely type of solution we'll persue in the shorter term - the remove volumes proposal seems not to be proceeding yet.

@so0k
Copy link

@so0k so0k commented Jan 7, 2015

yes, I noted that the windows docker deamon would only be useful for windows application development in my reply :)

I can see that building containers is easier with the native windows docker client, but having a containerized development environment is not.

Re: b2d - When on Windows 8.1, Hyper-V prevents the usage of Virtual Box / VMWare player. I can run boot2docker on Hyper-V.. However I would still have to learn all the pitfalls of Tiny Core Linux and won't be able to use VB Guest additions etc.... I invested time in learning CoreOS (prior to the Rocket announcement) and prefer to use a local CoreOS instance for app development as it matches the server I deploy to (for now).

I install cifs tools in my containers (FROM python:3) to mount & work on code (./manage.py syncdb ...) - not in my host (CoreOS)

@stuaxo
Copy link

@stuaxo stuaxo commented Feb 17, 2015

Re: ANSI.SYS - Maybe you could have a word with the Windows team and get this supported for Windows 11 if not 10 ?

@ksmutny
Copy link

@ksmutny ksmutny commented Feb 23, 2015

Please, is there an archive of official builds of Windows client? The builds on https://master.dockerproject.com are nightly and got out of sync with boot2docker. I have a boot2docker in Hyper-V (on Windows 8.1) with API version 1.17, the official Windows client build has API version 1.18. I want to avoid compiling the client myself. Thanks in advance for any help.

@ahmetb
Copy link
Contributor Author

@ahmetb ahmetb commented Feb 23, 2015

@ksmutny there is no official build of Windows CLI yet. We are planning to release one for Docker 1.6.0.

@ksmutny
Copy link

@ksmutny ksmutny commented Feb 23, 2015

@ahmetalpbalkan Ok then, thanks. I compiled it according to your instructions and it works like a charm :)

@markl17
Copy link

@markl17 markl17 commented Feb 23, 2015

you can always compile your own docker with go compiler

@BGaudreault-EDR
Copy link

@BGaudreault-EDR BGaudreault-EDR commented Feb 24, 2015

Hi, not sure if this is the [Docker repo] you mentioned in ahmetalpbalkan.com/blog/compiling-docker-cli-on-windows, but I'm getting the following error installing on a 32-bit OS (hope that's supported!):

"..\utils\utils.go:22:2: cannot find package "github.com/docker/docker/autogen/do
ckerversion" in any of:
C:\Go\src\github.com\docker\docker\autogen\dockerversion (from $GOROOT)
C:\Go\src\github.com\docker\docker\autogen\dockerversion (from $GOPATH)
C:\Go\src\github.com\docker\docker\vendor\src\github.com\docker\docker\a
utogen\dockerversion"
cc: @ahmetalpbalkan

@ahmetb
Copy link
Contributor Author

@ahmetb ahmetb commented Feb 24, 2015

@BGaudreault this comment should go to the blog post's comment section; not here.

@petersellars
Copy link

@petersellars petersellars commented Mar 19, 2015

When trying to build as per https://ahmetalpbalkan.com/blog/compiling-docker-cli-on-windows/ I am getting the following error:

E:\Projects\docker-windows\src\github.com\docker\docker\docker>go build -v
..\utils\utils.go:22:2: cannot find package "github.com/docker/docker/autogen/dockerversion" in any of:
    C:\Go\src\github.com\docker\docker\autogen\dockerversion (from $GOROOT)
    E:\Projects\docker-windows\src\github.com\docker\docker\autogen\dockerversion (from $GOPATH)
    E:\Projects\docker-windows\src\github.com\docker\docker\vendor\src\github.com\docker\docker\autogen\dockerversion

Am new to Go and wondering if this is path related or environment related.

@ahmetalpbalkan - I have commented on the blog post, as this is an issue with the compilation was not sure if this should be posted here too....apologies if this is not the place for this!

@tiborvass
Copy link
Collaborator

@tiborvass tiborvass commented Mar 20, 2015

Hey everyone, if you have docker-related questions, feel free to reach out on the docker-user mailing list instead of commenting on this merged PR, that'd be much appreciated, thanks!

@moby moby locked and limited conversation to collaborators Mar 20, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet