Docker: support using from non-x86_64-linux platforms

[borsboom]

This will require downloading an x86_64-linux stack binary to bind-mount into the container, and will probably require some changes to bind-mounted path handling on Windows. It also requires a VM with specific settings to make the user ID mapping and project bind-mounting work.

Note: I personally use stack with docker from OS X already (but manually copy in the x86-64-linux binary in). I have a Vagrantfile that isn't too far from being publishable.

[rrnewton]

Is that vagrant file ready perhaps ;-)?

I was reluctant to set up a full sized linux VM for launching stack/docker (too much redundancy). Thus I wondered if it would be possible to run the linux stack executable inside boot2docker directly on Mac OS.

The boot2docker image itself is some kind of core linux (so says /etc/issue). After getting into it with:

boot2docker ssh

I could then scp or wget a copy of the stack executable. And further, I grabbed a 64 bit version of libgmp.so.3 into the current directory, then tried running stack. The problem is, surprisingly, it ran out of memory and crashed. The VM has 2 gigs of mem. If I run stack with a restricted heap size, I get a heap exhausted error as expected:

$ LD_LIBRARY_PATH=. ./stack-0.1.2.7 --help +RTS -A512K -N1 -M512M
stack-0.1.2.7: Heap exhausted;
Current maximum heap size is 536870912 bytes (512 MB);
use `+RTS -M<size>' to increase it.

Is this legit? Does stack need >2G mem? Or is this some problem with libgmp or something else...

[borsboom]

stack definitely shouldn't need > 2 GB memory, and I use it all the time on a VM with only 1 GB (to build itself).

[borsboom]

This turns out even to be a problem when the host and guest have both run x86_64-linux, but have different system libraries. Copying part of my comment from here:

This bind-mounting of stack's own executable in a container is tricky business and I'm not entirely happy with it. It's especially problematic when the host's platform is significantly different than the guest's (especially with a different operating system where it doesn't work at all, but as we see here it can even be an issue with different Linux distributions). In the the past, we tried "baking" a stack executable into the images, but then there were problems when the host-stack's version was out of sync with guest-stack's. An option we've considered is having the container download a binary that matches the host's version, but that wouldn't play nicely with building unreleased versions from source. Maybe in that case it could do an in-container build of stack from the same commit ID. But none of these options are ideal, and it hasn't become clear yet which direction we should go in

[borsboom]

Two upstream issues (at least) are blocking the possibility of having this "just work" out of the box:

• Volume mounting with boot2docker/Docker Machine is really slow (Volume sharing performance issues boot2docker/boot2docker#593). This can be worked around using Vagrant and NFS sharing.

[borsboom]

With bf71d04 and 8f35021, we're a bit closer to supporting this. I've also cleaned up, documented, and published the Vagrantfile that I use to run Stack w/ Docker on Mac OS X: https://github.com/borsboom/vagrant-stack-docker. This is not "officially supported" in any way, but perhaps it will be helpful to some.

[borsboom]

moby/moby#16289 made it into Docker 1.9.1, and does indeed seem to fix the problem where interactive containers "hang" on exit.

[songpp]

Windows 10 + latest Docker ToolBox

spp@april MINGW64 /d/dev/haskell/stack-playground/stack-playground
$ stack build
Downloading Docker-compatible stack executable
Downloaded stack-1.0.2.
Running C:\Program Files\Git\usr\bin\tar.exe xf C:\Users\spp\AppData\Local\Programs\stack\x86_64-linux\stack-1.0.2.tar.gz --strip-components 1 in directory C:\Users\spp\AppData\Local\Programs\stack\x86_64-linux\stack-1.0.2\ exited with ExitFailure 128

/usr/bin/tar: Cannot connect to C: resolve failed

I thought it's the Windows Drive C: issue, but nonetheless, the binary in stack-1.0.2.tar.gz cannot be executed on windows:

spp@april MINGW64 /d/dev/haskell/stack-playground/stack-playground
$ stack build
bash: /d/dev/tools/stack: cannot execute binary file: Exec format error

[sacundim]

Well, just out of curiosity I tried building a project of mine under OS X with Docker. And it worked just fine! All I did was:

1. Create a default Docker Machine in the standard way (docker-machine create --driver virtualbox default)
2. Export environment variables (eval $(docker-machine env))
3. Enable Docker build support in stack.yaml
4. stack test and stack bench

I did not time the build, which is something that would be worth looking into.

Versions:

$ stack --version
Version 1.1.2 x86_64 hpack-0.14.0

$ docker --version
Docker version 1.11.1, build 5604cbe

$ docker-machine --version
docker-machine version 0.7.0, build a650a40

$ sw_vers 
ProductName:    Mac OS X
ProductVersion: 10.10.4
BuildVersion:   14E46

Note however that Docker's Mac and Windows integration is planned to change a lot, so it might be best to wait and see on this front.

EDIT: Well, I went ahead and timed it a bit. But this is the timing of running builds and test suites after all dependencies have been pre-built. Without Docker this takes 37 seconds, of which 3.8 is spent executing the tests:

$ stack clean && time stack test
[compilation...]
[unit tests...]
Finished in 3.8167 seconds 
46 examples, 0 failures

Completed 2 action(s).

real    0m37.334s
user    0m33.258s
sys 0m3.252s

With Docker it takes 44 seconds, 3.6 of which is spent running tests:

$ stack clean && time stack test
[compilation...]
[unit tests...]
Finished in 3.5859 seconds
46 examples, 0 failures


real    0m43.680s
user    0m0.138s
sys 0m0.065s

[borsboom]

With large projects it makes a much bigger difference going through Virtualbox's synced folders (just figuring out whether there's any work to do can take a long time). I'm keeping an eye on developments with Docker for Mac and Docker for Windows. So far I've had trouble with Docker for Mac so awaiting for it to stabilize a bit more. I know Stack won't work with Docker on Windows because it doesn't do anything to handle the different path format on host vs. in the container, but we'd like to fix that as well.

[borsboom]

I've tested Stack with the recently out of beta Docker for Mac and it appears to work seamlessly. Only trouble I ran into was that building a really big project that uses a huge (> 10 GB) Docker image caused the Docker Engine to hang, but I think that's a Docker issue and has nothing to do with Stack itself.

[borsboom]

We now have a separate issue for the Windows case: #2421

I think we can now say that OS X support is working, and the OS X approach should work for any other non-Linux Un*x that gets first-class Docker support (or by using an approach like my vagrant-stack-docker), so my feeling is that any remaining non-Windows issues aren't really Stack specific but just upstream Docker issues.

Any objection to closing this issue?

[mgsloan]

No objection!

[deiga]

@borsboom Would you like to elaborate as to what you did to make it work? My fresh install of Docker For Mac creates the following error for me:
https://gist.github.com/deiga/da4e553662a13ad76553e7ea947eafe0