New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker: support using from non-x86_64-linux platforms #194

Closed
borsboom opened this Issue Jun 5, 2015 · 13 comments

Comments

Projects
None yet
7 participants
@borsboom
Contributor

borsboom commented Jun 5, 2015

This will require downloading an x86_64-linux stack binary to bind-mount into the container, and will probably require some changes to bind-mounted path handling on Windows. It also requires a VM with specific settings to make the user ID mapping and project bind-mounting work.

Note: I personally use stack with docker from OS X already (but manually copy in the x86-64-linux binary in). I have a Vagrantfile that isn't too far from being publishable.

@borsboom borsboom self-assigned this Jun 5, 2015

@borsboom borsboom changed the title from Docker: support from non-x86_64-linux platforms to Docker: support using from non-x86_64-linux platforms Jun 5, 2015

@snoyberg snoyberg added this to the Third release milestone Jun 6, 2015

@borsboom borsboom removed their assignment Jun 9, 2015

@snoyberg snoyberg modified the milestones: Later improvements, First stable release (0.1.0.0?) Jun 9, 2015

@borsboom borsboom modified the milestones: Docker, Later improvements Jul 22, 2015

@rrnewton

This comment has been minimized.

Contributor

rrnewton commented Aug 4, 2015

Is that vagrant file ready perhaps ;-)?

I was reluctant to set up a full sized linux VM for launching stack/docker (too much redundancy). Thus I wondered if it would be possible to run the linux stack executable inside boot2docker directly on Mac OS.

The boot2docker image itself is some kind of core linux (so says /etc/issue). After getting into it with:

boot2docker ssh

I could then scp or wget a copy of the stack executable. And further, I grabbed a 64 bit version of libgmp.so.3 into the current directory, then tried running stack. The problem is, surprisingly, it ran out of memory and crashed. The VM has 2 gigs of mem. If I run stack with a restricted heap size, I get a heap exhausted error as expected:

$ LD_LIBRARY_PATH=. ./stack-0.1.2.7 --help +RTS -A512K -N1 -M512M
stack-0.1.2.7: Heap exhausted;
Current maximum heap size is 536870912 bytes (512 MB);
use `+RTS -M<size>' to increase it.

Is this legit? Does stack need >2G mem? Or is this some problem with libgmp or something else...

@borsboom

This comment has been minimized.

Contributor

borsboom commented Aug 8, 2015

stack definitely shouldn't need > 2 GB memory, and I use it all the time on a VM with only 1 GB (to build itself).

borsboom referenced this issue in archhaskell/habs Sep 10, 2015

Upgrades 2015-09-06
    cassava
    contravariant
    fclabels
    iproute
    stack
    test-framework-hunit
    yaml

Added dependencies

    ignore
    pcre-heavy
    base-compat
    pcre-light
    string-conversions

Signed-off-by: Magnus Therning <magnus@therning.org>
@borsboom

This comment has been minimized.

Contributor

borsboom commented Sep 10, 2015

This turns out even to be a problem when the host and guest have both run x86_64-linux, but have different system libraries. Copying part of my comment from here:

This bind-mounting of stack's own executable in a container is tricky business and I'm not entirely happy with it. It's especially problematic when the host's platform is significantly different than the guest's (especially with a different operating system where it doesn't work at all, but as we see here it can even be an issue with different Linux distributions). In the the past, we tried "baking" a stack executable into the images, but then there were problems when the host-stack's version was out of sync with guest-stack's. An option we've considered is having the container download a binary that matches the host's version, but that wouldn't play nicely with building unreleased versions from source. Maybe in that case it could do an in-container build of stack from the same commit ID. But none of these options are ideal, and it hasn't become clear yet which direction we should go in

@borsboom

This comment has been minimized.

Contributor

borsboom commented Oct 8, 2015

Two upstream issues (at least) are blocking the possibility of having this "just work" out of the box:

  • Volume mounting with boot2docker/Docker Machine is really slow (boot2docker/boot2docker#593). This can be worked around using Vagrant and NFS sharing.
  • ~~~Interactive containers "hang" on exit until you hit enter on Mac OS X. Might be moby/moby#11957. We'll see if the recent PR (moby/moby#16289) fixes it, once a new version of the Docker client is released that includes it. A workaround is to set run-args: ["--interactive=false"] in the config, but that means you can't interact with processes running in the container.~~~ (see #194 (comment))

borsboom added a commit that referenced this issue Oct 8, 2015

@borsboom

This comment has been minimized.

Contributor

borsboom commented Oct 8, 2015

With bf71d04 and 8f35021, we're a bit closer to supporting this. I've also cleaned up, documented, and published the Vagrantfile that I use to run Stack w/ Docker on Mac OS X: https://github.com/borsboom/vagrant-stack-docker. This is not "officially supported" in any way, but perhaps it will be helpful to some.

@borsboom

This comment has been minimized.

Contributor

borsboom commented Nov 22, 2015

moby/moby#16289 made it into Docker 1.9.1, and does indeed seem to fix the problem where interactive containers "hang" on exit.

@songpp

This comment has been minimized.

songpp commented Feb 15, 2016

Windows 10 + latest Docker ToolBox

spp@april MINGW64 /d/dev/haskell/stack-playground/stack-playground
$ stack build
Downloading Docker-compatible stack executable
Downloaded stack-1.0.2.
Running C:\Program Files\Git\usr\bin\tar.exe xf C:\Users\spp\AppData\Local\Programs\stack\x86_64-linux\stack-1.0.2.tar.gz --strip-components 1 in directory C:\Users\spp\AppData\Local\Programs\stack\x86_64-linux\stack-1.0.2\ exited with ExitFailure 128

/usr/bin/tar: Cannot connect to C: resolve failed

I thought it's the Windows Drive C: issue, but nonetheless, the binary in stack-1.0.2.tar.gz cannot be executed on windows:

spp@april MINGW64 /d/dev/haskell/stack-playground/stack-playground
$ stack build
bash: /d/dev/tools/stack: cannot execute binary file: Exec format error
@sacundim

This comment has been minimized.

sacundim commented Jun 10, 2016

Well, just out of curiosity I tried building a project of mine under OS X with Docker. And it worked just fine! All I did was:

  1. Create a default Docker Machine in the standard way (docker-machine create --driver virtualbox default)
  2. Export environment variables (eval $(docker-machine env))
  3. Enable Docker build support in stack.yaml
  4. stack test and stack bench

I did not time the build, which is something that would be worth looking into.

Versions:

$ stack --version
Version 1.1.2 x86_64 hpack-0.14.0

$ docker --version
Docker version 1.11.1, build 5604cbe

$ docker-machine --version
docker-machine version 0.7.0, build a650a40

$ sw_vers 
ProductName:    Mac OS X
ProductVersion: 10.10.4
BuildVersion:   14E46

Note however that Docker's Mac and Windows integration is planned to change a lot, so it might be best to wait and see on this front.

EDIT: Well, I went ahead and timed it a bit. But this is the timing of running builds and test suites after all dependencies have been pre-built. Without Docker this takes 37 seconds, of which 3.8 is spent executing the tests:

$ stack clean && time stack test
[compilation...]
[unit tests...]
Finished in 3.8167 seconds 
46 examples, 0 failures

Completed 2 action(s).

real    0m37.334s
user    0m33.258s
sys 0m3.252s

With Docker it takes 44 seconds, 3.6 of which is spent running tests:

$ stack clean && time stack test
[compilation...]
[unit tests...]
Finished in 3.5859 seconds
46 examples, 0 failures


real    0m43.680s
user    0m0.138s
sys 0m0.065s
@borsboom

This comment has been minimized.

Contributor

borsboom commented Jun 10, 2016

With large projects it makes a much bigger difference going through Virtualbox's synced folders (just figuring out whether there's any work to do can take a long time). I'm keeping an eye on developments with Docker for Mac and Docker for Windows. So far I've had trouble with Docker for Mac so awaiting for it to stabilize a bit more. I know Stack won't work with Docker on Windows because it doesn't do anything to handle the different path format on host vs. in the container, but we'd like to fix that as well.

@borsboom

This comment has been minimized.

Contributor

borsboom commented Aug 4, 2016

I've tested Stack with the recently out of beta Docker for Mac and it appears to work seamlessly. Only trouble I ran into was that building a really big project that uses a huge (> 10 GB) Docker image caused the Docker Engine to hang, but I think that's a Docker issue and has nothing to do with Stack itself.

@borsboom

This comment has been minimized.

Contributor

borsboom commented Aug 4, 2016

We now have a separate issue for the Windows case: #2421

I think we can now say that OS X support is working, and the OS X approach should work for any other non-Linux Un*x that gets first-class Docker support (or by using an approach like my vagrant-stack-docker), so my feeling is that any remaining non-Windows issues aren't really Stack specific but just upstream Docker issues.

Any objection to closing this issue?

@mgsloan

This comment has been minimized.

Collaborator

mgsloan commented Aug 4, 2016

No objection!

@mgsloan mgsloan closed this Aug 4, 2016

@deiga

This comment has been minimized.

deiga commented Aug 26, 2016

@borsboom Would you like to elaborate as to what you did to make it work? My fresh install of Docker For Mac creates the following error for me:
https://gist.github.com/deiga/da4e553662a13ad76553e7ea947eafe0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment