Issue while starting unik daemon in docker-machine setup

[hanneslehmann]

Hello,

I think i successfully have build the unik binary, but I have some issues in starting it. Actually I am on Ubuntu, but running docker in a seperate virtual machine for security reasons, no fancy setup, just using docker-machine. The binary already ended up within the docker-machine "default", I copied it to my local, bare-metal machine, I can execute it by calling:
unik daemon --debug --trace --logfile logs.txt

But then I get the error:

DEBU[0013] util.LogCommand.func2[util/log.go:95]
./gomaincaller.go:20[/tmp/go-build882971369/command-line-arguments/_obj/gomaincaller.cgo1.go:26]: undefined: main 
ERRO[0013] main.main[unik/unik.go:9]
cobra.(*Command).Execute[cobra/command.go:618]
cobra.(*Command).ExecuteC[cobra/command.go:662]
cobra.(*Command).execute[cobra/command.go:572]
cmd.glob.func5[cmd/daemon.go:91]
running daemon failed: [cmd/daemon.go:86] daemon failed to initialize: {[daemon/daemon.go:106] initializing virtualbox provider: {[virtualbox/virtualbox_provider.go:49] deploying virtualbox instance listener: {[virtualbox/deploy_instance_listener.go:39] compiling instance listener source to unikernel: {exit status 2}}}}
[logs.txt](https://github.com/emc-advanced-dev/unik/files/541336/logs.txt)

As I still don't have the understanding how virtualbox, docker, unik are playing together, I don't know where the issue could be.

[hanneslehmann]

Ah, tried with sudo as well! No difference.

Is this the UDP Server listening on 0.0.0.0:9967? Why does it point to Vbox process?

>netstat -ant | grep 99                              
tcp        0      0 0.0.0.0:9967            0.0.0.0:*               LISTEN

>fuser 9967/tcp
9967/tcp:            15014

>ls -l /proc/15014/exe
lrwxrwxrwx 1 root root 0 Okt 20 11:18 /proc/15014/exe -> /usr/lib/virtualbox/VBoxHeadless

[hanneslehmann]

That's the error...

running command  command=[docker run --rm -e ROOT_PATH=github.com/emc-advanced-dev/unik/instance-listener -v /tmp/vbox.instancelistener.713120509:/opt/code --name=8231ae89-8c25-45b7-9eb6-e69bb32b1333 projectunik/compilers-rump-go-hw-no-stub:d8c50ae987531c6b]
DEBU[0010] util.LogCommand.func2[util/log.go:95]
docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.

I can run the docker command manually:
docker run --rm -e ROOT_PATH=github.com/emc-advanced-dev/unik/instance-listener -v /tmp/vbox.instancelistener.068468199:/opt/code --name=4e743eb8-b051-4535-a06b-70fdb958d836 projectunik/compilers-rump-go-hw-no-stub:d8c50ae987531c6b

Which then gives

...
s/_obj/_cgo_main.o $WORK/command-line-arguments/_obj/_cgo_export.o $WORK/command-line-arguments/_obj/gomaincaller.cgo2.o -g -O2
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/command-line-arguments.a -trimpath $WORK -p main -buildid f6f6e98068985043556a7ac99e4760bddefe506e -D _/opt/go/src/github.com/emc-advanced-dev/unik/instance-listener -I $WORK -pack $WORK/command-line-arguments/_obj/_cgo_gotypes.go $WORK/command-line-arguments/_obj/gomaincaller.cgo1.go
# command-line-arguments
./gomaincaller.go:20[/tmp/go-build403694523/command-line-arguments/_obj/gomaincaller.cgo1.go:26]: undefined: main

[hanneslehmann]

Hi, one step further: I had to start the daemon within the default docker-machine and open the port 3000. Now I get following:

unik build --name myImage --path ./ --base rump --language go --provider virtualbox
INFO[0000] running unik build                            args= base=rump force=false host=localhost:3000 language=go mountPoints=[] name=myImage path=./ provider=virtualbox
INFO[0000] App packaged as tarball: /tmp/sources.tar.gz.405379101

ERRO[0000] build failed: [cmd/build.go:105] building image failed: {[client/images.go:63] failed with status 404: 404 page not found
} 

At least I got the App tarball :)

[hanneslehmann]

Ok, I found out that I have to run the daemon outside the docker-machine. Is there any way to remove the docker depenecy on this whole toolchain?

➜  http unik target --host 192.168.99.100                                                  
INFO[0000] target set: 192.168.99.100:3000              
➜  http unik build --name myImage --path ./ --base rump --language go --provider virtualbox
INFO[0000] running unik build                            args= base=rump force=false host=192.168.99.100:3000 language=go mountPoints=[] name=myImage path=./ provider=virtualbox
INFO[0000] App packaged as tarball: /tmp/sources.tar.gz.241066811

ERRO[0000] build failed: [cmd/build.go:105] building image failed: {[client/images.go:63] failed with status 400: [daemon/daemon.go:468] virtualbox is not a known provider. Available: } 

And the daemon logs heavily (so it works). But: inside docker-machine VM there is obviously no VirtualBox installation...

[ilackarms]

How docker, virtualbox, and unik work together:

unik runs on the same machine as virtualbox. it's controlling virtualbox by running commands with VBoxManage, so wherever **unik runs, that's where virtualbox is going to run.

docker can run anywhere as long as the terminal session that unik is running in has access to it. if you can run docker run docker ps, etc. from the current terminal window, unik will be able to run what it needs to. the only requirement is that mounting directories from the host to the container with -v works. certain docker-machine drivers such as AWS don't support mounting host directories as volumes on the container. the virtualbox driver does support this feature.

docker is a necessary part of unik because we package unik's many dependencies (unikernel toolchains, disk image utils, etc.) as docker containers. it simplifies the work users have to do in order to build images with unik, and allows the devs to manage dependencies as part of unik's codebase.

you should be running unik locally (or wherever you're running virtualbox), and simply give it access to the docker backend with eval $(docker-machine env <machine-name>)

[ilackarms]

in the last error it looks like you don't have a properly configured daemon-config.yaml. this file should live at ~/.unik/daemon-config.yaml on the machine where you're running the daemon. you can also put this file in a custom location and point to it with unik daemon --f <path/to/daemon-cfg>

unik is booting with 0 providers configured (hence the error virtualbox is not a known provider. Available: }. if you are running unik on the docker-machine vm, you won't be able to use virtualbox as a provider (since it requires virtualbox to be running on the host).

two solutions are, either:
1 - run unik on the same machine where you have virtualbox running (and add virtualbox to your daemon-config.yaml, as per the instructions here: https://github.com/emc-advanced-dev/unik/blob/master/docs/providers/virtualbox.md)

2 - use a provider that targets a remote hypervisor, such as AWS, Vsphere, or Openstack (probably not what you are looking for)

[hanneslehmann]

Hi, thanks for the suggestions, I already tried eval $(docker-machine env <machine-name>) (otherwise docker commands are not working anyway) -> and I am able to run the docker command which fails when running unik. I will double check anyway and in parallel got KVM/rumprun/c running, but wanting go to run :)

Maybe the issue is the mounting -v command. Have to check which directories have to be mounted where....

[hanneslehmann]

So, docker command is still failing, although manually docker daemon is responding:

....
running command  command=[docker run --rm -e ROOT_PATH=github.com/emc-advanced-dev/unik/instance-listener -v /tmp/vbox.instancelistener.949677976:/opt/code --name=35fb8e6f-87f1-402e-99cf-75f9b8c32427 projectunik/compilers-rump-go-hw-no-stub:d8c50ae987531c6b]
DEBU[0010] util.LogCommand.func2[util/log.go:95]
docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?. 
DEBU[0010] util.LogCommand.func2[util/log.go:95]
See 'docker run --help'. 
ERRO[0010] main.main[unik/unik.go:9]
cobra.(*Command).Execute[cobra/command.go:618]
cobra.(*Command).ExecuteC[cobra/command.go:662]
cobra.(*Command).execute[cobra/command.go:572]
cmd.glob.func5[cmd/daemon.go:91]
running daemon failed: [cmd/daemon.go:86] daemon failed to initialize: {[daemon/daemon.go:106] initializing virtualbox provider: {[virtualbox/virtualbox_provider.go:49] deploying virtualbox instance listener: {[virtualbox/deploy_instance_listener.go:39] compiling instance listener source to unikernel: {exit status 125}}}} 

➜  unik git:(master) ✗ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

[ilackarms]

Interesting. Are you running Unik as sudo? Unik calls docker by making exec calls on the docker binary, so it should just work

[hanneslehmann]

Yes, I was running Unik as sudo. My docker exec is in /usr/local/bin/docker, but in the PATH as well. Don't know, but I have overcome the docker issue, now I am on (I guess it's some drive mapping issue):

DEBU[0012] util.LogCommand.func2[util/log.go:95]
./gomaincaller.go:20[/tmp/go-build650151351/command-line-arguments/_obj/gomaincaller.cgo1.go:26]: undefined: main 
ERRO[0012] main.main[unik/unik.go:9]
cobra.(*Command).Execute[cobra/command.go:618]
cobra.(*Command).ExecuteC[cobra/command.go:662]
cobra.(*Command).execute[cobra/command.go:572]
cmd.glob.func5[cmd/daemon.go:91]
running daemon failed: [cmd/daemon.go:86] daemon failed to initialize: {[daemon/daemon.go:106] initializing virtualbox provider: {[virtualbox/virtualbox_provider.go:49] deploying virtualbox instance listener: {[virtualbox/deploy_instance_listener.go:39] compiling instance listener source to unikernel: {exit status 2}}}} 

What would be the command to start the listener manually? I am able to login to the container via

docker run -it --rm -e ROOT_PATH=github.com/emc-advanced-dev/unik/instance-listener -v /tmp/vbox.instancelistener.766653488:/opt/code --name=e0a880b1-37d7-4516-afdc-3e83ad110cf2 projectunik/compilers-rump-go-hw-no-stub:d8c50ae987531c6b /bin/bash

and can see the mounted DIR /opt/code filled with the files: gomaincaller.go and mainstub.c.

go run gomaincaller.go gives:
./gomaincaller.go:20: undefined: main

[ilackarms]

This is a different error than the one you posted before. Before it was

docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?. 

How did you solve this?

The new error

./gomaincaller.go:20[/tmp/go-build650151351/command-line-arguments/_obj/gomaincaller.cgo1.go:26]: undefined: main

means that the main file for the instance listener was not found. The instance listener main.go is stored in a local directory (in the example you showed, it's /tmp/vbox.instancelistener.766653488). We give the container access to this file by mounting the directory as a volume using docker's -v flag. In this case, it looks like mounting that directory didn't work. I don't believe docker-machine supports mounting directories as volumes over a network. I've tried it before using the AWS plugin for docker-machine, and when you run with -v local_dir:container:dir, the contents of local_dir are not copied to the container (nor are any files placed in container_dir copied back to the local).

This sharing of files between unik and the docker containers it uses is essential to compiling unikernels. I recommend running unik on the same host you're running docker. (Note: the cli can be run on a different host than the daemon, but the daemon should be run on the same host that's running docker containers). The other recommendation I can make is to figure out how to make -v work on docker-machine vms over the network

[hanneslehmann]

Hi,
I am not sure what I did, I had to restart my box as it crashed (due to other reasons).

I am running docker within a virtualbox (boot2docker) and using docker-machine for controling. Due to security issues I don't want to have running docker directly on my host. So I have VirtualBox running on my host, with a boot2docker client. I have installed unik on my host. This setup seems not to work due to missing libs/files (as seen above). I would need to understand which steps need which access, I might even help to enhance unik for docker-machine compatiblity. But I have to understand the interaction between the complete toolchain (e.g. sequence diagram, architecture or so). Meanwhile I will try and poke around and see what I understand.

[ilackarms]

try the following steps from your host machine:

mkdir foo
echo hello > foo/bar
docker run -ti --rm -v $PWD/foo:/foo busybox cat /foo/bar

you should see the output hello

if this doesn't work, the problem is the issue i mentioned; that directories are not shared properly with the docker-machine vm. if it works, the problem is something else

[hanneslehmann]

hmm....is something different, I guess... :(

./gomaincaller.go:20[/tmp/go-build194448468/command-line-arguments/_obj/gomaincaller.cgo1.go:26]: undefined: main
➜ unik git:(master) ✗ docker run -ti --rm -v /home/hannes/foo:/foo busybox cat /foo/bar
hello

It works with PWD as well:
➜ unik git:(master) ✗ docker run -ti --rm -v $PWD/foo:/foo busybox cat /foo/bar
hello

[ilackarms]

try one more thing...
try repeating the same steps i mentioned before, but from with in the /tmp directory. it might be that docker-machine isn't mounting directories from within /tmp. this is a known problem on OS X; we usually don't encounter it with Linux since most people run docker right on their linux ;)

if this is indeed the problem, you can set the environment variable TMPDIR=$HOME/any/existing/directory/ which Go uses to deposit temp files. then try running unik

[hanneslehmann]

Hi, I gave mounted the host /tmp into docker-machine after sharing directory to docker vm within virtualbox. And: now I got passed that error, but it's stuck in downloading fs layers (since 2 hours)

docker run --rm --privileged -v /dev/:/dev/ -v /tmp/:/opt/vol --name=fd36e419-eabf-4c80-bfd5-487a0545e7d1 projectunik/image-creator:a2d32477df4808ee -v empty.data.folder.825129854,10485760 -o data.image.result.img.721285317 -t ext2

Will try at home.

What I did to overcome the issue was:

• configure /home and /tmp within virtualbox as shared folders
  
• run docker-machine mount command
  

[hanneslehmann]

I think issue can be closed, I have the daemon running and now a different issue (not able to connect to localhost:3000). Will go along to try to solve this one.....

[ilackarms]

thanks for the update. are you able to compile & build a unikernel? if so, i'll close this issue

[hanneslehmann]

Still not able to compile, but I think at least I got beyond this issue here - so I would be ok to close (not to mix up topics)

[ilackarms]

can you open a new issue with the logs from your failed compile attempt? I'll close this issue for now