running daemon failed

[mikeatm]

Im facing this error when i try to run unik daemon

INFO[0010] compiling new instance listener              
INFO[0010] Build cmd for container projectunik/compilers-rump-go-hw:d1bfbc13602e306d  args=[run --rm -e ROOT_PATH=github.com/emc-advanced-dev/unik/instance-listener -e BOOTSTRAP_TYPE=nostub -v /tmp/vbox.instancelistener.598339822:/opt/code --name=4d37fc0d-e160-4abc-965a-dd0e56836898 projectunik/compilers-rump-go-hw:d1bfbc13602e306d]
INFO[0029] Build cmd for container projectunik/boot-creator:d0665c8793b16c51  args=[run --rm --privileged -v /dev/:/dev/ -v /tmp/bootable-image-directory.064284533:/opt/vol/ --name=1d79738e-65da-4ad0-8d32-5611335928de projectunik/boot-creator:d0665c8793b16c51 -d /opt/vol/ -p program.bin -a {"rc":[{"bin":"program","argv":["-prefix","unik_virtualbox","-enablePersistence"]}],"net":{"if":"vioif0","type":"inet","method":"dhcp"},"net":{"if":"vioif1","type":"inet","method":"dhcp"},"blk": {"source":"dev","path":"/dev/sd0e","fstype":"blk","mountpoint":"/bootpart"},"blk": {"source":"dev","path":"/dev/sd1a","fstype":"blk","mountpoint":"/data"}} -o boot-creator-result.img.619518032 -part=true]
ERRO[0039] running daemon failed: [cmd/daemon.go:86] daemon failed to initialize: {[daemon/daemon.go:108] initializing virtualbox provider: {[virtualbox/virtualbox_provider.go:49] deploying virtualbox instance listener: {[virtualbox/deploy_instance_listener.go:39] compiling instance listener source to unikernel: {[rump/rump-go.go:54] creating boot volume from kernel binary: {exit status 1}}}}} 

This is on fedora 24, unik git-HEAD from 27th/nov/2016.

Running with debug it shows:

DEBU[0034] time="2016-11-27T21:42:14Z" level=debug msg="created sparse file" imgFile="/opt/vol/boot.image.2b96a579-6b76-47f2-ba9f-35320c519e1f" size=61MiB  
DEBU[0034] time="2016-11-27T21:42:14Z" level=debug msg="attaching sparse file" imgFile="/opt/vol/boot.image.2b96a579-6b76-47f2-ba9f-35320c519e1f"  
DEBU[0034] time="2016-11-27T21:42:14Z" level=debug msg="running losetup -f" cmd=losetup device="/opt/vol/boot.image.2b96a579-6b76-47f2-ba9f-35320c519e1f"  
DEBU[0034] time="2016-11-27T21:42:14Z" level=debug msg="losetup -f failed" cmd=losetup device="/opt/vol/boot.image.2b96a579-6b76-47f2-ba9f-35320c519e1f" out="losetup: could not find any free loop device\n"  
DEBU[0034] time="2016-11-27T21:42:14Z" level=fatal msg="exit status 255"  
ERRO[0037] running daemon failed: [cmd/daemon.go:86] daemon failed to initialize: {[daemon/daemon.go:108] initializing virtualbox provider: {[virtualbox/virtualbox_provider.go:49] deploying virtualbox instance listener: {[virtualbox/deploy_instance_listener.go:39] compiling instance listener source to unikernel: {[rump/rump-go.go:54] creating boot volume from kernel binary: {exit status 1}}}}} 
[mikef@sandy unikernel]$ unik instances
INFO[0000] listing instances                             host=localhost:9967
ERRO[0000] failed listing instances: [client/instances.go:21] request failed: {[lxhttpclient/httpclient.go:90] error performing get request: {Get http://localhost:9967/instances: dial tcp [::1]:9967: getsockopt: connection refused}} 

[ilackarms]

If you are using docker machine, try restarting your docker vm. If not, see what loop devices are in use with losetup -a

You can then free one or more devices with losetup -do /dev/loop[0-9]

[mikeatm]

I restarted, but no change to the error:
i only seem to have two loop devices:

# losetup -a
/dev/loop1: [64768]:3015445 (/var/lib/docker/devicemapper/devicemapper/metadata)
/dev/loop0: [64768]:3015443 (/var/lib/docker/devicemapper/devicemapper/data)
# 

trying to free one of them gives this error:

# losetup -do /dev/loop1
losetup: o: failed to use device: No such device

[yuval-k]

@mikeatm Where did you run this command from? the docker machine vm? or a docker container? if a docker container, what were the arguments?

[mikeatm]

I ran the command from the host, all the errors/outputs are from the host not the docker vm, the vm's never start.

[yuval-k]

What do you mean by the docker vm never starts?
Can you provider more information on your system?
Are you using linux or mac os?

[mikeatm]

This all happens on Fedora 24, i was attempting to run one of the examples,
https://github.com/emc-advanced-dev/unik/blob/master/docs/getting_started_python3.md on python3.
but i cant get the daemon to start.
But now it has changed (i havent changed anything, but iv probably restarted the machine a few times since).
Here is the new error:

INFO[0010] compiling new instance listener              
INFO[0010] Build cmd for container projectunik/compilers-rump-go-hw:d1bfbc13602e306d  args=[run --rm -e ROOT_PATH=github.com/emc-advanced-dev/unik/instance-listener -e BOOTSTRAP_TYPE=nostub -v /tmp/vbox.instancelistener.713006637:/opt/code --name=11beab06-890f-4fc8-bf1f-82f44ed36ed0 projectunik/compilers-rump-go-hw:d1bfbc13602e306d]
DEBU[0010] running command                               command=[docker run --rm -e ROOT_PATH=github.com/emc-advanced-dev/unik/instance-listener -e BOOTSTRAP_TYPE=nostub -v /tmp/vbox.instancelistener.713006637:/opt/code --name=11beab06-890f-4fc8-bf1f-82f44ed36ed0 projectunik/compilers-rump-go-hw:d1bfbc13602e306d]
DEBU[0012] + cp /tmp/build/bootstrap_ec2.go /tmp/build/bootstrap_gcloud.go /tmp/build/bootstrap_nostub.go /tmp/build/bootstrap_udp.go /tmp/build/gomaincaller.go . 
DEBU[0012] cp: cannot create regular file './bootstrap_ec2.go': Permission denied 
DEBU[0012] cp: cannot create regular file './bootstrap_gcloud.go': Permission denied 
DEBU[0012] cp: cannot create regular file './bootstrap_nostub.go': Permission denied 
DEBU[0012] cp: cannot create regular file './bootstrap_udp.go': Permission denied 
DEBU[0012] cp: cannot create regular file './gomaincaller.go': Permission denied 
ERRO[0016] running daemon failed: [cmd/daemon.go:86] daemon failed to initialize: {[daemon/daemon.go:108] initializing virtualbox provider: {[virtualbox/virtualbox_provider.go:49] deploying virtualbox instance listener: {[virtualbox/deploy_instance_listener.go:39] compiling instance listener source to unikernel: {exit status 1}}}}

im not sure whether these are bugs or support problems,

[ilackarms]

try making a temporary directory somewhere like ${HOME}/tmp
then run
export TMPDIR=${HOME}/tmp

(the problem seems related to your /tmp directory having write permission by your user)

[mikeatm]

Ok, the permissions issue was SELinux, i disabled it, and were back to the other error on losetup,

DEBU[0039] running command                               command=[docker run --rm --privileged -v /dev/:/dev/ -v /home/data/unikernel/tmp/bootable-image-directory.123649147:/opt/vol/ --name=6b468b81-842f-4a2d-8318-8d06240fefd9 projectunik/boot-creator:d0665c8793b16c51 -d /opt/vol/ -p program.bin -a {"rc":[{"bin":"program","argv":["-prefix","unik_virtualbox","-enablePersistence"]}],"net":{"if":"vioif0","type":"inet","method":"dhcp"},"net":{"if":"vioif1","type":"inet","method":"dhcp"},"blk": {"source":"dev","path":"/dev/sd0e","fstype":"blk","mountpoint":"/bootpart"},"blk": {"source":"dev","path":"/dev/sd1a","fstype":"blk","mountpoint":"/data"}} -o boot-creator-result.img.025557918 -part=true]
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="calling CreateBootImageWithSize" args="{\"rc\":[{\"bin\":\"program\",\"argv\":[\"-prefix\",\"unik_virtualbox\",\"-enablePersistence\"]}],\"net\":{\"if\":\"vioif0\",\"type\":\"inet\",\"method\":\"dhcp\"},\"net\":{\"if\":\"vioif1\",\"type\":\"inet\",\"method\":\"dhcp\"},\"blk\": {\"source\":\"dev\",\"path\":\"/dev/sd0e\",\"fstype\":\"blk\",\"mountpoint\":\"/bootpart\"},\"blk\": {\"source\":\"dev\",\"path\":\"/dev/sd1a\",\"fstype\":\"blk\",\"mountpoint\":\"/data\"}}" imgFile="/opt/vol/boot.image.c989885d-a6f8-4dc8-b58d-e53f29fff9e7" kernelFile="/opt/vol/program.bin" usePartitionTables=true  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file vol"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file Godeps"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file Godeps.json"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file boot-creator-result.img.025557918"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file bootstrap_ec2.go"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file bootstrap_gcloud.go"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file bootstrap_nostub.go"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file bootstrap_udp.go"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file main.go"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 20 after adding file program"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 47 after adding file program.bin"  
DEBU[0044] time="2016-12-03T06:49:38Z" level=debug msg="created sparse file" imgFile="/opt/vol/boot.image.c989885d-a6f8-4dc8-b58d-e53f29fff9e7" size=61MiB  
DEBU[0044] time="2016-12-03T06:49:38Z" level=debug msg="attaching sparse file" imgFile="/opt/vol/boot.image.c989885d-a6f8-4dc8-b58d-e53f29fff9e7"  
DEBU[0044] time="2016-12-03T06:49:38Z" level=debug msg="running losetup -f" cmd=losetup device="/opt/vol/boot.image.c989885d-a6f8-4dc8-b58d-e53f29fff9e7"  
DEBU[0044] time="2016-12-03T06:49:38Z" level=debug msg="losetup -f failed" cmd=losetup device="/opt/vol/boot.image.c989885d-a6f8-4dc8-b58d-e53f29fff9e7" out="losetup: could not find any free loop device\n"  
DEBU[0044] time="2016-12-03T06:49:38Z" level=fatal msg="exit status 255"  
ERRO[0048] running daemon failed: [cmd/daemon.go:86] daemon failed to initialize: {[daemon/daemon.go:108] initializing virtualbox provider: {[virtualbox/virtualbox_provider.go:49] deploying virtualbox instance listener: {[virtualbox/deploy_instance_listener.go:39] compiling instance listener source to unikernel: {[rump/rump-go.go:54] creating boot volume from kernel binary: {exit status 1}}}}}

[ilackarms]

can you show the output of losetup -a?

[mikeatm]

Here it is.

$ losetup -a
/dev/loop1: []: (/var/lib/docker/devicemapper/devicemapper/metadata)
/dev/loop0: []: (/var/lib/docker/devicemapper/devicemapper/data)

[mikeatm]

if anyone runs into this on Fedora try 👍
mknod -m 0660 /dev/loop3 b 7 3
to create a new one

[mjbright]

I'm having the same issue on Fedora 24.

I've tried
mknod -m 0660 /dev/loopN b 7 3

for N = 0 to 3, without success.

losetup -a gives me nothing.

losetup -f gives me /dev/loop0

mjb@zBook ~/src/git/Unikernels/unik> losetup -a
mjb@zBook ~/src/git/Unikernels/unik> losetup -f
/dev/loop0
mjb@zBook ~/src/git/Unikernels/unik> ll /dev/loop[0-9]*
brw-rw----. 1 root root 7, 3 Jun 26 09:12 /dev/loop1
brw-rw----. 1 root root 7, 3 Jun 26 09:12 /dev/loop2
brw-rw----. 1 root root 7, 3 Jun 26 09:13 /dev/loop0
brw-rw----. 1 root disk 259, 2 Jun 26 09:13 /dev/loop3p1
brw-rw----. 1 root disk 7, 3 Jun 26 09:13 /dev/loop3

I'll open another issue for this ...

[mjbright]

... replying to myself here.
As mentioned when closing #133, the problem was

• I was not setting the group correctly on my devices (chgrp root:disk /dev/loop*)
• I'd used the same minor number on all nodes.

[schnorea]

UBUNTU - if you don't have enough loop devices.

sudo nano /etc/default/grub
on line
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash max_loop=100"
add the "max_loop=100" save the file then run
sudo update-grub
then reboot