Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

running daemon failed #112

Closed
mikeatm opened this issue Nov 27, 2016 · 14 comments
Closed

running daemon failed #112

mikeatm opened this issue Nov 27, 2016 · 14 comments

Comments

@mikeatm
Copy link

mikeatm commented Nov 27, 2016

Im facing this error when i try to run unik daemon

INFO[0010] compiling new instance listener              
INFO[0010] Build cmd for container projectunik/compilers-rump-go-hw:d1bfbc13602e306d  args=[run --rm -e ROOT_PATH=github.com/emc-advanced-dev/unik/instance-listener -e BOOTSTRAP_TYPE=nostub -v /tmp/vbox.instancelistener.598339822:/opt/code --name=4d37fc0d-e160-4abc-965a-dd0e56836898 projectunik/compilers-rump-go-hw:d1bfbc13602e306d]
INFO[0029] Build cmd for container projectunik/boot-creator:d0665c8793b16c51  args=[run --rm --privileged -v /dev/:/dev/ -v /tmp/bootable-image-directory.064284533:/opt/vol/ --name=1d79738e-65da-4ad0-8d32-5611335928de projectunik/boot-creator:d0665c8793b16c51 -d /opt/vol/ -p program.bin -a {"rc":[{"bin":"program","argv":["-prefix","unik_virtualbox","-enablePersistence"]}],"net":{"if":"vioif0","type":"inet","method":"dhcp"},"net":{"if":"vioif1","type":"inet","method":"dhcp"},"blk": {"source":"dev","path":"/dev/sd0e","fstype":"blk","mountpoint":"/bootpart"},"blk": {"source":"dev","path":"/dev/sd1a","fstype":"blk","mountpoint":"/data"}} -o boot-creator-result.img.619518032 -part=true]
ERRO[0039] running daemon failed: [cmd/daemon.go:86] daemon failed to initialize: {[daemon/daemon.go:108] initializing virtualbox provider: {[virtualbox/virtualbox_provider.go:49] deploying virtualbox instance listener: {[virtualbox/deploy_instance_listener.go:39] compiling instance listener source to unikernel: {[rump/rump-go.go:54] creating boot volume from kernel binary: {exit status 1}}}}} 

This is on fedora 24, unik git-HEAD from 27th/nov/2016.

Running with debug it shows:

DEBU[0034] time="2016-11-27T21:42:14Z" level=debug msg="created sparse file" imgFile="/opt/vol/boot.image.2b96a579-6b76-47f2-ba9f-35320c519e1f" size=61MiB  
DEBU[0034] time="2016-11-27T21:42:14Z" level=debug msg="attaching sparse file" imgFile="/opt/vol/boot.image.2b96a579-6b76-47f2-ba9f-35320c519e1f"  
DEBU[0034] time="2016-11-27T21:42:14Z" level=debug msg="running losetup -f" cmd=losetup device="/opt/vol/boot.image.2b96a579-6b76-47f2-ba9f-35320c519e1f"  
DEBU[0034] time="2016-11-27T21:42:14Z" level=debug msg="losetup -f failed" cmd=losetup device="/opt/vol/boot.image.2b96a579-6b76-47f2-ba9f-35320c519e1f" out="losetup: could not find any free loop device\n"  
DEBU[0034] time="2016-11-27T21:42:14Z" level=fatal msg="exit status 255"  
ERRO[0037] running daemon failed: [cmd/daemon.go:86] daemon failed to initialize: {[daemon/daemon.go:108] initializing virtualbox provider: {[virtualbox/virtualbox_provider.go:49] deploying virtualbox instance listener: {[virtualbox/deploy_instance_listener.go:39] compiling instance listener source to unikernel: {[rump/rump-go.go:54] creating boot volume from kernel binary: {exit status 1}}}}} 
[mikef@sandy unikernel]$ unik instances
INFO[0000] listing instances                             host=localhost:9967
ERRO[0000] failed listing instances: [client/instances.go:21] request failed: {[lxhttpclient/httpclient.go:90] error performing get request: {Get http://localhost:9967/instances: dial tcp [::1]:9967: getsockopt: connection refused}} 
@ilackarms
Copy link
Member

If you are using docker machine, try restarting your docker vm. If not, see what loop devices are in use with losetup -a

You can then free one or more devices with losetup -do /dev/loop[0-9]

@mikeatm
Copy link
Author

mikeatm commented Nov 28, 2016

I restarted, but no change to the error:
i only seem to have two loop devices:

# losetup -a
/dev/loop1: [64768]:3015445 (/var/lib/docker/devicemapper/devicemapper/metadata)
/dev/loop0: [64768]:3015443 (/var/lib/docker/devicemapper/devicemapper/data)
# 

trying to free one of them gives this error:

# losetup -do /dev/loop1
losetup: o: failed to use device: No such device

@yuval-k
Copy link
Member

yuval-k commented Nov 29, 2016

@mikeatm Where did you run this command from? the docker machine vm? or a docker container? if a docker container, what were the arguments?

@mikeatm
Copy link
Author

mikeatm commented Nov 29, 2016

I ran the command from the host, all the errors/outputs are from the host not the docker vm, the vm's never start.

@yuval-k
Copy link
Member

yuval-k commented Nov 30, 2016

What do you mean by the docker vm never starts?
Can you provider more information on your system?
Are you using linux or mac os?

@mikeatm
Copy link
Author

mikeatm commented Dec 2, 2016

This all happens on Fedora 24, i was attempting to run one of the examples,
https://github.com/emc-advanced-dev/unik/blob/master/docs/getting_started_python3.md on python3.
but i cant get the daemon to start.
But now it has changed (i havent changed anything, but iv probably restarted the machine a few times since).
Here is the new error:

INFO[0010] compiling new instance listener              
INFO[0010] Build cmd for container projectunik/compilers-rump-go-hw:d1bfbc13602e306d  args=[run --rm -e ROOT_PATH=github.com/emc-advanced-dev/unik/instance-listener -e BOOTSTRAP_TYPE=nostub -v /tmp/vbox.instancelistener.713006637:/opt/code --name=11beab06-890f-4fc8-bf1f-82f44ed36ed0 projectunik/compilers-rump-go-hw:d1bfbc13602e306d]
DEBU[0010] running command                               command=[docker run --rm -e ROOT_PATH=github.com/emc-advanced-dev/unik/instance-listener -e BOOTSTRAP_TYPE=nostub -v /tmp/vbox.instancelistener.713006637:/opt/code --name=11beab06-890f-4fc8-bf1f-82f44ed36ed0 projectunik/compilers-rump-go-hw:d1bfbc13602e306d]
DEBU[0012] + cp /tmp/build/bootstrap_ec2.go /tmp/build/bootstrap_gcloud.go /tmp/build/bootstrap_nostub.go /tmp/build/bootstrap_udp.go /tmp/build/gomaincaller.go . 
DEBU[0012] cp: cannot create regular file './bootstrap_ec2.go': Permission denied 
DEBU[0012] cp: cannot create regular file './bootstrap_gcloud.go': Permission denied 
DEBU[0012] cp: cannot create regular file './bootstrap_nostub.go': Permission denied 
DEBU[0012] cp: cannot create regular file './bootstrap_udp.go': Permission denied 
DEBU[0012] cp: cannot create regular file './gomaincaller.go': Permission denied 
ERRO[0016] running daemon failed: [cmd/daemon.go:86] daemon failed to initialize: {[daemon/daemon.go:108] initializing virtualbox provider: {[virtualbox/virtualbox_provider.go:49] deploying virtualbox instance listener: {[virtualbox/deploy_instance_listener.go:39] compiling instance listener source to unikernel: {exit status 1}}}}

im not sure whether these are bugs or support problems,

@ilackarms
Copy link
Member

try making a temporary directory somewhere like ${HOME}/tmp
then run
export TMPDIR=${HOME}/tmp

(the problem seems related to your /tmp directory having write permission by your user)

@mikeatm
Copy link
Author

mikeatm commented Dec 3, 2016

Ok, the permissions issue was SELinux, i disabled it, and were back to the other error on losetup,

DEBU[0039] running command                               command=[docker run --rm --privileged -v /dev/:/dev/ -v /home/data/unikernel/tmp/bootable-image-directory.123649147:/opt/vol/ --name=6b468b81-842f-4a2d-8318-8d06240fefd9 projectunik/boot-creator:d0665c8793b16c51 -d /opt/vol/ -p program.bin -a {"rc":[{"bin":"program","argv":["-prefix","unik_virtualbox","-enablePersistence"]}],"net":{"if":"vioif0","type":"inet","method":"dhcp"},"net":{"if":"vioif1","type":"inet","method":"dhcp"},"blk": {"source":"dev","path":"/dev/sd0e","fstype":"blk","mountpoint":"/bootpart"},"blk": {"source":"dev","path":"/dev/sd1a","fstype":"blk","mountpoint":"/data"}} -o boot-creator-result.img.025557918 -part=true]
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="calling CreateBootImageWithSize" args="{\"rc\":[{\"bin\":\"program\",\"argv\":[\"-prefix\",\"unik_virtualbox\",\"-enablePersistence\"]}],\"net\":{\"if\":\"vioif0\",\"type\":\"inet\",\"method\":\"dhcp\"},\"net\":{\"if\":\"vioif1\",\"type\":\"inet\",\"method\":\"dhcp\"},\"blk\": {\"source\":\"dev\",\"path\":\"/dev/sd0e\",\"fstype\":\"blk\",\"mountpoint\":\"/bootpart\"},\"blk\": {\"source\":\"dev\",\"path\":\"/dev/sd1a\",\"fstype\":\"blk\",\"mountpoint\":\"/data\"}}" imgFile="/opt/vol/boot.image.c989885d-a6f8-4dc8-b58d-e53f29fff9e7" kernelFile="/opt/vol/program.bin" usePartitionTables=true  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file vol"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file Godeps"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file Godeps.json"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file boot-creator-result.img.025557918"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file bootstrap_ec2.go"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file bootstrap_gcloud.go"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file bootstrap_nostub.go"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file bootstrap_udp.go"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 10 after adding file main.go"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 20 after adding file program"  
DEBU[0042] time="2016-12-03T06:49:36Z" level=debug msg="total size 47 after adding file program.bin"  
DEBU[0044] time="2016-12-03T06:49:38Z" level=debug msg="created sparse file" imgFile="/opt/vol/boot.image.c989885d-a6f8-4dc8-b58d-e53f29fff9e7" size=61MiB  
DEBU[0044] time="2016-12-03T06:49:38Z" level=debug msg="attaching sparse file" imgFile="/opt/vol/boot.image.c989885d-a6f8-4dc8-b58d-e53f29fff9e7"  
DEBU[0044] time="2016-12-03T06:49:38Z" level=debug msg="running losetup -f" cmd=losetup device="/opt/vol/boot.image.c989885d-a6f8-4dc8-b58d-e53f29fff9e7"  
DEBU[0044] time="2016-12-03T06:49:38Z" level=debug msg="losetup -f failed" cmd=losetup device="/opt/vol/boot.image.c989885d-a6f8-4dc8-b58d-e53f29fff9e7" out="losetup: could not find any free loop device\n"  
DEBU[0044] time="2016-12-03T06:49:38Z" level=fatal msg="exit status 255"  
ERRO[0048] running daemon failed: [cmd/daemon.go:86] daemon failed to initialize: {[daemon/daemon.go:108] initializing virtualbox provider: {[virtualbox/virtualbox_provider.go:49] deploying virtualbox instance listener: {[virtualbox/deploy_instance_listener.go:39] compiling instance listener source to unikernel: {[rump/rump-go.go:54] creating boot volume from kernel binary: {exit status 1}}}}}

@ilackarms
Copy link
Member

can you show the output of losetup -a?

@mikeatm
Copy link
Author

mikeatm commented Dec 5, 2016

Here it is.

$ losetup -a
/dev/loop1: []: (/var/lib/docker/devicemapper/devicemapper/metadata)
/dev/loop0: []: (/var/lib/docker/devicemapper/devicemapper/data)

@mikeatm
Copy link
Author

mikeatm commented Feb 9, 2017

if anyone runs into this on Fedora try 👍
mknod -m 0660 /dev/loop3 b 7 3
to create a new one

@mikeatm mikeatm closed this as completed Feb 9, 2017
@mjbright
Copy link

I'm having the same issue on Fedora 24.

I've tried
mknod -m 0660 /dev/loopN b 7 3

for N = 0 to 3, without success.

losetup -a gives me nothing.

losetup -f gives me /dev/loop0

mjb@zBook ~/src/git/Unikernels/unik> losetup -a
mjb@zBook ~/src/git/Unikernels/unik> losetup -f
/dev/loop0
mjb@zBook ~/src/git/Unikernels/unik> ll /dev/loop[0-9]*
brw-rw----. 1 root root 7, 3 Jun 26 09:12 /dev/loop1
brw-rw----. 1 root root 7, 3 Jun 26 09:12 /dev/loop2
brw-rw----. 1 root root 7, 3 Jun 26 09:13 /dev/loop0
brw-rw----. 1 root disk 259, 2 Jun 26 09:13 /dev/loop3p1
brw-rw----. 1 root disk 7, 3 Jun 26 09:13 /dev/loop3

I'll open another issue for this ...

@mjbright
Copy link

... replying to myself here.
As mentioned when closing #133, the problem was

  • I was not setting the group correctly on my devices (chgrp root:disk /dev/loop*)
  • I'd used the same minor number on all nodes.

@schnorea
Copy link

UBUNTU - if you don't have enough loop devices.

sudo nano /etc/default/grub
on line
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash max_loop=100"
add the "max_loop=100" save the file then run
sudo update-grub
then reboot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants