grub not picking up devices from device.map

[internationils]

I'm trying to move vbox VMs (Debian 8) to bhyve. I've managed to get to the grub prompt.
Two issues:

• nothing I type is echoed, and newlines don't appear (even after a forcekill, reset in the window, and a conreset) The console doesn't do newlines, that's why the 'ls' command isn't shown.

[root@freenas /mnt/iohyve]# iohyve console web-bhyve
Starting console on web-bhyve...
~~. to escape console [uses cu(1) for console]
Connected
grub> (cd0) (host)

• grub doesn't show the hd0 device which is listed in the device.map ... I would assume that with that device.map hd0 and cd0 should show up. Is there any way to verify that the raw image is correct? I used VBoxManage to convert it from a .vdi and copied it over from the vbox jail using plain cp

[root@freenas /mnt/iohyve]# ll web-bhyve/
total 8938827
drwxr-xr-x  2 root  wheel  uarch          4 Dec 31 11:27 ./
drwxr-xr-x  5 root  wheel  uarch          5 Dec 31 11:20 ../
-rw-r--r--  1 root  wheel  uarch         96 Dec 31 11:52 device.map
-rw-------  1 root  wheel  uarch 8589934592 Dec 31 11:26 disk0
[root@freenas /mnt/iohyve]# cat web-bhyve/device.map 
(hd0) /dev/zvol/maindata/iohyve/web-bhyve/disk0
(cd0) /dev/zvol/maindata/iohyve/web-bhyve/disk0
[root@freenas /mnt/iohyve]# iohyve info         
Name             Size  RAM    CPU  OS      Loader
web-bhyve        -     4096M  2    debian  grub-bhyve
web-bhyve/disk0  8G    4096M  2    debian  grub-bhyve
[root@freenas /mnt/iohyve]# iohyve getall web-bhyve
Getting web-bhyve iohyve properties...
bargs          -A_-H_-P
boot           0
con            nmdm0
cpu            2
description    Sat Dec 31 11:20:39 CET 2016
install        no
loader         grub-bhyve
name           web-bhyve
net            igb0
os             debian
persist        1
ram            4096M
size           8589934592
tap            tap0
template       NO
vnc            NO
vnc_h          600
vnc_ip         127.0.0.1
vnc_tablet     NO
vnc_w          800
vnc_wait       NO

Forum post to try and get help is at https://forums.freenas.org/index.php?threads/vbox-to-bhyve-stuck-at-grub.49086/
Apart from the newline issue, it could be the same as ticket #224

[bushchannelstepper]

I tried migrating a vbox vm to bhyve via iohyve as well. I was able to walk through the grub steps to boot manually, but in the vm's boot volume grub2/device.map refers to the main volume as vda (because in vbox that volume is under /dev/vda) whereas it shows up in bhyve as /dev/sda. I was able to mount the boot volume on another system and change it accordingly in grub2/device.map.

[pr1ntf]

I think the problem is more closely related to issue #228. I can boot Debian just fine on my machine, but it lives under /iohyve not /mnt/iohyve.

In an effort to have better housekeeping, I'm moving this show to #228. Feel free to post there if you would like to follow along.

[internationils]

[root@freenas /mnt/iohyve]# sh -x /usr/local/sbin/iohyve start web-bhyve   
sh -x /usr/local/sbin/iohyve start web-bhyve
+ sh -x /usr/local/sbin/iohyve start web-bhyve
+ PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin
+ grep POPCNT /var/run/dmesg.boot
+ [ -n '  Features2=0x29ee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI>' ]
+ grep CPU: /var/run/dmesg.boot
+ grep Intel
+ [ -n 'CPU: Intel(R) Xeon(R) CPU           L5630  @ 2.13GHz (2133.45-MHz K8-class CPU)' ]
+ grep VT-x: /var/run/dmesg.boot
+ grep UG
+ [ -z '  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID' ]
+ [ -e ./lib/ioh-cmd ]
+ [ -e /usr/local/lib/iohyve ]
+ LIB=/usr/local/lib/iohyve
+ . /etc/rc.subr
+ : 69050
+ export RC_PID
+ [ -z '' ]
+ _rc_subr_loaded=YES
+ SYSCTL=/sbin/sysctl
+ SYSCTL_N='/sbin/sysctl -n'
+ SYSCTL_W=/sbin/sysctl
+ ID=/usr/bin/id
+ IDCMD='if [ -x /usr/bin/id ]; then /usr/bin/id -un; fi'
+ PS='/bin/ps -ww'
+ /bin/ps -ww -p 69050 -o jid=
+ JID=0
+ _rc_namevarlist='program chroot chdir env flags fib nice user group groups prepend'
+ _rc_subr_loaded=:
+ . /usr/local/lib/iohyve/ioh-cmd
+ [ ]
+ . /usr/local/lib/iohyve/ioh-setup
+ . /usr/local/lib/iohyve/ioh-zfs
+ . /usr/local/lib/iohyve/ioh-console
+ . /usr/local/lib/iohyve/ioh-iso
+ . /usr/local/lib/iohyve/ioh-firmware
+ . /usr/local/lib/iohyve/ioh-guest
+ . /usr/local/lib/iohyve/ioh-disk
+ . /usr/local/lib/iohyve/ioh-tap
+ [ -z start ]
+ __root_req_cmd start
+ return 0
+ whoami
+ [ root != root ]
+ __parse_cmd start web-bhyve
+ [ 2 -gt 0 ]
+ __guest_start start web-bhyve
+ local name=web-bhyve
+ [ -z web-bhyve ]
+ __guest_exist web-bhyve
+ zfs get -H -s local,received -o value iohyve:name
+ grep '^web-bhyve$'
+ local flag=
+ local pci=
+ local runmode=1
+ zfs get -H -s local,received -o name,value -t filesystem iohyve:name
+ printf '\t'
+ grep '	web-bhyve$'
+ cut -f1
+ local dataset=maindata/iohyve/web-bhyve
+ zfs get -H -o value iohyve:loader maindata/iohyve/web-bhyve
+ local loader=grub-bhyve
+ zfs get -H -o value iohyve:template maindata/iohyve/web-bhyve
+ local template=NO
+ [ NO = YES ]
+ [ grub-bhyve = uefi ]
+ zfs get -H -o value mountpoint maindata/iohyve/web-bhyve
+ local mountpoint=/mnt/iohyve/web-bhyve
+ [ -d /mnt/iohyve/web-bhyve ]
+ pgrep -fx 'bhyve: ioh-web-bhyve'
+ local running=
+ [ -z ]
+ runmode=1
+ echo 'Starting web-bhyve... (Takes 15 seconds for FreeBSD guests)'
Starting web-bhyve... (Takes 15 seconds for FreeBSD guests)
+ __guest_prepare web-bhyve
+ local name=web-bhyve
+ zfs get -H -s local,received -o name,value -t filesystem iohyve:name
+ printf '\t'
+ grep '	web-bhyve$'
+ cut -f1
+ local dataset=maindata/iohyve/web-bhyve
+ __zfs_get_pcidev_conf maindata/iohyve/web-bhyve
+ local pool=maindata/iohyve/web-bhyve
+ local 'oldifs= 	
'
+ IFS='
'
+ zfs get -H -o property,value all maindata/iohyve/web-bhyve
+ grep iohyve:pcidev:
+ sort
+ IFS=' 	
'
+ local pci=
+ zfs get -H -o value iohyve:tap maindata/iohyve/web-bhyve
+ local listtap=tap0
+ echo tap0
+ sed -n 1p
+ tr , '\n'
+ [ tap0 ]
+ [ tap0 != - ]
+ ifconfig -l
+ tr ' ' '\n'
+ grep -F -w tap0
+ local tapif=tap0
+ [ -z tap0 ]
+ zfs get -H -o value iohyve:mac_tap0 maindata/iohyve/web-bhyve
+ local mac=-
+ [ - = - ]
+ pci=' virtio-net,tap0'
+ pci='ahci-hd,/dev/zvol/maindata/iohyve/web-bhyve/disk0  virtio-net,tap0'
+ pci='hostbridge lpc ahci-hd,/dev/zvol/maindata/iohyve/web-bhyve/disk0  virtio-net,tap0'
+ echo hostbridge lpc ahci-hd,/dev/zvol/maindata/iohyve/web-bhyve/disk0 virtio-net,tap0
+ pci='hostbridge lpc ahci-hd,/dev/zvol/maindata/iohyve/web-bhyve/disk0 virtio-net,tap0'
+ __guest_boot web-bhyve 1 'hostbridge lpc ahci-hd,/dev/zvol/maindata/iohyve/web-bhyve/disk0 virtio-net,tap0'
+ local name=web-bhyve
+ [ -z web-bhyve ]
+ __guest_exist web-bhyve
+ zfs get -H -s local,received -o value iohyve:name
+ grep '^web-bhyve$'
+ local runmode=1
+ local 'pci=hostbridge lpc ahci-hd,/dev/zvol/maindata/iohyve/web-bhyve/disk0 virtio-net,tap0'
+ zfs get -H -s local,received -o name,value -t filesystem iohyve:name
+ printf '\t'
+ grep '	web-bhyve$'
+ cut -f1
+ local dataset=maindata/iohyve/web-bhyve
+ zfs get -H -o value iohyve:ram maindata/iohyve/web-bhyve
+ local ram=4096M
+ zfs get -H -o value iohyve:con maindata/iohyve/web-bhyve
+ local con=nmdm0
+ zfs get -H -o value iohyve:cpu maindata/iohyve/web-bhyve
+ local cpu=2
+ zfs get -H -o value iohyve:persist maindata/iohyve/web-bhyve
+ local persist=1
+ zfs get -H -o value iohyve:bargs maindata/iohyve/web-bhyve
+ local bargexist=-A_-H_-P
+ echo -A_-H_-P
+ sed -e 's/_/ /g'
+ local 'bargs=-A -H -P'
+ zfs set iohyve:install=no maindata/iohyve/web-bhyve
+ __guest_get_bhyve_cmd 'hostbridge lpc ahci-hd,/dev/zvol/maindata/iohyve/web-bhyve/disk0 virtio-net,tap0'
+ local 'devices=hostbridge lpc ahci-hd,/dev/zvol/maindata/iohyve/web-bhyve/disk0 virtio-net,tap0'
+ local pci_slot_count=0
+ echo '-s 0,hostbridge'
+ pci_slot_count=1
+ echo '-s 1,lpc'
+ pci_slot_count=2
+ echo '-s 2,ahci-hd,/dev/zvol/maindata/iohyve/web-bhyve/disk0'
+ pci_slot_count=3
+ echo '-s 3,virtio-net,tap0'
+ pci_slot_count=4
+ local 'pci_args=-s 0,hostbridge
-s 1,lpc
-s 2,ahci-hd,/dev/zvol/maindata/iohyve/web-bhyve/disk0
-s 3,virtio-net,tap0'
+ local runstate=1
+ exit
+ [ 1 = 1 ]
+ __guest_load web-bhyve /dev/zvol/maindata/iohyve/web-bhyve/disk0
+ local name=web-bhyve
+ local media=/dev/zvol/maindata/iohyve/web-bhyve/disk0
+ [ -z /dev/zvol/maindata/iohyve/web-bhyve/disk0 ]
+ __guest_exist web-bhyve
[root@freenas /mnt/iohyve]# + zfs get -H -s local,received -o value iohyve:name
+ grep '^web-bhyve$'
+ zfs get -H -t volume -o name,value iohyve:name
+ printf 'disk0\t'
+ grep 'disk0	web-bhyve$'
+ cut -f1
+ local disk=maindata/iohyve/web-bhyve/disk0
+ zfs get -H -o name,value -s local,received -t filesystem iohyve:name
+ printf '\t'
+ grep '	web-bhyve$'
+ cut -f1
+ local dataset=maindata/iohyve/web-bhyve
+ zfs get -H -o value iohyve:ram maindata/iohyve/web-bhyve
+ local ram=4096M
+ zfs get -H -o value iohyve:con maindata/iohyve/web-bhyve
+ local con=nmdm0
+ zfs get -H -o value iohyve:loader maindata/iohyve/web-bhyve
+ local loader=grub-bhyve
+ zfs get -H -o value iohyve:install maindata/iohyve/web-bhyve
+ local install=no
+ zfs get -H -o value iohyve:os maindata/iohyve/web-bhyve
+ local os=debian8
+ zfs get -H -o value iohyve:autogrub maindata/iohyve/web-bhyve
+ local autogrub=-
+ zfs get -H -o value iohyve:bargs maindata/iohyve/web-bhyve
+ local bargexist=-A_-H_-P
+ echo -A_-H_-P
+ sed -e 's/_/ /g'
+ local 'bargs=-A -H -P'
+ local test_for_wire_memory=-S
+ local wire_memory=
+ [ grub-bhyve = grub-bhyve ]
+ echo 'GRUB Process does not run in background....'
GRUB Process does not run in background....
+ echo 'If your terminal appears to be hanging, check iohyve console web-bhyve in second terminal to complete GRUB process...'
If your terminal appears to be hanging, check iohyve console web-bhyve in second terminal to complete GRUB process...
+ [ no = yes ]
+ [ no = no ]
+ [ debian8 = openbsd60 ]
+ [ debian8 = openbsd59 ]
+ [ debian8 = openbsd58 ]
+ [ debian8 = openbsd57 ]
+ [ debian8 = netbsd ]
+ [ debian8 = debian ]
+ [ debian8 = d8lvm ]
+ [ debian8 = centos6 ]
+ [ debian8 = centos7 ]
+ [ debian8 = custom ]
+ printf '\(hd0\)\ /dev/zvol/maindata/iohyve/web-bhyve/disk0\n'
+ printf '\(cd0\)\ /dev/zvol/maindata/iohyve/web-bhyve/disk0\n'
+ grub-bhyve -m /iohyve/web-bhyve/device.map -r hd0,msdos1 -c /dev/nmdm0A -M 4096M ioh-web-bhyve

[root@freenas /mnt/iohyve]# 

[internationils]

grub> ls
(hd0) (cd0) (host)
grub> ls (cd0)
Device cd0: No known filesystem detected - Total size 16777216 sectors
grub> ls (host)
Device host: Filesystem type hostfs - Total size 0 sectors
grub> ls (hd0)
Device hd0: No known filesystem detected - Total size 16777216 sectors
grub> exit
Killed
[nils@freenas /mnt/iohyve]$ sudo file web-bhyve/disk0
web-bhyve/disk0: DOS/MBR boot sector; GRand Unified Bootloader, stage1 version 0x3, stage2 address 0x2000, stage2 segment 0x200

[internationils]

...after the kill, in the main iohyve start terminal:

+ local vmpid=69618
+ wait 69618
+ bhyve -c 2 -A -H -P -m 4096M -s 0,hostbridge -s 1,lpc -s 2,ahci-hd,/dev/zvol/maindata/iohyve/web-bhyve/disk0 -s 3,virtio-net,tap0 -lcom1,/dev/nmdm0A ioh-web-bhyve
vm exit[0]
	reason		VMX
	rip		0x0000000000000000
	inst_length	0
	status		0
	exit_reason	33
	qualification	0x0000000000000000
	inst_type		0
	inst_error		0
+ vmrc=134
+ sleep 5
+ [ 1 == 0 ]
+ [ 134 == 1 ]
+ zfs get -H -o value iohyve:persist maindata/iohyve/web-bhyve
+ [ 1 != 1 ]
+ [ 1 = 1 ]
+ __guest_load web-bhyve /dev/zvol/maindata/iohyve/web-bhyve/disk0
+ local name=web-bhyve
+ local media=/dev/zvol/maindata/iohyve/web-bhyve/disk0
+ [ -z /dev/zvol/maindata/iohyve/web-bhyve/disk0 ]
+ __guest_exist web-bhyve
+ zfs get -H -s local,received -o value iohyve:name
+ grep '^web-bhyve$'
+ zfs get -H -t volume -o name,value iohyve:name
+ printf 'disk0\t'
+ grep 'disk0	web-bhyve$'
+ cut -f1
+ local disk=maindata/iohyve/web-bhyve/disk0
+ zfs get -H -o name,value -s local,received -t filesystem iohyve:name
+ printf '\t'
+ grep '	web-bhyve$'
+ cut -f1
+ local dataset=maindata/iohyve/web-bhyve
+ zfs get -H -o value iohyve:ram maindata/iohyve/web-bhyve
+ local ram=4096M
+ zfs get -H -o value iohyve:con maindata/iohyve/web-bhyve
+ local con=nmdm0
+ zfs get -H -o value iohyve:loader maindata/iohyve/web-bhyve
+ local loader=grub-bhyve
+ zfs get -H -o value iohyve:install maindata/iohyve/web-bhyve
+ local install=no
+ zfs get -H -o value iohyve:os maindata/iohyve/web-bhyve
+ local os=debian8
+ zfs get -H -o value iohyve:autogrub maindata/iohyve/web-bhyve
+ local autogrub=-
+ zfs get -H -o value iohyve:bargs maindata/iohyve/web-bhyve
+ local bargexist=-A_-H_-P
+ echo -A_-H_-P
+ sed -e 's/_/ /g'
+ local 'bargs=-A -H -P'
+ local test_for_wire_memory=-S
+ local wire_memory=
+ [ grub-bhyve = grub-bhyve ]
+ echo 'GRUB Process does not run in background....'
GRUB Process does not run in background....
+ echo 'If your terminal appears to be hanging, check iohyve console web-bhyve in second terminal to complete GRUB process...'
If your terminal appears to be hanging, check iohyve console web-bhyve in second terminal to complete GRUB process...
+ [ no = yes ]
+ [ no = no ]
+ [ debian8 = openbsd60 ]
+ [ debian8 = openbsd59 ]
+ [ debian8 = openbsd58 ]
+ [ debian8 = openbsd57 ]
+ [ debian8 = netbsd ]
+ [ debian8 = debian ]
+ [ debian8 = d8lvm ]
+ [ debian8 = centos6 ]
+ [ debian8 = centos7 ]
+ [ debian8 = custom ]
+ printf '\(hd0\)\ /dev/zvol/maindata/iohyve/web-bhyve/disk0\n'
+ printf '\(cd0\)\ /dev/zvol/maindata/iohyve/web-bhyve/disk0\n'
+ grub-bhyve -m /iohyve/web-bhyve/device.map -r hd0,msdos1 -c /dev/nmdm0A -M 4096M ioh-web-bhyve
Could not reinit VM ioh-web-bhyve
Error in initializing VM
+ local vmpid=69658
+ wait 69658
+ bhyve -c 2 -A -H -P -m 4096M -s 0,hostbridge -s 1,lpc -s 2,ahci-hd,/dev/zvol/maindata/iohyve/web-bhyve/disk0 -s 3,virtio-net,tap0 -lcom1,/dev/nmdm0A ioh-web-bhyve
bhyve: could not activate CPU 0: Device busy
+ vmrc=71
+ sleep 5