x/build: run DragonflyBSD VMs on GCE?

[bradfitz]

Looks like Dragonfly now supports virtio:

https://leaf.dragonflybsd.org/cgi/web-man?command=virtio&section=4

So it should run on GCE?

If somebody could prepare make.bash scripts to script the install to prepare bootable images, we could run it on GCE.

See the netbsd, openbsd, and freebsd directories as examples: https://github.com/golang/build/tree/master/env

(The script must run on Linux and use qemu to do the image creation.)

/cc @tdfbsd

[bradfitz]

More background on the dfly users list:
http://lists.dragonflybsd.org/pipermail/users/2017-December/313731.html

[bradfitz]

In that thread, @rickard-von-essen says:

I have a working packer build of DragonFly BSD https://github.com/boxcutter/bsd.

The most interesting parts are the boot_command
https://github.com/boxcutter/bsd/blob/master/dragonflybsd.json#L5
and actual installer script https://github.com/boxcutter/bsd/blob/master/http/install.sh.dfly

[bradfitz]

/cc @dmitshur

[bradfitz]

Update: I just ran Dragonfly (5.2.2) at home on QEMU/KVM with virtio-scsi and virtio net and it works fine.

So it should work fine on GCE, of course (which we already heard).

At this point I'm thinking we should just do this builder "by hand" for now, with a readme file with notes. I'll prepare the image by hand, then shut it down and copy its disk to a GCE image. (uploading it as a sparse tarball)

We can automate it with expect or whatnot later. Perfect is the enemy of good, etc.

[bradfitz]

I shut down my KVM/QEMU instance, copied its disk to a new GCE image, and created a GCE VM. It kernel panics on boot (over serial) with:

panic() at panic+0x236 0xffffffff805f8666 
panic() at panic+0x236 0xffffffff805f8666 
vfs_mountroot() at vfs_mountroot+0xfe 0xffffffff80672c7e 
mi_startup() at mi_startup+0x84 0xffffffff805c2a64 
Debugger("panic")
CPU0 stopping CPUs: 0x0000000e
 stopped
Stopped at      Debugger+0x7c:  movb    $0,0xe67a49(%rip)
db> 

So, uh, not as easy as I'd hoped.

[bradfitz]

Perhaps if we already have to do the whole double virtualization thing for Solaris (#15581 (comment)) anyway, we could just reuse that mechanism to run Dragonfly in qemu/kvm under GCE.

[cnst]

I've tried working on this earlier this year (back in 2018-02), and had it scripted to make the image automatically, but I had the same issue that it'd work on my machines with vanilla QEMU just fine, including with the disk being accessible on DFly through DragonFly's vtscsi(4) with a local QEMU as per the QEMU configuration magic described over at http://wiki.netbsd.org/tutorials/how_to_setup_virtio_scsi_with_qemu/, but it still wouldn't work on GCE with GCE's virtio_scsi. Is there any info on how GCE's virtio_scsi different from QEMU's virtio_scsi?

I've also tried running DragonFly BSD side by side with FreeBSD with CAMDEBUG, but it didn't seem to reveal anything obvious, although the underlying CAM logic does seem to be quite different, so, it's probably the one to blame. I didn't run out of ideas, but did ran out of time back in February, and recently my GCE credits ran out as well.

Nested virtualisation sounds interesting. Does it require Linux on GCE, or would FreeBSD also work?

[tuxillo]

@cnst do you have instructions on how you tried DragonFly on GCE?

[gopherbot]

Change https://golang.org/cl/162959 mentions this issue: dashboard, buildlet: add a disabled builder with nested virt, for testing

[tuxillo]

I've tried myself and it seems DragonFly is unable to find the disk.
We're working on it already: https://bugs.dragonflybsd.org/issues/3175

[gopherbot]

Change https://golang.org/cl/163057 mentions this issue: buildlet: change image name for COS-with-vmx buildlet

[gopherbot]

Change https://golang.org/cl/163301 mentions this issue: env/linux-x86-vmx: add new Debian host that's like Container-Optimized OS + vmx