Skip to content
This repository was archived by the owner on Feb 8, 2021. It is now read-only.

Conversation

@bergwolf
Copy link
Member

@bergwolf bergwolf commented Jan 5, 2017

The PR rebases #355 on top of latest master. It changes qemu and libvirt driver to support vsock device in guest vm:

  1. try to load vsock host kernel module at startup
  2. if vsock host modules are supported and vsock is enabled via config, add vsock device to guest vm and create control and tty channels via vsock
  3. persistent vsock cid and load it at startup
[hypervsock@~]$sudo hyperctl run -d bergwolf/ubuntu-fio
POD id is ubuntu-fio-3551741209
Time to run a POD is 4477 ms

[hypervsock@~]$pa|grep vsock
root     128768  3.3  6.1 1368832 109172 ?      Sl   16:37   0:07 /usr/bin/qemu-system-x86_64 -machine pc-i440fx-2.1,accel=kvm,usb=off -global kvm-pit.lost_tick_policy=discard -cpu host -kernel /var/lib/hyper/kernel -initrd /var/lib/hyper/hyper-initrd.img -append console=ttyS0 panic=1 no_timer_check -realtime mlock=off -no-user-config -nodefaults -no-hpet -rtc base=utc,driftfix=slew -no-reboot -display none -boot strict=on -m size=64,slots=1,maxmem=32768M -smp cpus=1,maxcpus=8 -numa node,nodeid=0,cpus=0-7,mem=64 -qmp unix:/var/run/hyper/vm-HAvUnepiTO/qmp.sock,server,nowait -serial unix:/var/run/hyper/vm-HAvUnepiTO/console.sock,server,nowait -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3 -chardev socket,id=charch0,path=/var/run/hyper/vm-HAvUnepiTO/hyper.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch1,path=/var/run/hyper/vm-HAvUnepiTO/tty.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -fsdev local,id=virtio9p,path=/var/run/hyper/vm-HAvUnepiTO/share_dir,security_model=none -device virtio-9p-pci,fsdev=virtio9p,mount_tag=share_dir -daemonize -pidfile /var/run/hyper/vm-HAvUnepiTO/pidfile -D /var/log/hyper/qemu/vm-HAvUnepiTO.log -device vhost-vsock-pci,id=vsock0,bus=pci.0,addr=5,guest-cid=1024

root@ubuntu-fio-3551741209:/# ./nc-vsock -l 1234
Connection from cid 2 port 1158...

Requires: hyperhq/hyperstart#249

TODOs:

  1. need to handle vsock cid increment for runv cmd
  2. need to figure out how to make it work with template

const hyperDefaultVsockCid = 1024
const hyperDefaultVsockBitmapSize = 16384

func Dial(cid uint32, port uint32) (net.Conn, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems better if it is moved to lib/vsock/vsock_conn.go

Patch it in vendor so we don't have to use cgo for vsock related
syscalls.

Signed-off-by: Peng Tao <bergwolf@gmail.com>
It only works for Linux.

Signed-off-by: Peng Tao <bergwolf@gmail.com>
Signed-off-by: Peng Tao <bergwolf@gmail.com>
…ctions

So that it works for both serial and vsock connections. Also rename
serial.go to json.go to reflect the fact that both connections work and
it is all about json protocol here.

Signed-off-by: Peng Tao <bergwolf@gmail.com>
Signed-off-by: Peng Tao <bergwolf@gmail.com>
It indicates if underlying driver supports the linux vsock feature.

Signed-off-by: Peng Tao <bergwolf@gmail.com>
Signed-off-by: Peng Tao <bergwolf@gmail.com>
When vsock is configured and supported, grab new vsock guest cid
when creating new vm context. Also save and load guest cid in persistent
info.

Signed-off-by: Peng Tao <bergwolf@gmail.com>
Signed-off-by: Peng Tao <bergwolf@gmail.com>
Signed-off-by: Peng Tao <bergwolf@gmail.com>
libvirt by default uses cgroup controller to limit and grant guest
permissions. However, when adding device via qemu monitor directly,
libvirt has no knowledge and thus does not grant proper permissions to
qemu process. As a result, qemu will get EPERM opening /dev/vhost-vsock
device.

To work around it, remove devices from cgroup_controllers in
/etc/libvirt/qemu.conf.

Signed-off-by: Peng Tao <bergwolf@gmail.com>
Signed-off-by: Peng Tao <bergwolf@gmail.com>
@bergwolf
Copy link
Member Author

@laijs updated with following requested changes:

  1. use vsock:// and unix:// to denote vsock and unix socket addresses, and convert utils/libhyperstart to handle both
  2. move vsock guest cid to vmcontext and enable vsock support for libvirt
  3. rename VsockCid to VsockCidAllocator
  4. move vsock dialer to vsock_conn.go

One thing to note:

    libvirt by default uses cgroup controller to limit and grant guest
    permissions. However, when adding device via qemu monitor directly,
    libvirt has no knowledge and thus does not grant proper permissions to
    qemu process. As a result, qemu will get EPERM opening /dev/vhost-vsock
    device.

    To work around it, remove devices from cgroup_controllers in
    /etc/libvirt/qemu.conf.

@laijs laijs merged commit 956133b into hyperhq:master Mar 1, 2017
@bergwolf bergwolf deleted the vsock-rebased branch March 2, 2017 07:54
jimoosciuc pushed a commit to jimoosciuc/runv that referenced this pull request May 26, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants