CLOUDSTACK-8715: Add VirtIO channel to all Instances for the Qemu Gue… #985

wido · 2015-10-27T10:08:00Z

…st Agent

This commit adds a additional VirtIO channel with the name 'org.qemu.guest_agent.0'
to all Instances.

With the Qemu Guest Agent the Hypervisor gains more control over the Instance if
these tools are present inside the Instance, for example:

Power control
Flushing filesystems

In the future this should allow safer snapshots on KVM since we can instruct the
Instance to flush the filesystems prior to snapshotting the disk.

More information: http://wiki.qemu.org/Features/QAPI/GuestAgent

remibergsma · 2015-10-28T12:43:32Z

@wido Thanks for working on this, awesome!

I was testing this PR and found that the SSVM and CPVM systemvms do not start properly. This is the error:

2015-10-28 12:36:30,145 WARN  [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-4:null) Li
bvirtException 
org.libvirt.LibvirtException: internal error: process exited while connecting to monitor: 2015-10-28T12:36:2
9.904910Z qemu-kvm: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/v-273-VM.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied
2015-10-28T12:36:29.904949Z qemu-kvm: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/v-273-VM.org.qemu.guest_agent.0,server,nowait: chardev: opening backend "socket" failed

        at org.libvirt.ErrorHandler.processError(Unknown Source)
        at org.libvirt.Connect.processError(Unknown Source)
        at org.libvirt.Connect.processError(Unknown Source)
        at org.libvirt.Connect.domainCreateXML(Unknown Source)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1292)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:82)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:46)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1316)
        at com.cloud.agent.Agent.processRequest(Agent.java:518)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:823)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Here is the channel dir:

[root@kvm1 channel]# ls -al /var/lib/libvirt/qemu/channel/
total 12
drwxr-x---. 3 qemu qemu 4096 Oct 28 09:55 .
drwxr-x---. 7 root root 4096 Oct 28 12:39 ..
drwxr-x---. 2 qemu qemu 4096 Sep 15 14:03 target

The target folder was empty.

I've installed the rpm package that I've built from this branch. The agent seems to run as root:

root      9815  0.4  3.1 4815960 255916 ?      Sl   10:17   0:41 /usr/lib/jvm/jre/bin/java -Xms256m -Xmx2048 <cut>

It was tested on a KVM hypervisor on CentOS 7. Ping me if you need more details.

wido · 2015-10-28T12:58:15Z

@remibergsma Hmm, was SELinux enabled on that system? Can't see any reason why it wouldn't work. All the directories exist.

Do you by any chance have the XML the Agent generated for the SSVM for example?

remibergsma · 2015-10-28T13:05:38Z

Hi @wido :

SELinux is off:

[root@kvm1 channel]# getenforce 
Permissive

The generated XML:

2015-10-28 13:02:00,216 DEBUG [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-5:null) st
arting s-324-VM: <domain type='kvm'>
<name>s-324-VM</name>
<uuid>acdaffdf-9676-42eb-a386-f8aee0aaccca</uuid>
<description>Debian GNU/Linux 5.0 (64-bit)</description>
<cpu></cpu><sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software Foundation</entry>
<entry name='product'>CloudStack KVM Hypervisor</entry>
<entry name='uuid'>acdaffdf-9676-42eb-a386-f8aee0aaccca</entry>
</system>
</sysinfo>
<os>
<type  arch='x86_64' machine='pc'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<on_reboot>restart</on_reboot>
<on_poweroff>destroy</on_poweroff>
<on_crash>destroy</on_crash>
<memory>524288</memory>
<devices>
<memballoon model='none'/>
</devices>
<vcpu>1</vcpu>
<features>
<pae/>
<apic/>
<acpi/>
</features>
<cputune>
<shares>500</shares>
</cputune>
<clock offset='utc'>
<timer name='kvmclock' >
</timer>
</clock>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<interface type='bridge'>
<source bridge='cloud0'/>
<mac address='0e:00:a9:fe:00:66'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<interface type='bridge'>
<source bridge='cloudbr0'/>
<mac address='06:03:40:00:00:08'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<interface type='bridge'>
<source bridge='breth0-50'/>
<mac address='06:ce:f8:00:00:17'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<interface type='bridge'>
<source bridge='cloudbr0'/>
<mac address='06:16:8c:00:00:12'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<console type='pty'>
<target port='0'/>
</console>
<disk  device='disk' type='file'>
<driver name='qemu' type='qcow2' cache='none' />
<source file='/mnt/812ea6a3-7ad0-30f4-9cab-01e3f2985b98/544cbd54-7cfc-423c-b956-c0135efa059f'/>
<target dev='vda' bus='virtio'/>
</disk>
<disk  device='cdrom' type='file'>
<driver name='qemu' type='raw' cache='none' />
<source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
<target dev='hdc' bus='ide'/>
</disk>
<serial type='pty'>
<target port='0'/>
</serial>
<graphics type='vnc' autoport='yes' listen='192.168.22.21' passwd='NYq46aTBPtud7Nj4D62C9Q'/>
<channel type='unix'>
<source mode='bind'/>
<address type='virtio-serial'/>
<target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
</channel>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/s-324-VM.agent'/>
<address type='virtio-serial'/>
<target type='virtio' name='s-324-VM.vport'/>
</channel>
<input type='tablet' bus='usb'/>
</devices>
</domain>

wido · 2015-10-28T14:56:27Z

@remibergsma Ok, that is odd.

A proper XML should look like this:

    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/man.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>

path is not required, that is computed by libvirt. But the state connected is probably the issue. I'll fix that.

wido · 2015-10-29T11:57:41Z

@borisroman Could you give this PR a spin?

remibergsma · 2015-10-29T12:19:34Z

@wido I will also test it again today.

ustcweizhou · 2015-10-29T13:46:05Z

I have same issue with Remi,
there are two virtio-serial in the vm definition.

ustcweizhou · 2015-10-29T13:54:06Z

succeed after the following change:

diff --git a/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java b/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingReso
index 4ce1889..59b0c3c 100755
--- a/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
+++ b/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
@@ -1994,13 +1994,13 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv
         final SerialDef serial = new SerialDef("pty", null, (short)0);
         devices.addDevice(serial);

-        /* Add a VirtIO channel for the Qemu Guest Agent tools */
-        devices.addDevice(new ChannelDef("org.qemu.guest_agent.0", ChannelDef.ChannelType.UNIX));
-
         /* Add a VirtIO channel for SystemVMs for communication and provisioning */
         if (vmTO.getType() != VirtualMachine.Type.User) {
             devices.addDevice(new ChannelDef(vmTO.getName() + ".vport", ChannelDef.ChannelType.UNIX,
                                              "/var/lib/libvirt/qemu/" + vmTO.getName() + ".agent"));
+        } else {
+            /* Add a VirtIO channel for the Qemu Guest Agent tools */
+            devices.addDevice(new ChannelDef("org.qemu.guest_agent.0", ChannelDef.ChannelType.UNIX));
         }

         final VideoDef videoCard = new VideoDef(_videoHw, _videoRam);

wido · 2015-10-29T13:57:08Z

Hmm, ok. That should be allowed. Since we want the Guest tools also to be supported on SSVMs to control them properly.

ustcweizhou · 2015-10-29T14:07:45Z

that is good idea. then we need to rebuild systemvm template with qemu-guest-agent

ustcweizhou · 2015-10-29T14:15:45Z

by the way, for user vms, I think it is better to add the field in vm_instance/user_vm or zone/cluster/global setting to determine if the vm is qemu guest agent enabled.

ustcweizhou · 2015-10-29T21:22:48Z

uservm can not start

2015-10-29 21:04:38,017 WARN resource.wrapper.LibvirtStartCommandWrapper LibvirtException
org.libvirt.LibvirtException: internal error Missing source path attribute for char device

wido · 2015-10-30T08:38:47Z

@ustcweizhou Which version of libvirt are you using?

If you use libvirt 1.0.6 or newer, you can omit the path='...' attribute of the  element, and libvirt will manage things automatically on your behalf.

ustcweizhou · 2015-10-30T08:44:05Z

@wido ja, you got it. The issue happened on a host running with Ubuntu 12.04 (QEMU 1.2.1 and libvirt 0.9.13)
There is no issue on Ubuntu 14.04 (QEMU 2.0.0 and libvirt 1.2.2)

ustcweizhou · 2015-10-30T08:47:23Z

@wido by the way, I just remember I have implemented some codes for qemu-guest-agent support , based on cloudstack 4.2.0 maybe.
It is not fully tested. I will share that with you if you need (maybe create a pull request to your github branch)

wido · 2015-10-30T10:53:34Z

@ustcweizhou @remibergsma I just pushed a new version of the commit.

On Ubuntu AppArmor needs to be disabled since the default profile for libvirt doesn't allow writing into /var/lib/libvirt/qemu. This is however already the case with the SSVM.

This could be fixed by adding this to '/etc/apparmor.d/abstractions/libvirt-qemu':

/var/lib/libvirt/qemu/channel/target/* rw,

wido · 2015-11-09T15:38:44Z

I justed pushed a new commit which uses libvirt-java for the communication with the Guest Agent.

The build will fail since this patch for libvirt-java has not been accepted upstream:

ustcweizhou · 2015-11-12T19:42:32Z

@wido nice, I will test it.
does this need the change on libvirt-java ?

wido · 2015-11-13T08:27:00Z

@ustcweizhou Yes, it does. See the patches I posted above.

wido · 2015-11-30T14:41:58Z

Still working on this one. However, I need to get stuff into libvirt-java upstream. Working on that here: https://github.com/wido/libvirt-java/commits/qemu-guest-command

NuxRo · 2015-11-30T15:03:36Z

Ok, godspeed.
With a bit of luck perhaps we'll see this somewhere in 4.7.x?

nislim · 2016-01-17T10:04:23Z

Any progress on getting the patches into upstream libvirt-java? @wido

wido · 2016-01-17T10:08:08Z

@nislim No, not really. The people at libvirt-java aren't the fastest. Thinking about forking it into CloudStack itself.

wido · 2016-04-26T13:02:11Z

I just rebased the commit against master, but still waiting on libvirt-java.

Really think we should fork libvirt-java to get this fixed.

rohityadavcloud · 2016-05-02T11:00:45Z

systemvm/patches/debian/buildsystemvm.sh

@@ -389,6 +389,8 @@ packages() {
  chroot . apt-get --no-install-recommends -q -y --force-yes install ipcalc
  #irqbalance from wheezy-backports
  chroot . apt-get --no-install-recommends -q -y --force-yes -t wheezy-backports install irqbalance
+  #qemu guest agent
+  chroot . apt-get --no-install-recommends -q -y --force-yes install qemu-guest-agent


this needs to go in tools/appliance/definitions/systemvmtemplate

rohityadavcloud · 2016-05-02T11:01:38Z

@wido thanks for the feature, can you rebase against latest master and push -f; update on the state of the PR

We should have this feature.

tag:needlove

wido · 2016-05-02T11:48:51Z

@rhtyd Done! Rebased against master

swill · 2016-05-12T03:39:45Z

Does this require a change to the system VM template? I am pulling this in for testing now...

wido · 2016-05-12T06:00:25Z

@swill Yes, it does. It install the Qemu Quest Agent inside the SSVM.

It doesn't hurt to do that already, it is just that the libvirt-java bindings don't allow us to talk to the VMs.

wido · 2016-05-12T09:01:31Z

Thinking about closing this PR and coming up with two new ones.

One which only adds the channel and installs Qemu Guest Agent in the SSVM.

swill · 2016-05-12T18:36:32Z

This failed during compiling:

[INFO] Apache CloudStack Plugin - Hypervisor KVM ......... FAILURE [11.036s]

Here is the error

[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ cloud-engine-storage-volume ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /data/git/cs1/cloudstack/dist/rpmbuild/BUILD/cloudstack-4.9.0-SNAPSHOT/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java:[3425,30] error: cannot find symbol
[INFO] 1 error
[INFO] -------------------------------------------------------------

…st Agent This commit adds a additional VirtIO channel with the name 'org.qemu.guest_agent.0' to all Instances. With the Qemu Guest Agent the Hypervisor gains more control over the Instance if these tools are present inside the Instance, for example: * Power control * Flushing filesystems In the future this should allow safer snapshots on KVM since we can instruct the Instance to flush the filesystems prior to snapshotting the disk. More information: http://wiki.qemu.org/Features/QAPI/GuestAgent Keep in mind that on Ubuntu AppArmor still needs to be disabled since the default AppArmor profile doesn't allow libvirt to write into /var/lib/libvirt/qemu

wido · 2016-05-13T09:41:35Z

Closing this one in favor of #1545

wido force-pushed the CLOUDSTACK-8715 branch 2 times, most recently from d3a0669 to 2f43a04 Compare October 28, 2015 15:27

wido force-pushed the CLOUDSTACK-8715 branch from 2f43a04 to 9317ef7 Compare October 30, 2015 10:52

wido mentioned this pull request Nov 4, 2015

[4.7] CLOUDSTACK-8715: qemu-guest-agent support on KVM #1027

Closed

wido force-pushed the CLOUDSTACK-8715 branch from dbf8d02 to a09804d Compare November 9, 2015 15:36

wido force-pushed the CLOUDSTACK-8715 branch 2 times, most recently from b6a8707 to 68aad17 Compare April 26, 2016 12:59

wido force-pushed the CLOUDSTACK-8715 branch from 68aad17 to f029029 Compare April 26, 2016 13:00

rohityadavcloud reviewed May 2, 2016
View reviewed changes

wido force-pushed the CLOUDSTACK-8715 branch from f029029 to 1e930ea Compare May 2, 2016 11:48

wido mentioned this pull request May 4, 2016

Convert patchviasocket to python (removes perl dependency for KVM agent) #1533

Merged

wido force-pushed the CLOUDSTACK-8715 branch from 1e930ea to eb46482 Compare May 13, 2016 08:23

wido closed this May 13, 2016

wido mentioned this pull request May 13, 2016

CLOUDSTACK-8715: Add channel to Instances for Qemu Guest Agent #1545

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLOUDSTACK-8715: Add VirtIO channel to all Instances for the Qemu Gue… #985

CLOUDSTACK-8715: Add VirtIO channel to all Instances for the Qemu Gue… #985

wido commented Oct 27, 2015

remibergsma commented Oct 28, 2015

wido commented Oct 28, 2015

remibergsma commented Oct 28, 2015

wido commented Oct 28, 2015

wido commented Oct 29, 2015

remibergsma commented Oct 29, 2015

ustcweizhou commented Oct 29, 2015

ustcweizhou commented Oct 29, 2015

wido commented Oct 29, 2015

ustcweizhou commented Oct 29, 2015

ustcweizhou commented Oct 29, 2015

ustcweizhou commented Oct 29, 2015

wido commented Oct 30, 2015

ustcweizhou commented Oct 30, 2015

ustcweizhou commented Oct 30, 2015

wido commented Oct 30, 2015

wido commented Nov 9, 2015

ustcweizhou commented Nov 12, 2015

wido commented Nov 13, 2015

wido commented Nov 30, 2015

NuxRo commented Nov 30, 2015

nislim commented Jan 17, 2016

wido commented Jan 17, 2016

wido commented Apr 26, 2016

rohityadavcloud May 2, 2016

rohityadavcloud commented May 2, 2016

wido commented May 2, 2016

swill commented May 12, 2016

wido commented May 12, 2016

wido commented May 12, 2016

swill commented May 12, 2016

wido commented May 13, 2016

CLOUDSTACK-8715: Add VirtIO channel to all Instances for the Qemu Gue… #985

CLOUDSTACK-8715: Add VirtIO channel to all Instances for the Qemu Gue… #985

Conversation

wido commented Oct 27, 2015

remibergsma commented Oct 28, 2015

wido commented Oct 28, 2015

remibergsma commented Oct 28, 2015

wido commented Oct 28, 2015

wido commented Oct 29, 2015

remibergsma commented Oct 29, 2015

ustcweizhou commented Oct 29, 2015

ustcweizhou commented Oct 29, 2015

wido commented Oct 29, 2015

ustcweizhou commented Oct 29, 2015

ustcweizhou commented Oct 29, 2015

ustcweizhou commented Oct 29, 2015

wido commented Oct 30, 2015

ustcweizhou commented Oct 30, 2015

ustcweizhou commented Oct 30, 2015

wido commented Oct 30, 2015

wido commented Nov 9, 2015

ustcweizhou commented Nov 12, 2015

wido commented Nov 13, 2015

wido commented Nov 30, 2015

NuxRo commented Nov 30, 2015

nislim commented Jan 17, 2016

wido commented Jan 17, 2016

wido commented Apr 26, 2016

rohityadavcloud May 2, 2016

Choose a reason for hiding this comment

rohityadavcloud commented May 2, 2016

wido commented May 2, 2016

swill commented May 12, 2016

wido commented May 12, 2016

wido commented May 12, 2016

swill commented May 12, 2016

wido commented May 13, 2016