[libvirt] With default cpu mode guest is not starting

[nasastry]

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=161800 </cde:info>

Edited guest xml by adding,

  <cpu>
    <model fallback='allow'>power8</model>
    <topology sockets='1' cores='2' threads='8'/>
    <feature policy='force' name='abc'/>
  </cpu>

After saving the guest xml changes to the following

  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>power8</model>
    <topology sockets='1' cores='2' threads='8'/>
    <feature policy='force' name='abc'/>
  </cpu>

Try to start the guest, it fails with the following error.

[root@zzfp365-lp1 ~]# virsh start nrs
error: Failed to start domain nrs
error: the CPU is incompatible with host CPU: unknown host CPU

From /var/log/libvirt/qemu/<domain.log>
[root@zzfp365-lp1 qemu]# cat nrs.log
2017-11-28 11:54:01.991+0000: shutting down, reason=failed

[root@zzfp365-lp1 ~]# rpm -qa | grep "qemu|libvirt"
libvirt-daemon-driver-secret-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
qemu-system-ppc-2.10.92-1.dev.git7209583.el7.centos.ppc64le
libvirt-daemon-driver-storage-iscsi-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-devel-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-daemon-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-daemon-driver-qemu-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-daemon-driver-storage-scsi-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-daemon-kvm-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
qemu-debuginfo-2.10.92-1.dev.git7209583.el7.centos.ppc64le
libvirt-python-3.2.0-3.el7_4.1.ppc64le
libvirt-daemon-driver-network-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-daemon-driver-interface-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-daemon-driver-lxc-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
qemu-system-x86-2.10.92-1.dev.git7209583.el7.centos.ppc64le
libvirt-daemon-driver-storage-disk-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-daemon-config-nwfilter-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-nss-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-login-shell-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-daemon-driver-nwfilter-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
qemu-img-2.10.92-1.dev.git7209583.el7.centos.ppc64le
qemu-common-2.10.92-1.dev.git7209583.el7.centos.ppc64le
qemu-2.10.92-1.dev.git7209583.el7.centos.ppc64le
libvirt-daemon-driver-storage-mpath-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-daemon-config-network-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-lock-sanlock-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-docs-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-libs-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-daemon-driver-storage-core-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-daemon-driver-storage-logical-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-client-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-daemon-lxc-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-debuginfo-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
ipxe-roms-qemu-20170123-1.git4e85b27.el7_4.1.noarch
libvirt-daemon-driver-nodedev-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
qemu-kvm-2.10.92-1.dev.git7209583.el7.centos.ppc64le
libvirt-daemon-driver-storage-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le
libvirt-admin-3.9.0-1.dev.git2c30e7b.el7.centos.ppc64le

[root@zzfp365-lp1 qemu]# uname -a
Linux zzfp365-lp1.aus.stglabs.ibm.com 4.14.0-3.git68b4afb.el7.centos.ppc64le #1 SMP Mon Nov 27 17:05:18 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

[nasastry]

libvirt_cpu_failure_33_guest_xml.txt

Attached guest xml

[cdeadmin]

------- Comment From seg@us.ibm.com 2017-11-29 01:25:26 EDT-------
probably related to the log messages
Nov 27 12:44:56 kofr2 libvirtd: 2017-11-27 18:44:56.034+0000: 5482: warning : virQEMUCapsInit:1197 : Failed to get host CPU cache info
Nov 27 12:44:58 kofr2 libvirtd: 2017-11-27 18:44:58.471+0000: 5482: error : ppc64DriverDecode:691 : unsupported configuration: CPU model POWER8 is not supported by hypervisor

[cdeadmin]

------- Comment From mdroth@us.ibm.com 2017-11-29 21:27:21 EDT-------
(In reply to comment #4)
> Also,
> virsh qemu-monitor-command <domain> '{"execute":"query-cpu-definitions"}' -
> is only listing power9 and not POWER9.
>
> Want qemu to check as to why so.
>
> Thanks,
> Shiva

This is due to a rework on the QEMU side to allow case-insensitive specification of CPU models. As a side-effect of that some models no longer show up as all caps in -cpu ? / query-cpu-definitions. This requires libvirt changes to avoid relying on specific capitalized models, which I think Scott is looking at as part of 161676

Whether it's related to this bug or not I'm not sure, but it seems likely.

[cdeadmin]

------- Comment From shivapbh@in.ibm.com 2017-11-30 06:41:04 EDT-------
*** Bug 161676 has been marked as a duplicate of this bug. ***

[cdeadmin]

------- Comment From nasastry@in.ibm.com 2018-01-01 06:05:51 EDT-------
Tried with the reported xml and seeing the following result. with power8/power9

<cpu mode='custom' match='exact' check='partial'>
<model fallback='allow'>power8</model>
<topology sockets='1' cores='2' threads='8'/>
<feature policy='force' name='abc'/>
</cpu>

virsh start nrs --console

error: Failed to start domain nrs
error: the CPU is incompatible with host CPU: host CPU model does not match required CPU model POWER8

<cpu mode='custom' match='exact' check='partial'>
<model fallback='allow'>power9</model>
<topology sockets='1' cores='2' threads='8'/>
<feature policy='force' name='abc'/>
</cpu>

virsh start nrs --console

error: Failed to start domain nrs
error: internal error: qemu unexpectedly closed the monitor: 2018-01-01T10:58:57.952805Z qemu-system-ppc64: -chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label charserial0)
2018-01-01T10:58:57.973503Z qemu-system-ppc64: Expected key=value format, found +abc.

from /var/log/libvirt/qemu/nrs.log
from the following log observed that "-cpu POWER9,+abc" <<< +abc was added.

2018-01-01 11:01:22.994+0000: starting up libvirt version: 3.9.0, package: 1.rel.git99ed075.el7.centos (Unknown, 2017-12-20-09:13:02, host-os-jenkins-slave02.aus.stglabs.ibm.com), qemu version: 2.11.0, hostname: zzfp365-lp1.aus.stglabs.ibm.com
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name guest=nrs,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-3453-nrs/master-key.aes -machine pseries-2.10,accel=kvm,usb=off,dump-guest-core=off -cpu POWER9,+abc -m 10240 -realtime mlock=off -smp 16,sockets=1,cores=2,threads=8 -uuid 7434394a-8052-4499-912a-7555033435cf -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-3453-nrs/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device pci-ohci,id=usb,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -drive file=/home/nasastry/hostos-3.0-ppc64le.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=50:50:01:4a:4b:4c,bus=pci.0,addr=0x1 -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,reg=0x30000000 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
2018-01-01T11:01:23.113760Z qemu-system-ppc64: -chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label charserial0)
2018-01-01T11:01:23.136724Z qemu-system-ppc64: Expected key=value format, found +abc.
2018-01-01 11:01:23.826+0000: shutting down, reason=failed

Qemu & Libvirt rpms:

qemu-2.11.0-1.rel.gite7153e0.el7.centos.ppc64le
libvirt-3.9.0-1.rel.git99ed075.el7.centos.ppc64le

[cdeadmin]

------- Comment From nasastry@in.ibm.com 2018-01-01 06:35:49 EDT-------
Based on above result, re-opening this bug.

[cdeadmin]

------- Comment From seg@us.ibm.com 2018-01-02 00:08:57 EDT-------
Please try the current libvirt -devel build and let me know if this is still a problem.

[cdeadmin]

------- Comment From nasastry@in.ibm.com 2018-01-02 01:39:13 EDT-------
(In reply to comment #13)
> Please try the current libvirt -devel build and let me know if this is still
> a problem.

Tried with current libvirt -devel i.e. libvirt-3.9.0-3.dev.gitcb3885f.el7.centos.ppc64le
and qemu-2.11.0-1.dev.gite7153e0.el7.centos.ppc64le

Seeing the same result.

from qemu log

2018-01-02 06:37:18.891+0000: starting up libvirt version: 3.9.0, package: 3.dev.gitcb3885f.el7.centos (Unknown, 2017-12-31-09:08:01, host-os-jenkins-slave01.aus.stglabs.ibm.com), qemu version: 2.11.0, hostname: ltczzj2
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name guest=nrs,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-2-nrs/master-key.aes -machine pseries-2.10,accel=kvm,usb=off,dump-guest-core=off -cpu POWER9,+abc -m 10240 -realtime mlock=off -smp 16,sockets=1,cores=2,threads=8 -uuid 7434394a-8052-4499-912a-7555033435cf -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-2-nrs/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device pci-ohci,id=usb,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -drive file=/home/nasastry/hostos-3.0-ppc64le.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=50:50:01:4a:4b:4c,bus=pci.0,addr=0x1 -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,reg=0x30000000 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
2018-01-02T06:37:20.162589Z qemu-system-ppc64: -chardev pty,id=charserial0: char device redirected to /dev/pts/1 (label charserial0)
2018-01-02T06:37:20.168711Z qemu-system-ppc64: Expected key=value format, found +abc.
2018-01-02 06:37:20.565+0000: shutting down, reason=failed

from virsh

virsh start nrs

error: Failed to start domain nrs
error: internal error: qemu unexpectedly closed the monitor: 2018-01-02T06:37:20.162589Z qemu-system-ppc64: -chardev pty,id=charserial0: char device redirected to /dev/pts/1 (label charserial0)
2018-01-02T06:37:20.168711Z qemu-system-ppc64: Expected key=value format, found +abc.

[cdeadmin]

------- Comment From seg@us.ibm.com 2018-01-02 08:50:36 EDT-------
I'm not sure which build that is? Pulling from the host-os/deve/latest repo I end up with libvirt.ppc64le 3.9.0-3.dev.gitcb3885f.el7.centos

[cdeadmin]

------- Comment From seg@us.ibm.com 2018-02-07 22:47:38 EDT-------
Moving to Sprint 8. Need to find some time on an available p9.

[cdeadmin]

------- Comment From seg@us.ibm.com 2018-03-01 16:10:37 EDT-------
Now that we have a p9 machine assigned to our dept and also a (possibly) stable PNOR, Leonardo will have a look.

[cdeadmin]

------- Comment From seg@us.ibm.com 2018-03-08 01:28:30 EDT-------
There are two problems here. I fixed the original problem back in December. The one you're running into here is not a bug, it's bad reporting of a user error. So, please close this bug. If you really want to track the patch improving error reporting, please open a new bug.I put hit into NEEDINFO for now, but one way or the other it should get to CLOSED.