Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Unable to create pc-q35 VM with Error NO PCI buses available #5437

Open
lanfon72 opened this issue Mar 21, 2024 · 5 comments
Open

[BUG] Unable to create pc-q35 VM with Error NO PCI buses available #5437

lanfon72 opened this issue Mar 21, 2024 · 5 comments
Assignees
@a110605
Labels
area/vm-lifecycle backport-needed/1.2.2 backport-needed/1.3.1 kind/bug Issues that are defects reported by users or that we know have reached a real release need-reprioritize not-require/test-plan Skip to create a e2e automation test issue regression reproduce/always Reproducible 100% of the time severity/1 Function broken (a critical incident with very high impact)
Milestone
v1.4.0

Comments

@lanfon72
Copy link
Member

Describe the bug

image

To Reproduce

Steps to reproduce the behavior:

  1. Install Harvester with any nodes
  2. Create an image for VM creation
  3. Create VM vm1 with machine type pc-q35

Expected behavior

VM should started successfully

Environment:

  • Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): Baremetal DL360G9 3 nodes
  • Harvester ISO version: v1.3.0 and v1.2-d66812e6-head
  • ui-source Option: Auto

Additional context

defualt and q35 type is not affected.
@lanfon72 lanfon72 added kind/bug Issues that are defects reported by users or that we know have reached a real release severity/1 Function broken (a critical incident with very high impact) regression need-reprioritize reproduce/always Reproducible 100% of the time area/vm-lifecycle not-require/test-plan Skip to create a e2e automation test issue labels Mar 21, 2024
@lanfon72 lanfon72 mentioned this issue Mar 21, 2024
Merged
@bk201 bk201 added this to the v1.4.0 milestone May 9, 2024
@bk201
Copy link
Member

bk201 commented May 9, 2024 •
edited
Loading

See libvirt error in the virt-launcher log :

{"component":"virt-launcher","level":"error","msg":"XML error: No PCI buses available","pos":"virDomainPCIAddressGetNextAddr:1097","subcomponent":"libvirt","thread":"36","timestamp":"2024-05-09T09:15:34.072000Z"}

virt-launcher.log

`pc-q35` domain XML 
<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
	<name>default_a</name>
	<memory unit="b">968884224</memory>
	<os>
		<type arch="x86_64" machine="pc-q35">hvm</type>
		<smbios mode="sysinfo"></smbios>
	</os>
	<sysinfo type="smbios">
		<system>
			<entry name="uuid">8d9794ca-8c84-5204-b98a-721424eaae6f</entry>
			<entry name="manufacturer">KubeVirt</entry>
			<entry name="family">KubeVirt</entry>
			<entry name="product">None</entry>
			<entry name="sku"></entry>
			<entry name="version"></entry>
		</system>
		<bios></bios>
		<baseBoard></baseBoard>
		<chassis></chassis>
	</sysinfo>
	<devices>
		<interface type="ethernet">
			<source></source>
			<target dev="tap37a8eec1ce1" managed="no"></target>
			<model type="virtio-non-transitional"></model>
			<mac address="0e:9c:4a:62:89:9d"></mac>
			<mtu size="1500"></mtu>
			<alias name="ua-default"></alias>
			<rom enabled="no"></rom>
		</interface>
		<interface type="ethernet">
			<source></source>
			<target dev="placeholder-0" managed="no"></target>
			<model type="virtio-non-transitional"></model>
		</interface>
		<interface type="ethernet">
			<source></source>
			<target dev="placeholder-1" managed="no"></target>
			<model type="virtio-non-transitional"></model>
		</interface>
		<interface type="ethernet">
			<source></source>
			<target dev="placeholder-2" managed="no"></target>
			<model type="virtio-non-transitional"></model>
		</interface>
		<channel type="unix">
			<target name="org.qemu.guest_agent.0" type="virtio"></target>
		</channel>
		<controller type="usb" index="0" model="qemu-xhci"></controller>
		<controller type="scsi" index="0" model="virtio-non-transitional"></controller>
		<controller type="virtio-serial" index="0" model="virtio-non-transitional"></controller>
		<video>
			<model type="vga" heads="1" vram="16384"></model>
		</video>
		<graphics type="vnc">
			<listen type="socket" socket="/var/run/kubevirt-private/fb93df91-5942-4af9-a30b-ad4f354c6f5f/virt-vnc"></listen>
		</graphics>
		<memballoon model="virtio-non-transitional" freePageReporting="on">
			<stats period="10"></stats>
		</memballoon>
		<disk device="disk" type="block" model="virtio-non-transitional">
			<source dev="/dev/disk-0" name="disk-0"></source>
			<target bus="virtio" dev="vda"></target>
			<driver cache="none" error_policy="stop" io="native" name="qemu" type="raw" discard="unmap"></driver>
			<alias name="ua-disk-0"></alias>
			<boot order="1"></boot>
			<filesystemOverhead>0.055</filesystemOverhead>
			<capacity>10737418240</capacity>
		</disk>
		<disk device="disk" type="file" model="virtio-non-transitional">
			<source file="/var/run/kubevirt-ephemeral-disks/cloud-init-data/default/a/noCloud.iso"></source>
			<target bus="virtio" dev="vdb"></target>
			<driver cache="none" error_policy="stop" name="qemu" type="raw" discard="unmap"></driver>
			<alias name="ua-cloudinitdisk"></alias>
		</disk>
		<input type="tablet" bus="usb">
			<alias name="ua-tablet"></alias>
		</input>
		<serial type="unix">
			<target port="0"></target>
			<source mode="bind" path="/var/run/kubevirt-private/fb93df91-5942-4af9-a30b-ad4f354c6f5f/virt-serial0"></source>
			<log file="/var/run/kubevirt-private/fb93df91-5942-4af9-a30b-ad4f354c6f5f/virt-serial0-log" append="on"></log>
		</serial>
		<console type="pty">
			<target type="serial" port="0"></target>
		</console>
	</devices>
	<metadata>
		<kubevirt xmlns="http://kubevirt.io">
			<uid></uid>
		</kubevirt>
	</metadata>
	<features>
		<acpi></acpi>
	</features>
	<cpu mode="host-model">
		<topology sockets="1" cores="1" threads="1"></topology>
	</cpu>
	<vcpu placement="static">1</vcpu>
	<iothreads>1</iothreads>
</domain>
`q35` machine type domain XML: 
<domain type="kvm" xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
	<name>harvester-public_g1-pool1-0819f90d-fpgpl</name>
	<memory unit="b">4190109696</memory>
	<os>
		<type arch="x86_64" machine="q35">hvm</type>
		<smbios mode="sysinfo"></smbios>
	</os>
	<sysinfo type="smbios">
		<system>
			<entry name="uuid">ee83bed4-fce3-5e6b-b777-06ec95bbffd5</entry>
			<entry name="manufacturer">KubeVirt</entry>
			<entry name="family">KubeVirt</entry>
			<entry name="product">None</entry>
			<entry name="sku"></entry>
			<entry name="version"></entry>
		</system>
		<bios></bios>
		<baseBoard></baseBoard>
		<chassis></chassis>
	</sysinfo>
	<devices>
		<emulator>/usr/bin/qemu-system-x86_64</emulator>
		<interface type="ethernet">
			<source></source>
			<target dev="tap0e6b58809dd" managed="no"></target>
			<model type="virtio-non-transitional"></model>
			<mac address="16:b7:8d:a5:b5:14"></mac>
			<mtu size="1500"></mtu>
			<alias name="ua-nic-0"></alias>
			<rom enabled="no"></rom>
		</interface>
		<channel type="unix">
			<target name="org.qemu.guest_agent.0" type="virtio"></target>
		</channel>
		<controller type="usb" index="0" model="none"></controller>
		<controller type="scsi" index="0" model="virtio-non-transitional">
			<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"></address>
		</controller>
		<controller type="virtio-serial" index="0" model="virtio-non-transitional">
			<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"></address>
		</controller>
		<controller type="sata" index="0">
			<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"></address>
		</controller>
		<controller type="pci" index="0" model="pcie-root"></controller>
		<controller type="pci" index="1" model="pcie-root-port">
			<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"></address>
		</controller>
		<controller type="pci" index="2" model="pcie-root-port">
			<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"></address>
		</controller>
		<controller type="pci" index="3" model="pcie-root-port">
			<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"></address>
		</controller>
		<controller type="pci" index="4" model="pcie-root-port">
			<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"></address>
		</controller>
		<controller type="pci" index="5" model="pcie-root-port">
			<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"></address>
		</controller>
		<controller type="pci" index="6" model="pcie-root-port">
			<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"></address>
		</controller>
		<controller type="pci" index="7" model="pcie-root-port">
			<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"></address>
		</controller>
		<controller type="pci" index="8" model="pcie-root-port">
			<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"></address>
		</controller>
		<controller type="pci" index="9" model="pcie-root-port">
			<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"></address>
		</controller>
		<controller type="pci" index="10" model="pcie-root-port">
			<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"></address>
		</controller>
		<video>
			<model type="vga" heads="1" vram="16384"></model>
		</video>
		<graphics type="vnc">
			<listen type="socket" socket="/var/run/kubevirt-private/080f6cd8-da7f-48df-8d1d-c458f3499a29/virt-vnc"></listen>
		</graphics>
		<memballoon model="virtio-non-transitional" freePageReporting="on">
			<stats period="10"></stats>
			<address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"></address>
		</memballoon>
		<disk device="disk" type="file" model="virtio-non-transitional">
			<source file="/var/run/kubevirt-ephemeral-disks/cloud-init-data/harvester-public/g1-pool1-0819f90d-fpgpl/noCloud.iso"></source>
			<target bus="virtio" dev="vda"></target>
			<driver cache="none" error_policy="stop" name="qemu" type="raw" discard="unmap"></driver>
			<alias name="ua-cloudinitdisk"></alias>
			<address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"></address>
		</disk>
		<disk device="disk" type="block" model="virtio-non-transitional">
			<source dev="/dev/disk-1"></source>
			<target bus="virtio" dev="vdb"></target>
			<driver cache="none" error_policy="stop" io="native" name="qemu" type="raw" discard="unmap"></driver>
			<alias name="ua-disk-1"></alias>
			<boot order="1"></boot>
			<address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"></address>
		</disk>
		<input type="mouse" bus="ps2"></input>
		<input type="keyboard" bus="ps2"></input>
		<serial type="unix">
			<target port="0"></target>
			<source mode="bind" path="/var/run/kubevirt-private/080f6cd8-da7f-48df-8d1d-c458f3499a29/virt-serial0"></source>
			<log file="/var/run/kubevirt-private/080f6cd8-da7f-48df-8d1d-c458f3499a29/virt-serial0-log" append="on"></log>
		</serial>
		<console type="unix">
			<target type="serial" port="0"></target>
			<source mode="bind" path="/var/run/kubevirt-private/080f6cd8-da7f-48df-8d1d-c458f3499a29/virt-serial0"></source>
		</console>
	</devices>
	<metadata>
		<kubevirt xmlns="http://kubevirt.io">
			<uid></uid>
		</kubevirt>
	</metadata>
	<features>
		<acpi></acpi>
	</features>
	<cpu mode="host-model">
		<topology sockets="1" cores="2" threads="1"></topology>
	</cpu>
	<vcpu placement="static">2</vcpu>
	<iothreads>1</iothreads>
</domain>

@ibrokethecloud
Copy link
Contributor

The error is because of the lack of a supported pc-q35 machine type in the version of qemu.

In the harvester packaging the supported machine types are

sh-4.4# /usr/bin/qemu-system-x86_64 -machine ?
Supported machines are:
microvm              microvm (i386)
xenfv-4.2            Xen Fully-virtualized PC
xenfv                Xen Fully-virtualized PC (alias of xenfv-3.1)
xenfv-3.1            Xen Fully-virtualized PC
pc                   Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-6.2)
pc-i440fx-6.2        Standard PC (i440FX + PIIX, 1996) (default)
pc-i440fx-6.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-6.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-5.2        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-5.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-5.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-4.2        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-4.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-4.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-3.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-3.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.9        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.8        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.7        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.6        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.5        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.4        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.3        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.2        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.12       Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.11       Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.10       Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.1        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-2.0        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.7        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.6        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.5        Standard PC (i440FX + PIIX, 1996)
pc-i440fx-1.4        Standard PC (i440FX + PIIX, 1996)
q35                  Standard PC (Q35 + ICH9, 2009) (alias of pc-q35-6.2)
pc-q35-6.2           Standard PC (Q35 + ICH9, 2009)
pc-q35-6.1           Standard PC (Q35 + ICH9, 2009)
pc-q35-6.0           Standard PC (Q35 + ICH9, 2009)
pc-q35-5.2           Standard PC (Q35 + ICH9, 2009)
pc-q35-5.1           Standard PC (Q35 + ICH9, 2009)
pc-q35-5.0           Standard PC (Q35 + ICH9, 2009)
pc-q35-4.2           Standard PC (Q35 + ICH9, 2009)
pc-q35-4.1           Standard PC (Q35 + ICH9, 2009)
pc-q35-4.0.1         Standard PC (Q35 + ICH9, 2009)
pc-q35-4.0           Standard PC (Q35 + ICH9, 2009)
pc-q35-3.1           Standard PC (Q35 + ICH9, 2009)
pc-q35-3.0           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.9           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.8           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.7           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.6           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.5           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.4           Standard PC (Q35 + ICH9, 2009)
pc-q35-2.12          Standard PC (Q35 + ICH9, 2009)
pc-q35-2.11          Standard PC (Q35 + ICH9, 2009)
pc-q35-2.10          Standard PC (Q35 + ICH9, 2009)
isapc                ISA-only PC
none                 empty machine
xenpv                Xen Para-virtualized PC

while upstream packaging contains the following

sh-5.1# /usr/libexec/qemu-kvm -machine ?
Supported machines are:
pc                   RHEL 7.6.0 PC (i440FX + PIIX, 1996) (alias of pc-i440fx-rhel7.6.0)
pc-i440fx-rhel7.6.0  RHEL 7.6.0 PC (i440FX + PIIX, 1996) (default) (deprecated)
q35                  RHEL-9.2.0 PC (Q35 + ICH9, 2009) (alias of pc-q35-rhel9.2.0)
pc-q35-rhel9.2.0     RHEL-9.2.0 PC (Q35 + ICH9, 2009)
pc-q35-rhel9.0.0     RHEL-9.0.0 PC (Q35 + ICH9, 2009)
pc-q35-rhel8.6.0     RHEL-8.6.0 PC (Q35 + ICH9, 2009) (deprecated)
pc-q35-rhel8.5.0     RHEL-8.5.0 PC (Q35 + ICH9, 2009) (deprecated)
pc-q35-rhel8.4.0     RHEL-8.4.0 PC (Q35 + ICH9, 2009) (deprecated)
pc-q35-rhel8.3.0     RHEL-8.3.0 PC (Q35 + ICH9, 2009) (deprecated)
pc-q35-rhel8.2.0     RHEL-8.2.0 PC (Q35 + ICH9, 2009) (deprecated)
pc-q35-rhel8.1.0     RHEL-8.1.0 PC (Q35 + ICH9, 2009) (deprecated)
pc-q35-rhel8.0.0     RHEL-8.0.0 PC (Q35 + ICH9, 2009) (deprecated)
pc-q35-rhel7.6.0     RHEL-7.6.0 PC (Q35 + ICH9, 2009) (deprecated)
none                 empty machine

as evident q35 is alias for always the latest pc-q35-version supported by version of qemu.

Coupled with the validator logic in kubevirt only supported emulated types for AM64 are https://github.com/kubevirt/kubevirt/blob/v1.1.1/pkg/virt-config/virt-config.go#L50

Based on this info we should probably just drop pc-q35, and only leave q35.

If a user wishes to use a specific version they can edit the yaml and choose specific machine say pc-q35-2.12.

Closed
@harvesterhci-io-github-bot
Copy link

added backport-needed/1.3.1 issue: #5784.

Closed
@harvesterhci-io-github-bot
Copy link

added backport-needed/1.2.2 issue: #5785.

@a110605 a110605 mentioned this issue May 13, 2024
Merged
3 tasks
This was referenced May 13, 2024
Merged
Merged
@bk201 bk201 assigned a110605 and unassigned torchiaf May 13, 2024
@harvesterhci-io-github-bot
Copy link

harvesterhci-io-github-bot commented May 13, 2024 •
edited by bk201
Loading

Pre Ready-For-Testing Checklist

  • If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted?
    The HEP PR is at:

  • Where is the reproduce steps/test steps documented?
    The reproduce steps/test steps are at:

  • Create a VM or Edit a VM from GUI, in the "Advanced Options" tab -> Show more -> Machine type should only display None or q35. VM should start with both option.

  • Is there a workaround for the issue? If so, where is it documented?
    The workaround is at:

  • Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)?
    The PR is at:

    • Does the PR include the explanation for the fix or the feature?

    • Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
      The PR for the YAML change is at:
      The PR for the chart change is at:

  • If labeled: area/ui Has the UI issue filed or ready to be merged?
    The UI issue/PR is at:

  • If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged?
    The documentation/KB PR is at:

  • If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?

    • The automation skeleton PR is at:
    • The automation test case PR is at:

  • If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
    The compatibility issue is filed at:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@a110605 a110605

Labels
area/vm-lifecycle backport-needed/1.2.2 backport-needed/1.3.1 kind/bug Issues that are defects reported by users or that we know have reached a real release need-reprioritize not-require/test-plan Skip to create a e2e automation test issue regression reproduce/always Reproducible 100% of the time severity/1 Function broken (a critical incident with very high impact)
Projects
None yet
Milestone
v1.4.0
Development

No branches or pull requests
6 participants
@bk201 @lanfon72 @a110605 @ibrokethecloud @torchiaf @harvesterhci-io-github-bot