[vhost] vm_wait_for_boot timed out

[ksztyber]

00:20:15.991  ...WARN: timeout waiting for machines to boot
00:20:15.991  WARN: ================
00:20:15.991  WARN: QEMU LOG:
00:20:15.991  WARN: VM LOG:
00:20:15.991  [    0.000000] Linux version 6.5.10-200.fc38.x86_64 (mockbuild@cbf61e0e869d4c6d90ab7044b9ed9a96) (gcc (GCC) 13.2.1 20231011 (Red Hat 13.2.1-4), GNU ld version 2.39-15.fc38) #1 SMP PREEMPT_DYNAMIC Thu Nov  2 19:59:55 UTC 2023
00:20:15.991  [    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.5.10-200.fc38.x86_64 root=UUID=a280b604-6023-4ba5-bb9e-80d612f84b0d ro rootflags=subvol=root selinux=0 apparmor=0 net.ifnames=0 console=ttyS0 scsi_mod.use_blk_mq=1 console=ttyS0
00:20:15.991  [    0.000000] BIOS-provided physical RAM map:
00:20:15.991  [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
00:20:15.991  [    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
00:20:15.991  [    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
00:20:15.991  [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003ffdcfff] usable
00:20:15.991  [    0.000000] BIOS-e820: [mem 0x000000003ffdd000-0x000000003fffffff] reserved
00:20:15.991  [    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
00:20:15.991  [    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
...
00:20:15.996   08:29:34	-- vhost/common.sh@967 -- # return 1
...
00:20:15.996     149 	
00:20:15.996     150 	$rpc_py vhost_get_controllers
00:20:15.996     151 	
00:20:15.996     152 	# Run VMs
00:20:15.996     153 	vm_run $used_vms
00:20:15.996  => 154 	vm_wait_for_boot 300 $used_vms
00:20:15.996     155 	
00:20:15.996     156 	# Get disk names from VMs and run FIO traffic
00:20:15.996     157 	
00:20:15.996     158 	fio_disks=""
00:20:15.996     159 	for vm_num in $used_vms; do

Link to the failed CI build

https://ci.spdk.io/results/autotest-per-patch/builds/120668/archive/vhost-phy-autotest_23443/build.log
https://ci.spdk.io/public_build/autotest-per-patch_120668.html

[spdkci]

Another instance of this failure. Reported by @ksztyber. log: https://ci.spdk.io/public_build/autotest-per-patch_120668.html

[mikeBashStuff]

This is literally unreproducible. I do have a one suspect but it's a complete stretch which is, considering how impossible it is to reproduce this issue, almost untestable. I will close it for now. If this appears in the wild once more I will try to go with my gut and try to implement a "workaround".

[spdkci]

Another instance of this failure. Reported by @slawekptak. log: https://ci.spdk.io/public_build/autotest-per-patch_123430.html