BMO e2e cleanup potentially leaving VMs?

[lentzi90]

What steps did you take and what happened:

See https://jenkins.nordix.org/view/Metal3/job/metal3-bmo-e2e-test-pull/214/

[2024-01-24T11:43:09.124Z] + virt-install --connect qemu:///system --name bmo-e2e-0 --description 'Virtualized BareMetalHost' --osinfo=ubuntu-lts-latest --ram=4096 --vcpus=2 --disk size=20 --graphics=none --console pty,target_type=serial --serial file,path=/var/log/libvirt/qemu/bmo-e2e-0-serial0.log --xml ./devices/serial/@type=pty --xml ./devices/serial/log/@file=/var/log/libvirt/qemu/bmo-e2e-0-serial0.log --xml ./devices/serial/log/@append=on --pxe --network network=baremetal-e2e,mac=00:60:2f:31:81:01 --noautoconsole

[2024-01-24T11:43:09.124Z] ERROR    The MAC address '00:60:2f:31:81:01' is in use by another virtual machine. (Use --check mac_in_use=off or --check all=off to override)

script returned exit code 1

Anything else you would like to add:

My suspicion is that there is an issue with the cleanup script that somehow left a VM behind from the previous job. Then when this job started we got the conflict.
I could easily be mistaken though. Perhaps minikube somehow happened to get exactly the same MAC as we are giving the BMH VM? 🤯

Edit:
I think I found the issue now!

baremetal-operator/hack/clean-e2e.sh

Line 11 in 53281e6

virsh_vms=$(virsh list --name --all)

This is not using the qemu:///system socket so it will not find the VMs.
I don't think we should select all VMs either. Somehow I missed that while reviewing.
We should instead do as here. In fact we should probably reuse that script and just take the prefix for what VMs to delete from a variable.

/kind bug

[lentzi90]

/triage accepted

[lentzi90]

/assign @mquhuy