Can not bring up networking When stop a VM and start it again

[27149chen]

What happened:
stop a VM by virtctl stop xxx and start it again, an error happened Failed to start LSB: Bring up/down networking. and interface eth0 is down

What you expected to happen:
eth0 is up

How to reproduce it (as minimally and precisely as possible):

1. create a dv
2. create a vm using a pvc generated by the dv as boot device
3. start it, eth0 is up
4. stop it and start it again, eth0 is down

Additional context:
dv.yaml

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: "block-pvc"
spec:
  source:
      http:
         url: "https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2"
  pvc:
    storageClassName: xxx
    volumeMode: Block
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: "10Gi"

vm.yaml

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    a: b
  name: block-pvc
spec:
  running: false
  template:
    metadata:
      labels:
        a: b
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: filepvcdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
        resources:
          requests:
            memory: 1G
      terminationGracePeriodSeconds: 0
      volumes:
      - name: filepvcdisk
        persistentVolumeClaim:
          claimName: block-pvc
      - cloudInitNoCloud:
          userData: |-
            #cloud-config
            chpasswd:
              expire: false
            password: fedora
            user: fedora
        name: cloudinitdisk

Environment:

• KubeVirt version (use virtctl version):

Client Version: version.Info{GitVersion:"v1.1.1", GitCommit:"689c0e66cc6893f311dd648ff32e247203b6c96a", GitTreeState:"clean", BuildDate:"2023-12-25T10:58:20Z", GoVersion:"go1.19.9", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{GitVersion:"v1.1.1", GitCommit:"689c0e66cc6893f311dd648ff32e247203b6c96a", GitTreeState:"clean", BuildDate:"2023-12-25T11:56:11Z", GoVersion:"go1.19.9", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes version (use kubectl version): N/A
• VM or VMI specifications: N/A
• Cloud provider or hardware configuration: N/A
• OS (e.g. from /etc/os-release): N/A
• Kernel (e.g. uname -a): N/A
• Install tools: N/A
• Others: N/A

[27149chen]

It seems that a new MAC address will be generated on every VM startup, but field HWADDR in /etc/sysconfig/network-scripts/ifcfg-eth0 is never changed, which causing the mismatch, and the network refuses to start

[27149chen]

Similar issues:
#11415
#11577
#11113 (comment)
#9993
#1251 (comment)

[aburdenthehand]

/cc @orelmisan

[reski-rukmantiyo]

this also happen in ubuntu 22.04. Any solutions for this?

[27149chen]

a workaround is removing field HWADDR in /etc/sysconfig/network-scripts/ifcfg-eth0, and restart again

[EdDev]

It seems that a new MAC address will be generated on every VM startup, but field HWADDR in /etc/sysconfig/network-scripts/ifcfg-eth0 is never changed, which causing the mismatch, and the network refuses to start

Yes, this is indeed correct.
Unless you specify explicitly an interface with a mac address, it will not be persisted between stop/start.
The preferred option is to use something like KMP that allocates a mac address from a pool and assures there are not collisions.

[reski-rukmantiyo]

In my case, i'm using Ubuntu families (20.04, 22.04), i remove mac-address and set-name in cloud-init process.
Because I cannot found /etc/sysconfig/network-scripts/ifcfg-eth0 or similiarities in the OS

[fabiand]

I believe that on the Linux side, https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/ is addressing this problem. MACADDR based syscfg scripts are indeed error prone in our flow.

…

Message ID: ***@***.***>

[27149chen]

It seems that a new MAC address will be generated on every VM startup, but field HWADDR in /etc/sysconfig/network-scripts/ifcfg-eth0 is never changed, which causing the mismatch, and the network refuses to start

Yes, this is indeed correct. Unless you specify explicitly an interface with a mac address, it will not be persisted between stop/start. The preferred option is to use something like KMP that allocates a mac address from a pool and assures there are not collisions.

It's too complex to me, can we just clear the field HWADDR or something similiar?

[fabiand]

Oh yes, you can clear the HWADDR field in sysconfig, however, the solution depends on your guest operating system/distribution.

[27149chen]

Oh yes, you can clear the HWADDR field in sysconfig, however, the solution depends on your guest operating system/distribution.

I mean can kubevirt do it automatically?

[EdDev]

You can set the mac address explicitly on the VM spec. Then it will persist between boots.
But you need to be careful not to define duplicate addresses in your cluster.

https://kubevirt.io/api-reference/v1.2.0/definitions.html#_v1_interface
The field name is macAddress.

[27149chen]

You can set the mac address explicitly on the VM spec. Then it will persist between boots. But you need to be careful not to define duplicate addresses in your cluster.

https://kubevirt.io/api-reference/v1.2.0/definitions.html#_v1_interface The field name is macAddress.

@EdDev Sorry, that's not what I want. Say I have 100 vms, I just want to make sure they are still work after restart, I don't want to carefully pick 100 valid MACs and assign them one by one

[EdDev]

You can set the mac address explicitly on the VM spec. Then it will persist between boots. But you need to be careful not to define duplicate addresses in your cluster.
https://kubevirt.io/api-reference/v1.2.0/definitions.html#_v1_interface The field name is macAddress.

@EdDev Sorry, that's not what I want. Say I have 100 vms, I just want to make sure they are still work after restart, I don't want to carefully pick 100 valid MACs and assign them one by one

As mentioned in #11585 (comment) , you can use KMP.

At the moment, there is no other option.

[27149chen]

@EdDev , ok, I see. thanks