[BUG] Longhorn storage network is incompatible with Multus version above v4.0.0

[c3y1huang]

Describe the bug (🐛 if you encounter this issue)

The network status annotation k8s.v1.cni.cncf.io/networks-status has been deprecated since Mutlus v3.6. And completely removed in Multus v4.0.0.

To Reproduce

https://longhorn.github.io/longhorn-tests/manual/release-specific/v1.3.0/test-storage-network/ but setup with Multus version above v4.0.0

Expected behavior

Tests should pass

Support bundle for troubleshooting

N/A

Environment

N/A

Additional context

#4937 (comment)

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at: test(manual): storage-network on Multus version above v4.0.0 longhorn-tests#1564

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:
  The PR for the chart change is at:

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at fix(storage-network): incompatible with Multus version above 4.0.0 longhorn-manager#2247

• Which areas/issues this PR might have potential impacts on?
  Area manager, storage-network
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at
  The automation test case PR is at
  The issue of automation test case implementation is at (please create by the template)

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at test(manual): storage-network on Multus version above v4.0.0 longhorn-tests#1564

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[innobead]

@c3y1huang Should we update the manual test case as well? longhorn/longhorn-tests#1564 it's good we already have that update.

And Should have storage network in k8s.v1.cni.cncf.io/networks-status instance manager pods annotations.

Should have 2 network in k8s.v1.cni.cncf.io/network-status annotation
kube-system/demo-192-168-0-0 should exist in k8s.v1.cni.cncf.io/network-status annotation
kube-system/demo-192-168-0-0 should use lhnet1 interface.
kube-system/demo-192-168-0-0 should be in 192.168.0.0/16 subnet. And Should be able to create/attach/detach/delete volumes successfully.

[innobead]

ref: k8snetworkplumbingwg/network-attachment-definition-client#17

[chriscchien]

Verified pass on longhorn master (longhorn manager 16c09f) with test steps

Environment: Ubuntu 22.04, v1.28.2+k3s1

v4.0.2 thin plugin

annotation

 Annotations:      k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "cbr0",
                        "interface": "eth0",
                        "ips": [
                            "10.42.2.14"
                        ],
                        "mac": "b2:bc:0e:ce:08:5d",
                        "default": true,
                        "dns": {},
                        "gateway": [
                            "10.42.2.1"
                        ]
                    },{
                        "name": "kube-system/demo-192-168-0-0",
                        "interface": "lhnet1",
                        "ips": [
                            "192.168.3.2"
                        ],
                        "mac": "06:90:3b:7f:18:8d",
                        "dns": {}
                    }]
                  k8s.v1.cni.cncf.io/networks: [{"namespace": "kube-system", "name": "demo-192-168-0-0", "interface": "lhnet1"}]
                  longhorn.io/last-applied-tolerations: []
                  longhorn.io/v2-data-engine: false

engine

root@ip-10-0-1-58:/home/ubuntu/k9s/k9s# k get engine pvc-281fa619-f354-438b-ba1b-b5f46e328f74-e-0 -n longhorn-system -o yaml | grep storage
  storageIP: 192.168.3.2

replica

root@ip-10-0-1-58:/home/ubuntu/k9s/k9s# k get replica pvc-281fa619-f354-438b-ba1b-b5f46e328f74-r-feee0239 -n longhorn-system -o yaml | grep storage
  storageIP: 192.168.3.2

v4.0.2 thick pulgin

annotation

Annotations:      k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "cbr0",
                        "interface": "eth0",
                        "ips": [
                            "10.42.0.17"
                        ],
                        "mac": "16:a1:b3:40:28:b2",
                        "default": true,
                        "dns": {},
                        "gateway": [
                            "10.42.0.1"
                        ]
                    },{
                        "name": "kube-system/demo-192-168-0-0",
                        "interface": "lhnet1",
                        "ips": [
                            "192.168.1.2"
                        ],
                        "mac": "06:5c:ad:65:8c:09",
                        "dns": {}
                    }]
                  k8s.v1.cni.cncf.io/networks: [{"namespace": "kube-system", "name": "demo-192-168-0-0", "interface": "lhnet1"}]
                  longhorn.io/last-applied-tolerations: []
                  longhorn.io/v2-data-engine: false

engine

root@ip-10-0-1-196:/home/ubuntu# k get engine pvc-c422eb22-11ff-4d50-8e5a-e3703e0ffda6-e-0 -n longhorn-system -o yaml | grep storage
  storageIP: 192.168.3.2

replica

root@ip-10-0-1-196:/home/ubuntu# k get replicas -A -o yaml | grep storage
    storageIP: 192.168.1.2
    storageIP: 192.168.3.2
    storageIP: 192.168.2.2