Multus ipvlan - 'Operation not supported'

[piyushewkani]

Hi,

I am kind of a rookie in k8s and container networking. For a specific project, I am trying to set up Multus with Calico on Openshift 3.11 platform. And in doing so, I have run into a particular issue which I am unable to figure out the fix. Would appreciate if someone can look into the error, I am facing and maybe suggest next steps.

I followed the quickstart guide to install Multus by running the Multus daemonset and then creating macvlan CRD. That all works perfectly fine. Now I am trying to create the secondary nic using ipvlan, but the pod initialization fails and gives an error which says:- Multus: Err adding pod to network "ipvlan-test": Multus: error in invoke Delegate add - "ipvlan": failed to create ipvlan: operation not supported

Information about my environment:-

• One master, one infra and one worker node -- Red Hat OpenShift 3.11 enterprise version running on RHOSP13. OS version - Red Hat Enterprise Linux Server release 7.6

• Each node has dual NICs, eth0 & eth1, set up with two different subnet '10.46.89.0/24' and '10.46.90.224/27'.

• Eth0 is used for OAM and external access at the moment.

• Eth1 is set up to provide pod's secondary network - 10.46.90.224/27. The expectation is to have pod externally accessible via this network for a specific application requirement which we are trying to deploy on Openshift.

• Calico was installed while deploying the cluster.

• CRD:-
  cat <<EOF | kubectl create -f - apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: ipvlan-test spec: config: '{ "cniVersion": "0.3.0", "type": "ipvlan", "master": "eth1", "ipam": { "type": "host-local", "ranges": [ [ { "subnet": "10.46.90.224/27", "rangeStart": "10.46.90.245", "rangeEnd": "10.46.90.250", "gateway": "10.46.90.225" } ] ], "routes": [ { "dst": "10.46.90.224/27", "gw": "10.46.90.225" } ] } }' EOF

• 00-multus.config (this was automatically created by deamonset):-
  { "name": "multus-cni-network", "type": "multus", "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig", "delegates": [{ "name": "k8s-pod-network", "cniVersion": "0.3.0", "plugins": [{ "type": "calico", "etcd_endpoints": "https://master.oc-test.cch-nam.nsn-rdnet.net:2379", "etcd_key_file": "/etc/cni/net.d/calico-tls/etcd-key", "etcd_cert_file": "/etc/cni/net.d/calico-tls/etcd-cert", "etcd_ca_cert_file": "/etc/cni/net.d/calico-tls/etcd-ca", "log_level": "info", "mtu": 1500, "ipam": { "type": "calico-ipam" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" } }, { "type": "portmap", "snat": true, "capabilities": { "portMappings": true } }] }] }

• sample POD yaml:-
  `cat <<EOF | kubectl create -f -
  apiVersion: v1
  kind: Pod
  metadata:
  name: samplepod-ipvlan
  annotations:
  k8s.v1.cni.cncf.io/networks: ipvlan-test
  spec:
  containers:

  • name: samplepod
    image: docker.io/centos/tools:latest
    command:
    • /sbin/init
      EOF`

• Error:-

(combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2d0d9dd69dc20a3a1f06c0517406985141c82ef25cbb3602da2f8dd552e80f6b" network for pod "samplepod-ipvlan": NetworkPlugin cni failed to set up pod "samplepod-ipvlan_piyush" network: Multus: Err adding pod to network "ipvlan-test": Multus: error in invoke Delegate add - "ipvlan": failed to create ipvlan: operation not supported

I am trying to deep dive into learning how multus creates secondary NICs and understand the config file to have a better understanding of what's happening here. Meanwhile, if anyone could look into it and provide their diagnoses, it will be greatly helpful.

Let me know if you need any further information.

[piyushewkani]

I just found out that ipvlan requires Linux kernel version higher than 4.1. The Red Hat Linux I am using has kernel version 3.10. I will update the kernel and check this deployment again.

If anyone has run into a similar issue, please let me know.

[piyushewkani]

Wow! Upgrading Linux kernel version to 4.4 fixed it! I am now able to create secondary nic using ipvlan.

However, I am unable to access the pod via nic2. I will troubleshoot this further.

Even though the main issue is resolved, I would like to keep this open while I figure out how to get external access working. Any suggestions?

[piyushewkani]

Hi,

So I was able to fix the issue of reachability. Basically, In OpenStack, I had to stop the anti-spoofing by disabling port-security on eth1 (10.46.90.224/27) of the worker node.

Glad that I was able to fix this by myself. Closing this issue!