Update custom-network examples

Signed-off-by: Rastislav Szabo <raszabo@cisco.com>

README.md

@@ -16,8 +16,9 @@ For more details see [https://contivpp.io/](https://contivpp.io/)
 ## Features
 * kube-proxy implementation on VPP - in the userspace (full implemenatation of [k8s services](docs/dev-guide/SERVICES.md) & [k8s policies](docs/dev-guide/POLICIES.md))
 * support for [multiple interfaces per pod](docs/operation/CUSTOM_POD_INTERFACES.md), including memif interfaces
+* support for [multiple isolated L2 or L3 networks](k8s/examples/custom-network/README.md)
 * [IPv6 support](docs/setup/IPV6.md), segment routing implementation of k8s services ([SRv6](docs/setup/SRV6.md))
-* (in progress) [service chaining between pods](k8s/examples/sfc/README.md) for CNF workloads
+* [service chaining between pods](k8s/examples/sfc/README.md) for CNF workloads
 
 
 ## Releases

docs/operation/CUSTOM_POD_INTERFACES.md

@@ -10,23 +10,28 @@ be one of the 3 supported types:
  - Linux veth (virtual ethernet) interface
  - memif interface (requires memif-compatible application running in the pod)
 
-In order to use this feature, the Kubelet feature gate `KubeletPodResources` needs to be enabled,
-e.g. add the following into in the `KUBELET_EXTRA_ARGS` in the `/etc/default/kubelet` file:
-```
-KUBELET_EXTRA_ARGS=--feature-gates KubeletPodResources=true
-```
-
 Custom interfaces can be requested using annotations in pod definition. The name
 of the annotation is `contivpp.io/custom-if` and its value can be a comma-separated
 list of custom interfaces in the `<custom-interface-name>/<interface-type>/<network>`
-format. The `<network>` part is optional, leaving it unspecified means the default pod
-network. Apart from the default pod network, only a special `stub` network is 
-currently supported, which leaves the interface without any IP address and routes pointing to it.
+format. 
+
+The `<network>` part is optional, leaving it unspecified means the default pod
+network. Apart from the default pod network, there is another special network with the name `stub`, 
+which leaves the interface without any IP address and routes pointing to it. The stub interfaces
+may be used together with [service function chaining](../../k8s/examples/sfc/README.md).
+Apart form `default` and `stub`, you can reference any other L2 or L3 
+[custom network](../../k8s/examples/custom-network/README.md) defined via CRDs.
 
 K8s service traffic can be also redirected towards custom interfaces using the `contivpp.io/service-endpoint-if` 
 pod annotation, where the value should refer to the name of a custom interface in the default pod network.
 An example of this feature can be found in the [k8s example folder](../../k8s/examples/memif).
 
+In order to use this feature, the Kubelet feature gate `KubeletPodResources` needs to be enabled,
+e.g. add the following into in the `KUBELET_EXTRA_ARGS` in the `/etc/default/kubelet` file:
+```
+KUBELET_EXTRA_ARGS=--feature-gates KubeletPodResources=true
+```
+
 An example of a pod definition that connects a pod with a default interface plus one 
 extra tap interface with the name `tap1` and one extra veth interface with the name `veth1`:
 

k8s/examples/custom-network/README.md

@@ -8,28 +8,35 @@ For more information on multi-interface pods in Contiv-VPP, look at the
 [custom pod interfaces documentation](../../../docs/operation/CUSTOM_POD_INTERFACES.md).
 
 ## Demo Topology
-This demo deploys 2 CNF pods, each with one additional TAP interface in a layer 2 
-custom network defined as a CRD. Additionally, an external DPDK interface is defined 
-with another CRD, and is placed into the same custom network as well. This means that the 
-CNF interfaces and the external DPDK interface will be interconnected in a bridge 
-domain on VPP:
+This demo deploys 2 custom networks defined in CRDs, one interconnecting pods on layer 2, 
+the other on layer 3. Both networks contains 2 CNF pods, each with one additional TAP interface in the 
+custom network. Additionally, two external DPDK sub-interfaces are defined 
+using CRDs, and are placed into the custom networks as well. This means that the 
+CNF interfaces and the external DPDK sub-interfaces will be interconnected either in a bridge 
+domain (L2 network) or in a separate VRF (L3 network) on VPP:
 
 ![CNF - Custom Network](custom-network.png)
 
-Note that this demo refers to a single-node k8s cluster deployment. If the CNFs are deployed
-on different k8s nodes, they will be connected to different bridge domains, but still
-interconnected on L2 layer using a VXLAN tunnel between the bridge domains 
-(actually, this is not working now, but will be implemented soon).
+Note that this demo refers to a single-node k8s cluster deployment for simplicity. If the CNFs are deployed
+on different k8s nodes, they will be connected to bridge domains / VRFs on their perspective nodes, 
+but still interconnected using a VXLAN tunnel between the nodes, so the L2 / L3 communication between the pods
+should work normally.
 
-This folder contains three yaml files:
-  - [custom-network.yaml](custom-network.yaml) defines a L2 custom network with the name `l2net`
-  - [cnfs-linux.yaml](cnfs-linux.yaml) defines CNF pods with additional interfaces in the `l2net`
+This folder contains 6 yaml files:
+  - [custom-network-l2.yaml](custom-network-l2.yaml) defines a L2 custom network with the name `l2net`
+  - [cnfs-linux-l2.yaml](cnfs-linux-l2.yaml) defines CNF pods with additional interfaces in the `l2net`
   network (using the annotation `contivpp.io/custom-if: tap1/tap/l2net`)
-  - [external-interface.yaml](external-interface.yaml) defines an external DPDK sub-interface
+  - [external-interface-l2.yaml](external-interface-l2.yaml) defines an external DPDK sub-interface
   in the `l2net` network.
-
+  - [custom-network-l3.yaml](custom-network-l3.yaml) defines a L3 custom network with the name `l3net`
+  - [cnfs-linux-l3.yaml](cnfs-linux-l3.yaml) defines CNF pods with additional interfaces in the `l3net`
+  network (using the annotation `contivpp.io/custom-if: tap1/tap/l3net`)
+  - [external-interface-l3.yaml](external-interface-l3.yaml) defines an external DPDK sub-interface
+  in the `l3net` network.
+
 ### Setup
- Before deploying, [external interface deployment yaml](external-interface.yaml) needs to be modified 
+ Before deploying, [L2 external interface deployment yaml](external-interface-l2.yaml) and 
+ [L3 external interface deployment yaml](external-interface-l3.yaml) need to be modified 
  to match your setup:
 
  - change `node` identifier to match your hostname:
@@ -38,27 +45,27 @@ This folder contains three yaml files:
 ```yaml
   nodes:
     - node: k8s-master
-      vppInterfaceName: GigabitEthernet0/a/0
+      vppInterfaceName: GigabitEthernet0/9/0
 ```
 
 Don't forget to [modify your VPP startup config file](../../../docs/setup/VPP_CONFIG.md) 
 with the PCI address of the external interface.
 
-## Demo
+
+## Demo - L2 Network
 Start by deploying the yaml files:
  ```bash
-kubectl apply -f custom-network.yaml
-kubectl apply -f cnfs-linux.yaml
-kubectl apply -f external-interface.yaml
+kubectl apply -f custom-network-l2.yaml
+kubectl apply -f cnfs-linux-l2.yaml
+kubectl apply -f external-interface-l2.yaml
 ```
 
-Verify that `linux-cnf` pods are running:
+Verify that `linux-cnf1` and `linux-cnf2` pods are running:
 ```bash
 $ kubectl get pods -o wide
 NAME         READY   STATUS    RESTARTS   AGE   IP         NODE      NOMINATED NODE   READINESS GATES
 linux-cnf1   1/1     Running   0          56m   10.1.1.4   lubuntu   <none>           <none>
 linux-cnf2   1/1     Running   0          56m   10.1.1.5   lubuntu   <none>           <none>
-
 ```
 
 Verify that CNF taps and DPDK sub-interface are all connected into the same bridge domain on the
@@ -72,14 +79,84 @@ $ sudo vppctl
 
 vpp# sh inter addr
 ...
-GigabitEthernet0/a/0.200 (up):
-  L2 bridge bd-id 2 idx 2 shg 1  
+GigabitEthernet0/9/0.200 (up):
+  L2 bridge bd-id 2 idx 2 shg 0  
 ...
 tap5 (up):
-  L2 bridge bd-id 2 idx 2 shg 1  
+  L2 bridge bd-id 2 idx 2 shg 0  
 tap6 (up):
-  L2 bridge bd-id 2 idx 2 shg 1  
+  L2 bridge bd-id 2 idx 2 shg 0  
 ```
 
 Try sending some L2 broadcast traffic (e.g. ARP) from one of the CNFs or external interface.
-You should see it coming to all other interfaces in the bridge domain.
+You should see it coming to all other interfaces in the bridge domain.
+
+
+## Demo - L3 Network
+Start by deploying the yaml files:
+ ```bash
+kubectl apply -f custom-network-l3.yaml
+kubectl apply -f cnfs-linux-l3.yaml
+kubectl apply -f external-interface-l3.yaml
+```
+
+Verify that `linux-cnf3` and `linux-cnf4` pods are running:
+```bash
+$ kubectl get pods -o wide
+NAME         READY   STATUS    RESTARTS   AGE   IP         NODE      NOMINATED NODE   READINESS GATES
+linux-cnf3   1/1     Running   0          56m   10.1.1.6   lubuntu   <none>           <none>
+linux-cnf4   1/1     Running   0          56m   10.1.1.7   lubuntu   <none>           <none>
+```
+
+Verify that CNF taps and DPDK sub-interface are all connected into the same VRF on the vswitch VPP:
+```bash
+$ sudo vppctl
+    _______    _        _   _____  ___ 
+ __/ __/ _ \  (_)__    | | / / _ \/ _ \
+ _/ _// // / / / _ \   | |/ / ___/ ___/
+ /_/ /____(_)_/\___/   |___/_/  /_/    
+
+vpp# sh inter addr
+...
+GigabitEthernet0/9/0.300 (up):
+  L3 10.100.100.100/16
+...
+tap10 (up): 
+  unnumbered, use loop2
+  L3 10.100.1.1/24 ip4 table-id 10 fib-idx 2
+tap9 (up): 
+  unnumbered, use loop2
+  L3 10.100.1.1/24 ip4 table-id 10 fib-idx 2
+```
+
+Verify that `tap1` interfaces in `linux-cnf3` and `linux-cnf4` have an ip address assigned:
+```bash
+$ kubectl exec -it linux-cnf3 -- ip addr
+46: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel qlen 1000
+    link/ether 02:fe:89:de:04:4f brd ff:ff:ff:ff:ff:ff
+    inet 10.100.1.2/32 brd 10.100.1.3 scope global tap1
+       valid_lft forever preferred_lft forever
+    inet6 fe80::b89f:e6ff:fed7:dedb/64 scope link 
+       valid_lft forever preferred_lft forever
+
+$ kubectl exec -it linux-cnf4 -- ip addr
+48: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel qlen 1000
+    link/ether 02:fe:89:de:04:4f brd ff:ff:ff:ff:ff:ff
+    inet 10.100.1.3/32 brd 10.100.1.3 scope global tap1
+       valid_lft forever preferred_lft forever
+    inet6 fe80::b89f:e6ff:fed7:dedb/64 scope link 
+       valid_lft forever preferred_lft forever
+```
+
+Ping from `linux-cnf3` to `linux-cnf4`:
+```bash
+$ kubectl exec -it linux-cnf3 -- ping 10.100.1.3
+PING 10.100.1.3 (10.100.1.3): 56 data bytes
+64 bytes from 10.100.1.3: seq=0 ttl=63 time=0.797 ms
+64 bytes from 10.100.1.3: seq=1 ttl=63 time=0.455 ms
+64 bytes from 10.100.1.3: seq=2 ttl=63 time=1.920 ms
+^C
+--- 10.100.1.3 ping statistics ---
+3 packets transmitted, 3 packets received, 0% packet loss
+round-trip min/avg/max = 0.455/1.057/1.920 ms
+```

k8s/examples/custom-network/cnfs-linux-l3.yaml

@@ -3,11 +3,11 @@
 apiVersion: v1
 kind: Pod
 metadata:
-  name: linux-cnf1
+  name: linux-cnf3
   annotations:
     contivpp.io/custom-if: tap1/tap/l3net
   labels:
-    cnf: linux-cnf1
+    cnf: linux-cnf3
 spec:
   containers:
     - name: busybox
@@ -24,11 +24,11 @@ spec:
 apiVersion: v1
 kind: Pod
 metadata:
-  name: linux-cnf2
+  name: linux-cnf4
   annotations:
     contivpp.io/custom-if: tap1/tap/l3net
   labels:
-    cnf: linux-cnf2
+    cnf: linux-cnf4
 spec:
   containers:
     - name: busybox

k8s/examples/custom-network/external-interface.yaml → k8s/examples/custom-network/external-interface-l2.yaml

@@ -2,11 +2,11 @@
 apiVersion: contivpp.io/v1
 kind: ExternalInterface
 metadata:
-  name: vlan-200
+  name: l2net-lan
 spec:
   type: L2
   network: l2net
   nodes:
-    - node: k8s-master
-      vppInterfaceName: GigabitEthernet0/a/0
+    - node: lubuntu
+      vppInterfaceName: GigabitEthernet0/9/0
       VLAN: 200

k8s/examples/custom-network/external-interface-l3.yaml

@@ -8,5 +8,6 @@ spec:
   network: l3net
   nodes:
     - node: lubuntu
-      vppInterfaceName: GigabitEthernet0/a/0
+      vppInterfaceName: GigabitEthernet0/9/0
       IP: 10.100.100.100/16
+      VLAN: 300