How to configure internet access if VPP uses non-default interface?

[Bolodya1997]

Environment

• Calico/VPP version: v0.17.0-calicov3.20.2.
• Kubernetes version: 1.21.1 .
• Deployment type: bare-metal on equinix.metal, host nodes OS is Ubuntu 20.04 LTS.
• Network configuration:
  • master node:
    1. default interface + ip - bond0 + 139.178.65.125.
    2. VPP interface + ip - eno2 + 10.0.0.1/30.
  • worker node:
    1. default interface + ip - bond0 + 139.178.65.127.
    2. VPP interface + ip - eno2 + 10.0.0.2/30.

Issue description
Pods are unable to ping internet when VPP uses non-default interface.

To Reproduce
Steps to reproduce the behavior:

1. Start 2 bare-metal nodes with the following interfaces configured:
   1. bond0 - bond interface, has access to the internet.
   2. eno2 - simple interface, has access to the eno2 on the other node.
2. Setup additional IP addresses:
   1. master node - 10.0.0.1/30 dev eno2.
   2. worker node - 10.0.0.2/30 dev eno2.
3. Create kubernetes cluster.
4. Edit Calico calico-vpp-nohuge.yaml deployment:

  vpp_dataplane_interface: eno2

5. Start Calico with the modified deployment.
6. Run kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml.
7. Run kubectl exec dnsutils -- ping -c 4 8.8.8.8.
8. ping fails.

Expected behavior
dnsutils pod should be able to access internet.

[Bolodya1997]

I am not using bond0 to configure Calico VPP because:

1. I am not sure if it works with bond interfaces.
2. I don't have local access to the nodes, so breaking this interface will break all access to the nodes.

[edwarnicke]

@Bolodya1997 Can't you get management access to the nodes via eno2 ?

[AloysAugustin]

@Bolodya1997 although not tested, bond interfaces should be supported with the af_packet driver which should be sufficient for testing.

As @edwarnicke suggests, if possible I think it would be good to setup an independent management interface so that you can keep ssh access if anything goes wrong - maybe by adding another server connected to your existing hosts on the eno2 network that you could use as a jump host.