Overlay network connectivity issue - Calico v3.13.4 legacy iptables rules are not applied in RHEL 8

[meldafrawi]

RKE version: v1.4.2

Docker version: (docker version,docker info preferred) v19.03

Operating system and kernel: (cat /etc/os-release, uname -r preferred) AWS AMI - RHEL 8.3

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) AWS EC2

cluster.yml file:

bastion_host:
  address: 
  port: 22
  ssh_key_path: ~/.ssh/id_rsa
  user: ec2-user
kubernetes_version: v1.18.6-rancher1-1
nodes:
- address: 
  hostname_override: controlplane
  role:
  - controlplane
  - etcd
  user: ec2-user
- address: 
  hostname_override: worker-0
  role:
  - worker
  user: ec2-user
- address: 
  hostname_override: worker-1
  role:
  - worker
  user: ec2-user
- address: 
  hostname_override: worker-2
  role:
  - worker
  user: ec2-user

Steps to Reproduce:

• Create 4 EC2 instances
• Deploy k8s v1.18 using provided cluster.yml file
• Create a daemonset
• Exec into one pod of the daemonset, and try to ping other pod

Results:
Fail to ping other pods.

Analysis:

• RHEL 8.3 uses iptables v1.8.4 (nf_tables) -- No legacy iptables binary
• Till RHEL 7.9 it uses iptables v1.4.21
• I'm deploying k8s v1.18.6 usin RKE v1.4.2, which is using Calico v3.13.4 to provide overlay network.
  According to reproduce steps in this issue Calico autoselects iptables-legacy mode if ip_tables.ko preloaded on CentOS 8 projectcalico/calico#3709 (comment), I was able to verify Step 3 : Observe kube-proxy rules but no calico rules in the output of iptables-save in RHEL 8, and not in RHEL 7.9

[superseb]

RHEL 8 requires k8s 1.19.x and firewalld disabled. You can't use anything older than 1.19.x as it doesn't contain the needed logic to correctly determine iptables/nftables.