IPAM working properly but pod-pod networking not working.

[chowmean]

Is this a request for help?:

Yes

---

Is this an ISSUE or FEATURE REQUEST? (choose one):

Issue

Which release version?:

v1.0.26

Which component (CNI/IPAM/CNM/CNS):

CNI
Which Operating System (Linux/Windows):

LINUX

For Linux: Include Distro and kernel version using "uname -a"

#44-Ubuntu SMP Thu Feb 21 14:24:01 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

While using this container networking with self-hosted Kubernetes cluster on Azure VMs. I have preallocated IPs as secondary IP to the machines and installed the CNIs. IP assignment to the pods are working totally fine from the allocated secondary IP but pod-pod and pod-host networking not working. Need help on this. If there is any installation document or how to run a test cluster. It will be very helpful.

Which Orchestrator and version (e.g. Kubernetes, Docker)

Kubernetes

What happened:

Pod-pod or pod-host networking not working.

What you expected to happen:

Pod-pod networking to happen.

[tamilmani1989]

Hi @chowmean I have a question - Does pod to internet connectivity work?

Can you send me following files from affected VM?

1. /var/log/azure-vnet*. All log files starts with azure-vnet under /var/log/.
2. /var/run/azure-vnet.json
3. /var/run/azure-vnet-ipam.json

[chowmean]

1. No pod-internet connectivity doesn't work.

2. There are no logs in /var/log/azure. There is no folder with name azure-vnet* only azure is present and there are no files inside it.

3. Content of the IPAM and VNET json are below.
   azure-vnet.json

    {
            "Network": {
                    "Version": "v1.0.26",
                    "TimeStamp": "2019-09-03T18:15:05.745124601+05:30",
                    "ExternalInterfaces": {
                            "eth0": {
                                    "Name": "eth0",
                                    "Networks": {
                                            "azure": {
                                                    "Id": "azure",
                                                    "Mode": "bridge",
                                                    "VlanId": 0,
                                                    "Subnets": [
                                                            {
                                                                    "Family": 2,
                                                                    "Prefix": {
                                                                            "IP": "10.50.11.0",
                                                                            "Mask": "////AA=="
                                                                    },
                                                                    "Gateway": "10.50.11.1"
                                                            }
                                                    ],
                                                    "Endpoints": {
                                                            "01ebef8b-eth0": {
                                                                    "Id": "01ebef8b-eth0",
                                                                    "SandboxKey": "",
                                                                    "IfName": "azv17a9884d53792",
                                                                    "HostIfName": "azv17a9884d5379",
                                                                    "MacAddress": "UpZC41QCc",
                                                                    "InfraVnetIP": {
                                                                            "IP": "",
                                                                            "Mask": null
                                                                    },
                                                                    "LocalIP": "",
                                                                    "IPAddresses": [
                                                                            {
                                                                                    "IP": "10.50.11.20",
                                                                                    "Mask": "////AA=="
                                                                            }
                                                                    ],
                                                                    "Gateways": [
                                                                            "10.50.11.1"
                                                                    ],
                                                                    "DNS": {
                                                                            "Suffix": "",
                                                                            "Servers": [
                                                                                    "168.63.129.16"
                                                                            ]
                                                                    },
                                                                    "Routes": [
                                                                            {
                                                                                    "Dst": {
                                                                                            "IP": "0.0.0.0",
                                                                                            "Mask": "AAAAAA=="
                                                                                    },
                                                                                    "Src": "",
                                                                                    "Gw": "10.50.11.1",
                                                                                    "Protocol": 0,
                                                                                    "DevName": "",
                                                                                    "Scope": 0
                                                                            }
                                                                    ],
                                                                    "VlanID": 0,
                                                                    "EnableSnatOnHost": false,
                                                                    "EnableInfraVnet": false,
                                                                    "EnableMultitenancy": false,
                                                                    "AllowInboundFromHostToNC": false,
                                                                    "AllowInboundFromNCToHost": false,
                                                                    "NetworkNameSpace": "/proc/28588/ns/net",
                                                                    "ContainerID": "01ebef8b04ced3dae7fd19d98d0b4951870544cde3b7120ace636b47b9df4192d",
                                                                    "PODName": "perf-cli2",
                                                                    "PODNameSpace": "default"
                                                            },
                                                            "5f416e2a-eth0": {
                                                                    "Id": "5f416e2a-eth0",
                                                                    "SandboxKey": "",
                                                                    "IfName": "azv43437848b08d2",
                                                                    "HostIfName": "azv43437848b08d",
                                                                    "MacAddress": "GjqmdDXpU",
                                                                    "InfraVnetIP": {
                                                                            "IP": "",
                                                                            "Mask": null
                                                                    },
                                                                    "LocalIP": "",
                                                                    "IPAddresses": [
                                                                            {
                                                                                    "IP": "10.50.11.19",
                                                                                    "Mask": "////AA=="
                                                                            }
                                                                    ],
                                                                    "Gateways": [
                                                                            "10.50.11.1"
                                                                    ],
                                                                    "DNS": {
                                                                            "Suffix": "",
                                                                            "Servers": [
                                                                                    "168.63.129.16"
                                                                            ]
                                                                    },
                                                                    "Routes": [
                                                                            {
                                                                                    "Dst": {
                                                                                            "IP": "0.0.0.0",
                                                                                            "Mask": "AAAAAA=="
                                                                                    },
                                                                                    "Src": "",
                                                                                    "Gw": "10.50.11.1",
                                                                                    "Protocol": 0,
                                                                                    "DevName": "",
                                                                                    "Scope": 0
                                                                            }
                                                                    ],
                                                                    "VlanID": 0,
                                                                    "EnableSnatOnHost": false,
                                                                    "EnableInfraVnet": false,
                                                                    "EnableMultitenancy": false,
                                                                    "AllowInboundFromHostToNC": false,
                                                                    "AllowInboundFromNCToHost": false,
                                                                    "NetworkNameSpace": "/proc/26243/ns/net",
                                                                    "ContainerID": "5f416e2a641534246802ee16ffd31c9cac555c48889d7ef265cb84cd87203ebd7",
                                                                    "PODName": "perf",
                                                                    "PODNameSpace": "default"
                                                            }
                                                    },
                                                    "DNS": {
                                                            "Suffix": "",
                                                            "Servers": [
                                                                    "168.63.129.16"
                                                            ]
                                                    },
                                                    "EnableSnatOnHost": false,
                                                    "SnatBridgeIP": ""
                                            }
                                    },
                                    "Subnets": [
                                            "10.50.11.0/24"
                                    ],
                                    "BridgeName": "azure0",
                                    "DNSInfo": {
                                            "Suffix": "",
                                            "Servers": null
                                    },
                                    "MacAddress": "AA06B8kU",
                                    "IPAddresses": [
                                            {
                                                    "IP": "10.50.11.15",
                                                    "Mask": "////AA=="
                                            }
                                    ],
                                    "Routes": [
                                            {
                                                    "Family": 2,
                                                    "Dst": null,
                                                    "Src": "",
                                                    "Gw": "10.50.11.1",
                                                    "Tos": 0,
                                                    "Table": 254,
                                                    "Protocol": 3,
                                                    "Scope": 0,
                                                    "Type": 1,
                                                    "Flags": 0,
                                                    "Priority": 0,
                                                    "LinkIndex": 5,
                                                    "ILinkIndex": 0
                                            }
                                    ],
                                    "IPv4Gateway": "10.50.11.1",
                                    "IPv6Gateway": "::"
                            }
                    }
            }
    }
   

azure-vnet-ipam.json

    {
            "IPAM": {
                    "Version": "v1.0.26",
                    "TimeStamp": "2019-09-03T18:15:05.699040282+05:30",
                    "AddressSpaces": {
                            "local": {
                                    "Id": "local",
                                    "Scope": 0,
                                    "Pools": {
                                            "10.50.11.0/24": {
                                                    "Id": "10.50.11.0/24",
                                                    "IfName": "eth0",
                                                    "Subnet": {
                                                            "IP": "10.50.11.0",
                                                            "Mask": "////AA=="
                                                    },
                                                    "Gateway": "10.50.11.1",
                                                    "Addresses": {
                                                            "10.50.11.19": {
                                                                    "ID": "",
                                                                    "Addr": "10.50.11.19",
                                                                    "InUse": true
                                                            },
                                                            "10.50.11.20": {
                                                                    "ID": "",
                                                                    "Addr": "10.50.11.20",
                                                                    "InUse": true
                                                            },
                                                            "10.50.11.26": {
                                                                    "ID": "",
                                                                    "Addr": "10.50.11.26",
                                                                    "InUse": false
                                                            },
                                                            "10.50.11.27": {
                                                                    "ID": "",
                                                                    "Addr": "10.50.11.27",
                                                                    "InUse": false
                                                            },
                                                            "10.50.11.28": {
                                                                    "ID": "",
                                                                    "Addr": "10.50.11.28",
                                                                    "InUse": false
                                                            }
                                                    },
                                                    "IsIPv6": false,
                                                    "Priority": 0,
                                                    "RefCount": 1
                                            }
                                    }
                            }
                    }
            }
    }

[tamilmani1989]

@chowmean Didnt you see azure-vnet.log or azure-vnet-ipam.log under /var/log? Can you do ls under /var/log directory?

[chowmean]

Got it. I thought it was in azure-vnet directory. Below is the content..
azure-vnet.log

[tamilmani1989]

Can you send me output of:

1. sudo sysctl net.ipv4.ip_forward
2. sudo iptables -t filter -L FORWARD

[chowmean]

1. Forwarding is enabled.
   net.ipv4.ip_forward = 1

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */```

[tamilmani1989]

ok its little confusing. Looking at .json files, i see 10.50.11.19 and 10.50.11.20 are allocated IPs but from log file i see 10.50.11.18 was only allocated IP. Are log files and json files from same VM?

[chowmean]

So sorry my mistake below are the json from this VM.
azure-vnet.json

    {
            "Network": {
                    "Version": "v1.0.26",
                    "TimeStamp": "2019-09-03T18:11:41.970862226+05:30",
                    "ExternalInterfaces": {
                            "eth0": {
                                    "Name": "eth0",
                                    "Networks": {
                                            "azure": {
                                                    "Id": "azure",
                                                    "Mode": "bridge",
                                                    "VlanId": 0,
                                                    "Subnets": [
                                                            {
                                                                    "Family": 2,
                                                                    "Prefix": {
                                                                            "IP": "10.50.11.0",
                                                                            "Mask": "////AA=="
                                                                    },
                                                                    "Gateway": "10.50.11.1"
                                                            }
                                                    ],
                                                    "Endpoints": {
                                                            "ebb957c7-eth0": {
                                                                    "Id": "ebb957c7-eth0",
                                                                    "SandboxKey": "",
                                                                    "IfName": "azv895bc5f56192",
                                                                    "HostIfName": "azv895bc5f5619",
                                                                    "MacAddress": "ltSgHMQD",
                                                                    "InfraVnetIP": {
                                                                            "IP": "",
                                                                            "Mask": null
                                                                    },
                                                                    "LocalIP": "",
                                                                    "IPAddresses": [
                                                                            {
                                                                                    "IP": "10.50.11.18",
                                                                                    "Mask": "////AA=="
                                                                            }
                                                                    ],
                                                                    "Gateways": [
                                                                            "10.50.11.1"
                                                                    ],
                                                                    "DNS": {
                                                                            "Suffix": "",
                                                                            "Servers": [
                                                                                    "168.63.129.16"
                                                                            ]
                                                                    },
                                                                    "Routes": [
                                                                            {
                                                                                    "Dst": {
                                                                                            "IP": "0.0.0.0",
                                                                                            "Mask": "AAAAAA=="
                                                                                    },
                                                                                    "Src": "",
                                                                                    "Gw": "10.50.11.1",
                                                                                    "Protocol": 0,
                                                                                    "DevName": "",
                                                                                    "Scope": 0
                                                                            }
                                                                    ],
                                                                    "VlanID": 0,
                                                                    "EnableSnatOnHost": false,
                                                                    "EnableInfraVnet": false,
                                                                    "EnableMultitenancy": false,
                                                                    "AllowInboundFromHostToNC": false,
                                                                    "AllowInboundFromNCToHost": false,
                                                                    "NetworkNameSpace": "/proc/15563/ns/net",
                                                                    "ContainerID": "ebb957c79c5e254e98f5f68e6fac095f0c49cc54cf5c9c51b483088a59b11627",
                                                                    "PODName": "perf-cli",
                                                                    "PODNameSpace": "default"
                                                            }
                                                    },
                                                    "DNS": {
                                                            "Suffix": "",
                                                            "Servers": [
                                                                    "168.63.129.16"
                                                            ]
                                                    },
                                                    "EnableSnatOnHost": false,
                                                    "SnatBridgeIP": ""
                                            }
                                    },
                                    "Subnets": [
                                            "10.50.11.0/24"
                                    ],
                                    "BridgeName": "azure0",
                                    "DNSInfo": {
                                            "Suffix": "",
                                            "Servers": null
                                    },
                                    "MacAddress": "AA06B8LP",
                                    "IPAddresses": [
                                            {
                                                    "IP": "10.50.11.14",
                                                    "Mask": "////AA=="
                                            }
                                    ],
                                    "Routes": [
                                            {
                                                    "Family": 2,
                                                    "Dst": null,
                                                    "Src": "",
                                                    "Gw": "10.50.11.1",
                                                    "Tos": 0,
                                                    "Table": 254,
                                                    "Protocol": 3,
                                                    "Scope": 0,
                                                    "Type": 1,
                                                    "Flags": 0,
                                                    "Priority": 0,
                                                    "LinkIndex": 5,
                                                    "ILinkIndex": 0
                                            }
                                    ],
                                    "IPv4Gateway": "10.50.11.1",
                                    "IPv6Gateway": "::"
                            }
                    }
            }
    }

azure-vnet-ipam.json

    {
            "IPAM": {
                    "Version": "v1.0.26",
                    "TimeStamp": "2019-09-03T18:11:41.89895961+05:30",
                    "AddressSpaces": {
                            "local": {
                                    "Id": "local",
                                    "Scope": 0,
                                    "Pools": {
                                            "10.50.11.0/24": {
                                                    "Id": "10.50.11.0/24",
                                                    "IfName": "eth0",
                                                    "Subnet": {
                                                            "IP": "10.50.11.0",
                                                            "Mask": "////AA=="
                                                    },
                                                    "Gateway": "10.50.11.1",
                                                    "Addresses": {
                                                            "10.50.11.17": {
                                                                    "ID": "",
                                                                    "Addr": "10.50.11.17",
                                                                    "InUse": false
                                                            },
                                                            "10.50.11.18": {
                                                                    "ID": "",
                                                                    "Addr": "10.50.11.18",
                                                                    "InUse": true
                                                            },
                                                            "10.50.11.23": {
                                                                    "ID": "",
                                                                    "Addr": "10.50.11.23",
                                                                    "InUse": false
                                                            },
                                                            "10.50.11.24": {
                                                                    "ID": "",
                                                                    "Addr": "10.50.11.24",
                                                                    "InUse": false
                                                            },
                                                            "10.50.11.25": {
                                                                    "ID": "",
                                                                    "Addr": "10.50.11.25",
                                                                    "InUse": false
                                                            }
                                                    },
                                                    "IsIPv6": false,
                                                    "Priority": 0,
                                                    "RefCount": 1
                                            }
                                    }
                            }
                    }
            }
    }

I have these VMs opened in the different terminals. Thats why this mistake.

[tamilmani1989]

ok no worries. Can you ping from 10.50.11.19 to 10.50.11.20? just checking if pod to pod on same VM is working or not.

[chowmean]

No its not able to connect.

[tamilmani1989]

Can you try following iptable cmd and check if connectivity works?
sudo iptables -t filter -A FORWARD -j ACCEPT.

If it doesnt can i get ssh access to one of the VMs?

[chowmean]

I have to run this command on VM or containers? I ran on VM got no output.
Also for the access to VM I doubt it is possible. I have to check with the authorities once.
Do you have any documentation on step by step installation and FAQs? I can try it once again in the mean time if you have any docs with you.

[tamilmani1989]

yes. have to run in VM. Can you send me output of -> sudo iptables -t filter -L FORWARD? Isnt it work after adding that iptable cmd?

[chowmean]

No it didn't work. Below is the output.

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
ACCEPT     all  --  anywhere             anywhere```

[tamilmani1989]

ok thanks. Can you do following:

1. Run tcpdump command -> tcpdump -ni azure0 -e icmp
2. start ping from pod to VM IP.
3. after 5 to 10 ping requests, stop ping and send me output of tcpdump command