New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong external interface detection if bridge already configured #80

Closed
r7vme opened this Issue Nov 15, 2017 · 5 comments

Comments

Projects
None yet
2 participants
@r7vme
Copy link

r7vme commented Nov 15, 2017

I have kubernetes cluster with azure vnet cni enabled and after kubelet restart i see the following errors for new pods.

CNI improperly detects external interface azure0, but should be eth0

azure-vnet.log

  1 2017/11/15 10:52:20 [net] Setting link azure0 state down.
  2 2017/11/15 10:52:20 [net] Setting link azure0 master azure0.
  3 2017/11/15 10:52:20 [netlink] Received {NlMsghdr:{Len:60 Type:2 Flags:0 Seq:4 Pid:67602} data:[216 255 255 255 40 0 0 0 19 0 5 0 4 0 0 0 18 8 1 0 0 0 19 0 4 0 0 0 1 0 0 0 255 255 255 255 8 0     10 0
  4  4 0 0 0] payload:[]}, err=too many levels of symbolic links
  5 2017/11/15 10:52:20 [net] Connecting interface azure0 completed with err:too many levels of symbolic links.
  6 2017/11/15 10:52:20 [net] Failed to create network azure, err:too many levels of symbolic links.
  7 2017/11/15 10:52:20 [azure-vnet] Failed to create network: too many levels of symbolic links.

ip addr output

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc mq master azure0 state UP group default qlen 1000
    link/ether 00:0d:3a:29:c9:1a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20d:3aff:fe29:c91a/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:1c:a4:f8:57 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
4: azure0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:0d:3a:29:c9:1a brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.7/24 brd 10.0.2.255 scope global azure0
       valid_lft forever preferred_lft forever
    inet6 fe80::20d:3aff:fe29:c91a/64 scope link tentative 
       valid_lft forever preferred_lft forever
5: azveth460f010@if6: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master azure0 state UP group default qlen 1000
    link/ether ce:86:6d:99:03:99 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::cc86:6dff:fe99:399/64 scope link 
       valid_lft forever preferred_lft forever
9: azveth11a7c76@if10: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master azure0 state UP group default qlen 1000
    link/ether 8a:ed:ca:70:85:83 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::88ed:caff:fe70:8583/64 scope link 
       valid_lft forever preferred_lft forever

@r7vme r7vme changed the title Wrong external interface detection Wrong external interface detection if bridge already configured Nov 15, 2017

@sharmasushant

This comment has been minimized.

Copy link
Collaborator

sharmasushant commented Nov 17, 2017

@r7vme we can take a look. Can you please share your cluster config?

@r7vme

This comment has been minimized.

Copy link
Author

r7vme commented Nov 20, 2017

hello, i'm not using acs-engine. I was evaluating Azure vnet CNI for using in our (@giantswarm) installation on Azure. Unfortunately i've already dropped a cluster, but here are the steps that i did.

CoreOS based k8s 1.7.5 with one dhcp NIC (eth0) with private address. Plus for testing i assigned 2 more IP configurations for the VM in Azure portal.

  • k8s-kubelet i had a flag --network-plugin=cni.
  • i've downloaded latest CNI plugin from here and put binaries to /opt/cni/bin.

config was default from archive.

cat /etc/cni/net.d/
{
  "cniVersion": "0.2.0",
  "name": "azure",
  "type": "azure-vnet",
  "mode": "bridge",
  "bridge": "azure0",
  "ipam": {
    "type": "azure-vnet-ipam"
  }
}

Hope this can help.

@sharmasushant

This comment has been minimized.

Copy link
Collaborator

sharmasushant commented Nov 21, 2017

@r7vme Sure, Thanks for the information. Let us try to repro. Will get back to you.

@sharmasushant

This comment has been minimized.

Copy link
Collaborator

sharmasushant commented Nov 28, 2017

@r7vme Is it a consistent repro for you?
If yes, can you please share the logs from the following locations: /var/log/azure-vnet.log and /var/log/azure-vnet-ipam.log

Based on above, it seems that somehow the cni plugin lost the state and does not remember that the bridge was already created.. Did kubelet crashed or some other error happened in there?

@r7vme

This comment has been minimized.

Copy link
Author

r7vme commented Dec 6, 2017

Sorry, cluster with Azure vnet already wiped. May be important that we run k8s-kubelet in container.

@sharmasushant sharmasushant self-assigned this Jan 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment