Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: InterfaceAddrs() "no such network interface" with rapidly changes interfaces #51934

Open
uablrek opened this issue Mar 25, 2022 · 2 comments
Open
Labels
help wanted NeedsInvestigation
Milestone

Comments

@uablrek
Copy link

@uablrek uablrek commented Mar 25, 2022

What version of Go are you using (go version)?

$ go version
go version go1.17.6 linux/amd64

The problem is in Kubernetes v1.23.3 which I think uses go1.17.6, but at least go1.17.x.

Does this issue reproduce with the latest release?

N/A. Have to wait for K8s.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
# (can't run this in K8s container)

System is openstack with;

Linux control-plane-n108-mast-n057 5.3.18-24.99-default #1 SMP Sun Jan 23 19:03:51 UTC 2022 (712a8e6) x86_64 x86_64 x86_64 GNU/Linux

What did you do?

In heavy stress tests on a very large K8s cluster (159 nodes) the update of addresses made by the internal k8s load-balancer (kube-proxy) sometimed (very infrequent) fails to read all addresses with net.InterfaceAddrs. The error is;

E0209 12:18:47.395530       1 proxier.go:1110] "Failed to list all node IPs from host" err="error listing LOCAL type addresses from host, error: Could not get addresses: route ip+net: no such network interface"

The stress test contantly reboot nodes and interfaces are created and removed all the time. At very infrequent times the error above i thrown.

Please see the K8s issue for more info, kubernetes/kubernetes#108065

What did you expect to see?

No interface related errors thrown by net.InterfaceAddrs, even if interfaces are created/removed all the time.

My interpretation is that the error is thrown if net.InterfaceAddrs firsts reads an interface name, then when to read addresses, the interface is gone.

A "no such network interface" must be handled (ignored) internally in net.InterfaceAddrs.

We as a users can of course add re-tries, but how many and for how long? Since interfaces are created/removed all the time we may never be able to read the addresses.

What did you see instead?

The error described above.

@uablrek
Copy link
Author

@uablrek uablrek commented Mar 25, 2022

A work-around may be to read interfaces with net.Interfaces and then read addresses per interface with net.Interface.Addrs and create our own "safe InterfaceAddrs". But IMO we shouldn't have to do that.

@seankhliao seankhliao changed the title affected/package: net. net.InterfaceAddrs() throws "no such network interface" net: InterfaceAddrs() "no such network interface" with rapidly changes interfaces Mar 25, 2022
@seankhliao seankhliao added the NeedsInvestigation label Mar 25, 2022
@seankhliao
Copy link
Contributor

@seankhliao seankhliao commented Mar 25, 2022

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted NeedsInvestigation
Projects
None yet
Development

No branches or pull requests

3 participants