Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: net: fetch all interfaces with associated addresses in a single call #53660

Open
mandarjog opened this issue Jul 2, 2022 · 10 comments

Comments

@mandarjog
Copy link

mandarjog commented Jul 2, 2022

Background

At present if we want to get all interface and associated Addresses we write the following

ifaceByName := make(map[string]net.Interface)
addrByName := make(map[string][]net.IP)

interfaces, _ := net.Interfaces()

for i := 0; i < len(interfaces); i++ {
    intf := interfaces[i]
    addrByName[intf.Name] = intf.Addrs()
}

I would like a function that returns this entire information.

Why is it needed ?

intf.Addrs() call is inefficient because it results in getting the entire RIB syscall.NetlinkRIB(syscall.RTM_GETADDR, syscall.AF_UNSPEC) every time. The size of the RIB is proportional to the number of interfaces/Addrs. If on a router you have 1000s of interfaces this O(n^2) behaviour becomes untenable.
When number of interfaces is small this problem does not show up.

details

The flamegraphs show excessive time spent making syscall and repeatedly parsing messages.
I also ran call count using bpftrace

/usr/local/bin/bpftrace -p ${prog-pid}\
    -e 'uprobe:/usr/local/bin/prog:*vmsysmon.*,uprobe:/usr/local/bin/prog:net.int*,uprobe:/usr/local/bin/prog:syscall.* { @[func] = count(); }'

@[prog.(*IfMon).calcStats]: 5922
@[net.interfaceAddrTable]: 6169
@[syscall.Close]: 6318
@[syscall.RawSyscall.abi0]: 10287
@[syscall.Syscall.abi0]: 10583
@[syscall.(*SockaddrNetlink).sockaddr]: 12320
@[syscall.ParseNetlinkRouteAttr]: 16582
@[syscall.recvfrom]: 398880
@[syscall.anyToSockaddr]: 506789
@[syscall.Syscall6.abi0]: 568181
@[syscall.Recvfrom]: 622379
@[syscall.ParseNetlinkMessage]: 655167

calcStats function results in

After I made the change to fetch the entire RIB and associate addresses in a single loop it makes significant change in the call counts and times.

@[syscall.Syscall.abi0]: 2265
@[syscall.recvfrom]: 3330
@[syscall.Recvfrom]: 3334
@[syscall.ParseNetlinkMessage]: 3335
@[syscall.anyToSockaddr]: 3347
@[syscall.Syscall6.abi0]: 3482
@[aviatrix.com/vmsysmon.(*IfMon).calcStats]: 5912
@[syscall.ParseNetlinkRouteAttr]: 18326

Proposed API

Option 1

// InterfaceAddrsWithIndex returns a map from interface index to a list of
// associated unicast addresses.
func InterfaceAddrsWithIndex()(map[int][]Addr, error)  {
	return nil, nil
}

Keeps the flavor of existing InterfaceAddrs API.
Typical use.

func testListInterfaceDetails() {
	interfaces, _ := Interfaces()
	addrsMap, _ := InterfaceAddrsWithIndex()

	for i := 0; i < len(interfaces); i++ {
		intf := interfaces[i]
		if addrs, found := addrsMap[intf.Index]; found {
			fmt.Printf("%v: %v\n", intf, addrs)
		}
	}
}

Option 2

Treat all the information above as a single DB and return it. More optimizations are possible with this API without changing the API signature.

// IntfRecord keeps interface and associated Addresses.
type IntfRecord struct {
	Intf  Interface
	Addrs []Addr
}
type IntfDB map[int]IntfRecord

// InterfaceDB returns a map from interface index to an IntfRecord.
// This is the most efficient way feth the entire Interface database.
func InterfaceDB() (IntfDB, error) {
	return nil, nil
}
@gopherbot gopherbot added this to the Proposal milestone Jul 2, 2022
@ianlancetaylor ianlancetaylor changed the title proposal: net/interface.go fetch all interfaces with associated addresses in a single call proposal: net: fetch all interfaces with associated addresses in a single call Jul 3, 2022
@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Jul 3, 2022

What specific new API do you propose to address this? Thanks.

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Jul 3, 2022
@mandarjog
Copy link
Author

mandarjog commented Jul 3, 2022

@ianlancetaylor I have updated the issue with the API proposal.

@mandarjog
Copy link
Author

mandarjog commented Jul 3, 2022

@ianlancetaylor can you point me to benchmark tests that test with a number of interfaces? I found benchmark tests in interface_tests.go however they operate with a small number of interfaces (perhaps just loopback?). Is there test infra to create a large number of interfaces, managing network namespaces to do that and cleanup ?

@seankhliao
Copy link
Member

seankhliao commented Jul 3, 2022

Alternate API: the net.Addrs returned by net.InterfaceAddrs() could expose an additional method for getting the associated interface

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Jul 3, 2022

Is there test infra to create a large number of interfaces, managing network namespaces to do that and cleanup ?

Not to my knowledge.

@seankhliao
Copy link
Member

seankhliao commented Jul 3, 2022

#51934 is a report for efficiency / race / ignoring failures from k8s

@mandarjog
Copy link
Author

mandarjog commented Jul 3, 2022

@seankhliao as you can see in the usage example I have specified, devices and interfaces are snapshots at slightly different times. This means that we should expect to see device indexes returned by 2nd call to be unknown to the interfaces() call. The only way to fix it is to get a full snapshot from the kernel, which i don’t think is available.

@mandarjog
Copy link
Author

mandarjog commented Jul 4, 2022

Alternate API: the net.Addrs returned by net.InterfaceAddrs() could expose an additional method for getting the associated interface

interesting idea however,
@seankhliao addresses are a property of an interface not the other way around. So from a modeling perspective it does not make sense.

@mandarjog
Copy link
Author

mandarjog commented Jul 11, 2022

@ianlancetaylor any comments? Does a PR need to be implemented for all platforms or can start with Linux only ?

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Jul 13, 2022

This will go to review by the proposal review committee as described at https://go.dev/s/proposal.

If we adopt this change, it needs to do something sensible on all platforms.

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Incoming
Development

No branches or pull requests

4 participants