Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

findalldevstest fails on Intel 82599 with SR-IOV enabled #389

Closed
leonexis opened this issue Oct 21, 2014 · 9 comments
Closed

findalldevstest fails on Intel 82599 with SR-IOV enabled #389

leonexis opened this issue Oct 21, 2014 · 9 comments
Labels
runtime

Comments

@leonexis
Copy link

I first found this issue with software that depends on libpcap (tcpreplay), but we found that the same error occurs using findalldevstest in libpcap-1.6.2 and they asked that I file a bug report here. SR-IOV is enabled, but no VMs are running. The following is the error that occurred after a full build from source (which was successful):

root@oa4-hv-9210:~/tmp/libpcap-1.6.2# make findalldevstest
gcc -fpic -I.  -DHAVE_CONFIG_H  -D_U_="__attribute__((unused))" -g -O2 -I. -L. -o findalldevstest ./tests/findalldevstest.c libpcap.a 
root@oa4-hv-9210:~/tmp/libpcap-1.6.2# ./findalldevstest 
Error in pcap_findalldevs: getifaddrs: Success

With SR-IOV disabled (by not specifying max_vfs to the ixgbe driver), the following occurs. The only complaint is about an interface without an address, which is true:

root@oa4-hv-9210:~/tmp/libpcap-1.6.2# ./findalldevstest 
eth0
        Loopback: no
        Address Family: Unknown (17)
        Address Family: AF_INET6
                Address: fe80::ea8d:f5ff:fe10:20
                Netmask: ffff:ffff:ffff:ffff::

eth2
        Loopback: no
        Address Family: Unknown (17)
        Address Family: AF_INET6
                Address: fe80::ea8d:f5ff:fe10:22
                Netmask: ffff:ffff:ffff:ffff::

eth10
        Loopback: no
        Address Family: Unknown (17)
        Address Family: AF_INET6
                Address: fe80::ea8d:f5ff:fe10:2c
                Netmask: ffff:ffff:ffff:ffff::

br10
        Loopback: no
        Address Family: Unknown (17)
        Address Family: AF_INET
                Address: 10.10.15.126
                Netmask: 255.255.240.0
                Broadcast Address: 10.10.15.255
        Address Family: AF_INET6
                Address: fe80::ea8d:f5ff:fe10:2c
                Netmask: ffff:ffff:ffff:ffff::

eth11
        Loopback: no
        Address Family: Unknown (17)
        Address Family: AF_INET6
                Address: fe80::ea8d:f5ff:fe10:2d
                Netmask: ffff:ffff:ffff:ffff::
br11
        Loopback: no
        Address Family: Unknown (17)
        Address Family: AF_INET
                Address: 192.168.0.32
                Netmask: 255.255.0.0
                Broadcast Address: 192.168.255.255
        Address Family: AF_INET6
                Address: fe80::ea8d:f5ff:fe10:2d
                Netmask: ffff:ffff:ffff:ffff::

br11.250
        Loopback: no
        Address Family: Unknown (17)
        Address Family: AF_INET
                Address: 10.250.0.32
                Netmask: 255.255.0.0
                Broadcast Address: 10.250.255.255
        Address Family: AF_INET6
                Address: fe80::ea8d:f5ff:fe10:2d
                Netmask: ffff:ffff:ffff:ffff::

any
        Description: Pseudo-device that captures on all interfaces
        Loopback: no

lo
        Loopback: yes
        Address Family: Unknown (17)
        Address Family: AF_INET
                Address: 127.0.0.1
                Netmask: 255.0.0.0
        Address Family: AF_INET6
                Address: ::1
                Netmask: ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff

virbr0
        Loopback: no
        Address Family: Unknown (17)
        Address Family: AF_INET
                Address: 192.168.122.1
                Netmask: 255.255.255.0
                Broadcast Address: 192.168.122.255

eth1
        Loopback: no
        Address Family: Unknown (17)

eth3
        Loopback: no
        Address Family: Unknown (17)

eth4
        Loopback: no
        Address Family: Unknown (17)

br4
        Loopback: no
        Address Family: Unknown (17)

eth5
        Loopback: no
        Address Family: Unknown (17)

br5
        Loopback: no
        Address Family: Unknown (17)

eth6
        Loopback: no
        Address Family: Unknown (17)

br6
        Loopback: no
        Address Family: Unknown (17)

eth7
        Loopback: no
        Address Family: Unknown (17)

br7
        Loopback: no
        Address Family: Unknown (17)

eth8
        Loopback: no
        Address Family: Unknown (17)

br8
        Loopback: no
        Address Family: Unknown (17)

eth9
        Loopback: no
        Address Family: Unknown (17)

br9
        Loopback: no
        Address Family: Unknown (17)

nflog
        Description: Linux netfilter log (NFLOG) interface
        Loopback: no

nfqueue
        Description: Linux netfilter queue (NFQUEUE) interface
        Loopback: no

Preferred device name: eth0
Error in pcap_lookupnet: eth0: no IPv4 address assigned

The first device tested is a ZNYX ZX9210 ATCA compute module with 2x Intel Niantic 82599 10GbE chips, for a total of 4x10GbE devices (eth0..3). By default, SR-IOV is enabled so virtual machines can use the much faster SR-IOV Virtual Functions rather than relying on forwarding traffic through the CPU to the VMs. The second device is the ZNYX B1 Compute Resource (beta unit) with 4x Intel Niantic 82599 10GbE chips for a total of 8x10GbE devices (eth2..9). The results are the same: with SR-IOV enabled, libpcap's findalldevstest fails with "Error in pcap_findalldevs: getifaddrs: Success"

The following might help (output from the ZX9210):

root@oa4-hv-9210:~# lsmod
Module                  Size  Used by
mlx4_en                85334  0 
mlx4_core             237416  1 mlx4_en
compat                 16773  2 mlx4_en,mlx4_core
ip6table_filter        12815  0 
ip6_tables             27342  1 ip6table_filter
ebtable_nat            12807  0 
ebtables               30571  1 ebtable_nat
ipt_MASQUERADE         12759  3 
iptable_nat            13182  1 
nf_nat                 25413  2 ipt_MASQUERADE,iptable_nat
nf_conntrack_ipv4      19545  4 iptable_nat,nf_nat
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
xt_state               12578  1 
nf_conntrack           81244  5 ipt_MASQUERADE,iptable_nat,nf_nat,nf_conntrack_ipv4,xt_state
ipt_REJECT             12541  2 
xt_CHECKSUM            12549  1 
iptable_mangle         12695  1 
xt_tcpudp              12603  5 
iptable_filter         12810  1 
ip_tables              26952  3 iptable_nat,iptable_mangle,iptable_filter
x_tables               29712  12 ip6table_filter,ip6_tables,ebtables,ipt_MASQUERADE,iptable_nat,xt_state,ipt_REJECT,xt_CHECKSUM,iptable_mangle,xt_tcpudp,iptable_filter,ip_tables
kvm_intel             132427  0 
kvm                   407127  1 kvm_intel
bridge                 90106  0 
igb                   177616  0 
vhost_net              27696  0 
macvtap                18252  1 vhost_net
macvlan                18840  1 macvtap
uvesafb                28803  1 
sb_edac                17794  0 
edac_core              53353  3 sb_edac
ixgbe                 221194  0 
ioatdma                43548  368 
dca                    15130  3 igb,ioatdma,ixgbe
ipmi_si                52888  1 
8021q                  23716  0 
garp                   14430  1 8021q
stp                    12931  2 bridge,garp
llc                    14597  3 bridge,garp,stp

root@oa4-hv-9210:~# cat /etc/modprobe.d/ixgbe.conf 
blacklist ixgbevf
#install ixgbe /sbin/modprobe igb ; /sbin/modprobe -i ixgbe max_vfs=32,32,32,32; true
install igb /sbin/modprobe ixgbe ; sleep 2 ; modprobe -i igb ; true
options ixgbe max_vfs=32,32,32,32

root@oa4-hv-9210:~# ethtool -i eth0
driver: ixgbe
version: 3.18.7
firmware-version: 0x800003e7
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes

root@oa4-hv-9210:~# uname -a
Linux oa4-hv-9210 3.2.16-4.1.3 #1 SMP Thu Dec 12 14:20:29 PST 2013 x86_64 x86_64 x86_64 GNU/Linux
root@oa4-hv-9210:~# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="ZNYX OpenArchitect 4 (Ubuntu 12.04 LTS)"
DISTRIB_OA_VERSION=4.1.3

Please let me know if there is any additional information I can provide or patches that need testing. If someone would like to recreate the issue, they will need an Intel 82599 NIC with the max_vfs option provided to the ixgbe driver. I have not tested other devices with SR-IOV.
@guyharris
Copy link
Member

Well, this is ultimately a getifaddrs() bug; the man page says:

On success, getifaddrs() returns zero; on error, -1 is returned, and
errno is set appropriately.

Apparently, it's returning a non-zero value and not setting errno appropriately. Either it's returning something other than zero or -1, or it's returning -1 and not setting errno appropriately.

getifaddrs() is a nice wrapper to hide differences between different OSes and different OS/kernel versions in the mechanisms they provide for getting interface lists with addresses, but perhaps, at least for newer Linux kernels, we should bypass it and directly use netlink or do something such as that.

It might be interesting to run findalldevstest with strace, to see what system calls it's making and which, if any, are failing.

@leonexis
Copy link
Author

Ah, that is interesting. I wonder what exactly it is getting caught on.

As requested:

root@oa4-hv-9210:~/tmp/libpcap-1.6.2# strace ./findalldevstest 
execve("./findalldevstest", ["./findalldevstest"], [/* 18 vars */]) = 0
brk(0)                                  = 0x14c5000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd4c0b46000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=23276, ...}) = 0
mmap(NULL, 23276, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fd4c0b40000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\30\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1815224, ...}) = 0
mmap(NULL, 3929304, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fd4c0566000
mprotect(0x7fd4c071b000, 2097152, PROT_NONE) = 0
mmap(0x7fd4c091b000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b5000) = 0x7fd4c091b000
mmap(0x7fd4c0921000, 17624, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fd4c0921000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd4c0b3f000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd4c0b3e000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd4c0b3d000
arch_prctl(ARCH_SET_FS, 0x7fd4c0b3e700) = 0
mprotect(0x7fd4c091b000, 16384, PROT_READ) = 0
mprotect(0x637000, 8192, PROT_READ)     = 0
mprotect(0x7fd4c0b48000, 4096, PROT_READ) = 0
munmap(0x7fd4c0b40000, 23276)           = 0
socket(PF_NETLINK, SOCK_RAW, 0)         = 3
bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(3, {sa_family=AF_NETLINK, pid=7842, groups=00000000}, [12]) = 0
sendto(3, "\24\0\0\0\22\0\1\3\300\310FT\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\344\3\0\0\20\0\2\0\300\310FT\242\36\0\0\0\0\4\3\1\0\0\0I\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=MSG_TRUNC}, 0) = 4096
close(3)                                = 0
write(2, "Error in pcap_findalldevs: getif"..., 47Error in pcap_findalldevs: getifaddrs: Success
) = 47
exit_group(1)

Full strings:

execve("./findalldevstest", ["./findalldevstest"], [/* 18 vars */]) = 0
brk(0)                                  = 0x20c8000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5b02c81000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=23276, ...}) = 0
mmap(NULL, 23276, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5b02c7b000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\30\2\0\0\0\0\0@\0\0\0\0\0\0\0\370\251\33\0\0\0\0\0\0\0\0\0@\0008\0\n\0@\0#\0\"\0\6\0\0\0\5\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0000\2\0\0\0\0\0\0000\2\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0\20K\30\0\0\0\0\0\20K\30\0\0\0\0\0\20K\30\0\0\0\0\0\34\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\34I\33\0\0\0\0\0\34I\33\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0\0W\33\0\0\0\0\0\0W;\0\0\0\0\0\0W;\0\0\0\0\0`Q\0\0\0\0\0\0\330\235\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0@\213\33\0\0\0\0\0@\213;\0\0\0\0\0@\213;\0\0\0\0\0\340\1\0\0\0\0\0\0\340\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0D\0\0\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\7\0\0\0\4\0\0\0\0W\33\0\0\0\0\0\0W;\0\0\0\0\0\0W;\0\0\0\0\0\20\0\0\0\0\0\0\0p\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0P\345td\4\0\0\0,K\30\0\0\0\0\0,K\30\0\0\0\0\0,K\30\0\0\0\0\0\254h\0\0\0\0\0\0\254h\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\0\0\0\0\0\0R\345td\4\0\0\0\0W\33\0\0\0\0\0\0W;\0\0\0\0\0\0W;\0\0\0\0\0\0009\0\0\0\0\0\0\0009\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\24\0\0\0\3\0\0\0GNU\0SQO|\232\326aG\3\254\1\375|\274\311\323\306\366!\315\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\6\0\0\0\30\0\0\0\0\0\0\0\363\3\0\0\t\0\0\0\0\1\0\0\16\0\0\0\0000\20D\240 \2\1\210\3\346\220\305E\214\0\300\0\10\0\5\200\0`\300\200\0\r\212\f\0\4\20\0\210D2\10.@\210P4, \16\"H&\204\300\214\4\10\0\2\2\16\241\254\32\4f\300\0\3002\0\300\0P\1 \201\10\204\v  ($\0\4 P\0\20X\200\312DB(\0\6\200\20\30B\0 @\200\0\tP\0Q\212@\20\0\0\0\0\10\0\0\21\20", 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1815224, ...}) = 0
mmap(NULL, 3929304, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f5b026a1000
mprotect(0x7f5b02856000, 2097152, PROT_NONE) = 0
mmap(0x7f5b02a56000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b5000) = 0x7f5b02a56000
mmap(0x7f5b02a5c000, 17624, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5b02a5c000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5b02c7a000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5b02c79000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5b02c78000
arch_prctl(ARCH_SET_FS, 0x7f5b02c79700) = 0
mprotect(0x7f5b02a56000, 16384, PROT_READ) = 0
mprotect(0x637000, 8192, PROT_READ)     = 0
mprotect(0x7f5b02c83000, 4096, PROT_READ) = 0
munmap(0x7f5b02c7b000, 23276)           = 0
socket(PF_NETLINK, SOCK_RAW, 0)         = 3
bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(3, {sa_family=AF_NETLINK, pid=7966, groups=00000000}, [12]) = 0
sendto(3, "\24\0\0\0\22\0\1\3\204\313FT\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\344\3\0\0\20\0\2\0\204\313FT\36\37\0\0\0\0\4\3\1\0\0\0I\0\1\0\0\0\0\0\7\0\3\0lo\0\0\10\0\r\0\0\0\0\0\5\0\20\0\0\0\0\0\5\0\21\0\0\0\0\0\10\0\4\0004@\0\0\10\0\33\0\0\0\0\0\f\0\6\0noqueue\0$\0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\n\0\1\0\0\0\0\0\0\0\0\0\n\0\2\0\0\0\0\0\0\0\0\0`\0\7\0\4\0\0\0\4\0\0\0\364\1\0\0\364\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\274\0\27\0\4\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\364\1\0\0\0\0\0\0\364\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0000\2\32\0l\0\2\0h\0\1\0\1\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\300\1\n\0\10\0\1\0\0\0\0\200\24\0\5\0\377\377\0\0\222\5\0\0\225j\0\0\350\3\0\0x\0\2\0\0\0\0\0@\0\0\0004@\0\0\1\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0\3\0\0\0\240\17\0\0\350\3\0\0\2\0\0\0\200:\t\0\200Q\1\0\3\0\0\0X\2\0\0\20\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\1\0\0\0`\352\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\377\377\377\377\0\0\0\0\374\0\3\0\37\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0,\0\6\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0p\16\0\0\20\0\2\0\204\313FT\36\37\0\0\0\0\1\0\2\0\0\0C\20\1\0\0\0\0\0\t\0\3\0eth0\0\0\0\0\10\0\r\0\350\3\0\0\5\0\20\0\6\0\0\0\5\0\21\0\0\0\0\0\10\0\4\0\334\5\0\0\10\0\33\0\0\0\0\0\7\0\6\0mq\0\0$\0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\n\0\1\0\350\215\365\20\0 \0\0\n\0\2\0\377\377\377\377\377\377\0\0`\0\7\0\0\0\0\0\6\0\0\0\0\0\0\0\324\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\274\0\27\0\0\0\0\0\0\0\0\0\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\324\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\25\0 \0\0\0\204\n\26\0T\0\1\0(\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\0\0\0\0\0\0\0\0\f\0\4\0\0\0\0\0\0\0\0\0T\0\1\0(\0\1\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\1\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\1\0\0\0\0\0\0\0\f\0\4\0\1\0\0\0\0\0\0\0T\0\1\0(\0\1\0\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\2\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\2\0\0\0\0\0\0\0\f\0\4\0\2\0\0\0\0\0\0\0T\0\1\0(\0\1\0\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\3\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\3\0\0\0\0\0\0\0\f\0\4\0\3\0\0\0\0\0\0\0T\0\1\0(\0\1\0\4\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\4\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\4\0\0\0\0\0\0\0\f\0\4\0\4\0\0\0\0\0\0\0T\0\1\0(\0\1\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\5\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\5\0\0\0\0\0\0\0\f\0\4\0\5\0\0\0\0\0\0\0T\0\1\0(\0\1\0\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\6\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\6\0\0\0\0\0\0\0\f\0\4\0\6\0\0\0\0\0\0\0T\0\1\0(\0\1\0\7\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\7\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\7\0\0\0\0\0\0\0\f\0\4\0\7\0\0\0\0\0\0\0T\0\1\0(\0\1\0\10\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\10\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\10\0\0\0\0\0\0\0\f\0\4\0\10\0\0\0\0\0\0\0T\0\1\0(\0\1\0\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\t\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\t\0\0\0\0\0\0\0\f\0\4\0\t\0\0\0\0\0\0\0T\0\1\0(\0\1\0\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\n\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\n\0\0\0\0\0\0\0\f\0\4\0\n\0\0\0\0\0\0\0T\0\1\0(\0\1\0\v\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\v\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\v\0\0\0\0\0\0\0\f\0\4\0\v\0\0\0\0\0\0\0T\0\1\0(\0\1\0\f\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\f\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\f\0\0\0\0\0\0\0\f\0\4\0\f\0\0\0\0\0\0\0T\0\1\0(\0\1\0\r\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\r\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\r\0\0\0\0\0\0\0\f\0\4\0\r\0\0\0\0\0\0\0T\0\1\0(\0\1\0\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\16\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\16\0\0\0\0\0\0\0\f\0\4\0\16\0\0\0\0\0\0\0T\0\1\0(\0\1\0\17\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\17\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\17\0\0\0\0\0\0\0\f\0\4\0\17\0\0\0\0\0\0\0T\0\1\0(\0\1\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\20\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\20\0\0\0\0\0\0\0\f\0\4\0\20\0\0\0\0\0\0\0T\0\1\0(\0\1\0\21\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\21\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\21\0\0\0\0\0\0\0\f\0\4\0\21\0\0\0\0\0\0\0T\0\1\0(\0\1\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\22\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\22\0\0\0\0\0\0\0\f\0\4\0\22\0\0\0\0\0\0\0T\0\1\0(\0\1\0\23\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\23\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\23\0\0\0\0\0\0\0\f\0\4\0\23\0\0\0\0\0\0\0T\0\1\0(\0\1\0\24\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\24\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\24\0\0\0\0\0\0\0\f\0\4\0\24\0\0\0\0\0\0\0T\0\1\0(\0\1\0\25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\25\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\25\0\0\0\0\0\0\0\f\0\4\0\25\0\0\0\0\0\0\0T\0\1\0(\0\1\0\26\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\26\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\26\0\0\0\0\0\0\0\f\0\4\0\26\0\0\0\0\0\0\0T\0\1\0(\0\1\0\27\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\27\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\27\0\0\0\0\0\0\0\f\0\4\0\27\0\0\0\0\0\0\0T\0\1\0(\0\1\0\30\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\30\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\30\0\0\0\0\0\0\0\f\0\4\0\30\0\0\0\0\0\0\0T\0\1\0(\0\1\0\31\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\31\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\31\0\0\0\0\0\0\0\f\0\4\0\31\0\0\0\0\0\0\0T\0\1\0(\0\1\0\32\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\32\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\32\0\0\0\0\0\0\0\f\0\4\0\32\0\0\0\0\0\0\0T\0\1\0(\0\1\0\33\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\33\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\33\0\0\0\0\0\0\0\f\0\4\0\33\0\0\0\0\0\0\0T\0\1\0(\0\1\0\34\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\34\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\34\0\0\0\0\0\0\0\f\0\4\0\34\0\0\0\0\0\0\0T\0\1\0(\0\1\0\35\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\35\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\35\0\0\0\0\0\0\0\f\0\4\0\35\0\0\0\0\0\0\0T\0\1\0(\0\1\0\36\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0\36\0\0\0\0\0\0\0\0\0\0\0\f\0\3\0\36\0\0\0\0\0\0\0\f\0\4\0\36\0\0\0\0\0\0\0T\0\1\0(\0\1\0\37\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\10Z\27\2\20\210\377\377H\326\24\201\377\377\377\377\310Y\27\2\20\0\2\0", 4096}], msg_controllen=0, msg_flags=MSG_TRUNC}, 0) = 4096
close(3)                                = 0
write(2, "Error in pcap_findalldevs: getifaddrs: Success\n", 47Error in pcap_findalldevs: getifaddrs: Success
) = 47
exit_group(1)                           = ?

@guyharris
Copy link
Member

OK, so there weren't any failures at the system call layer, so presumably glibc didn't like what it got back from the recvmsg() on the netlink socket.

If msg_flags=MSG_TRUNC means that MSG_TRUNC was set in msg.msg_flags, then that could be the issue. That appears to be one of the failure cases in glibc 2.15's getifaddrs() (2.15 is what the Wikipedia article on glibc claims is the version in Ubuntu 12.04), and it does not appear to set errno if it fails with MSG_TRUNC.

Glibc 2.15 appears to use a fixed one-page buffer for recvmsg() in that case, so if, for whatever reason, the kernel returns more than 4K, getifaddrs() will fail rather than, say, allocating a bigger buffer and trying again.

I don't (yet) know why the netlink socket wants to provide more than 4K bytes of data here.

@guyharris
Copy link
Member

What happens if you run ip addr show when SR-IOV is disabled and when it's enabled? Does it print the same results, or are the results different?

(If ip addr show, which I guess is the hot new thing on Linux, doesn't work, use ifconfig -a instead.)

@leonexis
Copy link
Author

glibc version is 2.15-0.ubuntu10.5. The only thing I can think of is that when SR-IOV is enabled, essentially each VF is its own NIC able to be passed through to a VM. That means that for a setting of 32VFs per device, that would come to 256 NICs on the ZX9210. However, the extra VFs will not be used by the hypervisor because the ixgbevf driver is blacklisted there, but available in the VMs when the device is passed through there.

As requested, SR-IOV disabled:

root@oa4-hv-9210:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether e8:8d:f5:10:00:20 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ea8d:f5ff:fe10:20/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:21 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether e8:8d:f5:10:00:22 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ea8d:f5ff:fe10:22/64 scope link 
       valid_lft forever preferred_lft forever
5: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:23 brd ff:ff:ff:ff:ff:ff
6: eth4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br4 state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:24 brd ff:ff:ff:ff:ff:ff
7: br4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether e8:8d:f5:10:00:24 brd ff:ff:ff:ff:ff:ff
8: eth5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br5 state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:25 brd ff:ff:ff:ff:ff:ff
9: eth6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br6 state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:26 brd ff:ff:ff:ff:ff:ff
10: br6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether e8:8d:f5:10:00:26 brd ff:ff:ff:ff:ff:ff
11: eth7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br7 state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:27 brd ff:ff:ff:ff:ff:ff
12: eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br10 state UP qlen 1000
    link/ether e8:8d:f5:10:00:2c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ea8d:f5ff:fe10:2c/64 scope link 
       valid_lft forever preferred_lft forever
13: br7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether e8:8d:f5:10:00:27 brd ff:ff:ff:ff:ff:ff
14: eth11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br11 state UP qlen 1000
    link/ether e8:8d:f5:10:00:2d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ea8d:f5ff:fe10:2d/64 scope link 
       valid_lft forever preferred_lft forever
15: br10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether e8:8d:f5:10:00:2c brd ff:ff:ff:ff:ff:ff
    inet 10.10.15.126/20 brd 10.10.15.255 scope global br10
    inet6 fe80::ea8d:f5ff:fe10:2c/64 scope link 
       valid_lft forever preferred_lft forever
16: br11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether e8:8d:f5:10:00:2d brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.32/16 brd 192.168.255.255 scope global br11
    inet6 fe80::ea8d:f5ff:fe10:2d/64 scope link 
       valid_lft forever preferred_lft forever
17: eth8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br8 state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:2a brd ff:ff:ff:ff:ff:ff
18: br8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether e8:8d:f5:10:00:2a brd ff:ff:ff:ff:ff:ff
19: eth9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br9 state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:2b brd ff:ff:ff:ff:ff:ff
20: br5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether e8:8d:f5:10:00:25 brd ff:ff:ff:ff:ff:ff
21: br9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether e8:8d:f5:10:00:2b brd ff:ff:ff:ff:ff:ff
22: br11.250@br11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether e8:8d:f5:10:00:2d brd ff:ff:ff:ff:ff:ff
    inet 10.250.0.32/16 brd 10.250.255.255 scope global br11.250
    inet6 fe80::ea8d:f5ff:fe10:2d/64 scope link 
       valid_lft forever preferred_lft forever
23: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether f6:e9:da:83:93:fa brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

SR-IOV enabled:

root@oa4-hv-9210:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether e8:8d:f5:10:00:20 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ea8d:f5ff:fe10:20/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:21 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether e8:8d:f5:10:00:22 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ea8d:f5ff:fe10:22/64 scope link 
       valid_lft forever preferred_lft forever
5: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:23 brd ff:ff:ff:ff:ff:ff
6: eth4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br4 state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:24 brd ff:ff:ff:ff:ff:ff
7: eth5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br5 state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:25 brd ff:ff:ff:ff:ff:ff
8: eth6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br6 state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:26 brd ff:ff:ff:ff:ff:ff
9: eth7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br7 state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:27 brd ff:ff:ff:ff:ff:ff
10: eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br10 state UP qlen 1000
    link/ether e8:8d:f5:10:00:2c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ea8d:f5ff:fe10:2c/64 scope link 
       valid_lft forever preferred_lft forever
11: eth11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br11 state UP qlen 1000
    link/ether e8:8d:f5:10:00:2d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ea8d:f5ff:fe10:2d/64 scope link 
       valid_lft forever preferred_lft forever
12: eth8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br8 state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:2a brd ff:ff:ff:ff:ff:ff
13: eth9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br9 state DOWN qlen 1000
    link/ether e8:8d:f5:10:00:2b brd ff:ff:ff:ff:ff:ff
14: br4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether e8:8d:f5:10:00:24 brd ff:ff:ff:ff:ff:ff
15: br6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether e8:8d:f5:10:00:26 brd ff:ff:ff:ff:ff:ff
16: br5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether e8:8d:f5:10:00:25 brd ff:ff:ff:ff:ff:ff
17: br11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether e8:8d:f5:10:00:2d brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.32/16 brd 192.168.255.255 scope global br11
    inet6 fe80::ea8d:f5ff:fe10:2d/64 scope link 
       valid_lft forever preferred_lft forever
18: br7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether e8:8d:f5:10:00:27 brd ff:ff:ff:ff:ff:ff
19: br9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether e8:8d:f5:10:00:2b brd ff:ff:ff:ff:ff:ff
20: br10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether e8:8d:f5:10:00:2c brd ff:ff:ff:ff:ff:ff
    inet 10.10.15.126/20 brd 10.10.15.255 scope global br10
    inet6 fe80::ea8d:f5ff:fe10:2c/64 scope link 
       valid_lft forever preferred_lft forever
21: br8: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether e8:8d:f5:10:00:2a brd ff:ff:ff:ff:ff:ff
22: br11.250@br11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether e8:8d:f5:10:00:2d brd ff:ff:ff:ff:ff:ff
    inet 10.250.0.32/16 brd 10.250.255.255 scope global br11.250
    inet6 fe80::ea8d:f5ff:fe10:2d/64 scope link 
       valid_lft forever preferred_lft forever
23: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether c2:ba:a9:d4:3b:86 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

Pretty much the same. However, this may help..

SR-IOV disabled:

root@oa4-hv-9210:~# ls -l /sys/module/ixgbe/drivers/pci\:ixgbe/0000\:01\:00.0/
total 0
-rw-r--r-- 1 root root   4096 Oct 21 21:58 broken_parity_status
-r--r--r-- 1 root root   4096 Oct 21 21:58 class
-rw-r--r-- 1 root root   4096 Oct 21 21:58 config
-r--r--r-- 1 root root   4096 Oct 21 21:58 consistent_dma_mask_bits
-r--r--r-- 1 root root   4096 Oct 21 21:58 device
-r--r--r-- 1 root root   4096 Oct 21 21:58 dma_mask_bits
lrwxrwxrwx 1 root root      0 Oct 21 21:58 driver -> ../../../../bus/pci/drivers/ixgbe
-rw------- 1 root root   4096 Oct 21 21:58 enable
-r--r--r-- 1 root root   4096 Oct 21 21:58 irq
-r--r--r-- 1 root root   4096 Oct 21 21:58 local_cpulist
-r--r--r-- 1 root root   4096 Oct 21 21:58 local_cpus
-r--r--r-- 1 root root   4096 Oct 21 21:58 modalias
-rw-r--r-- 1 root root   4096 Oct 21 21:58 msi_bus
drwxr-xr-x 3 root root      0 Oct 21 21:58 net
-r--r--r-- 1 root root   4096 Oct 21 21:58 numa_node
drwxr-xr-x 2 root root      0 Oct 21 21:58 power
--w--w---- 1 root root   4096 Oct 21 21:58 remove
--w--w---- 1 root root   4096 Oct 21 21:58 rescan
--w------- 1 root root   4096 Oct 21 21:58 reset
-r--r--r-- 1 root root   4096 Oct 21 21:58 resource
-rw------- 1 root root 131072 Oct 21 21:58 resource0
-rw------- 1 root root     32 Oct 21 21:58 resource2
-rw------- 1 root root  16384 Oct 21 21:58 resource4
lrwxrwxrwx 1 root root      0 Oct 21 21:58 subsystem -> ../../../../bus/pci
-r--r--r-- 1 root root   4096 Oct 21 21:58 subsystem_device
-r--r--r-- 1 root root   4096 Oct 21 21:58 subsystem_vendor
-rw-r--r-- 1 root root   4096 Oct 21 21:58 uevent
-r--r--r-- 1 root root   4096 Oct 21 21:58 vendor
-rw------- 1 root root  32768 Oct 21 21:58 vpd
root@oa4-hv-9210:~# ls /sys/devices/pci0000\:00/0000\:00\:01.0/
0000:00:01.0:pcie01  broken_parity_status  consistent_dma_mask_bits  driver         irq            modalias   pci_bus  rescan    subsystem         uevent
0000:01:00.0         class                 device                    enable         local_cpulist  msi_bus    power    reset     subsystem_device  vendor
0000:01:00.1         config                dma_mask_bits             firmware_node  local_cpus     numa_node  remove   resource  subsystem_vendor

SR-IOV enabled:

root@oa4-hv-9210:~# ls -l /sys/module/ixgbe/drivers/pci\:ixgbe/0000\:01\:00.0/
total 0
-rw-r--r-- 1 root root   4096 Oct 21 21:53 broken_parity_status
-r--r--r-- 1 root root   4096 Oct 21 21:53 class
-rw-r--r-- 1 root root   4096 Oct 21 21:49 config
-r--r--r-- 1 root root   4096 Oct 21 21:53 consistent_dma_mask_bits
-r--r--r-- 1 root root   4096 Oct 21 21:49 device
-r--r--r-- 1 root root   4096 Oct 21 21:53 dma_mask_bits
lrwxrwxrwx 1 root root      0 Oct 21 21:49 driver -> ../../../../bus/pci/drivers/ixgbe
-rw------- 1 root root   4096 Oct 21 21:53 enable
-r--r--r-- 1 root root   4096 Oct 21 21:53 irq
-r--r--r-- 1 root root   4096 Oct 21 21:53 local_cpulist
-r--r--r-- 1 root root   4096 Oct 21 21:53 local_cpus
-r--r--r-- 1 root root   4096 Oct 21 21:53 modalias
-rw-r--r-- 1 root root   4096 Oct 21 21:53 msi_bus
drwxr-xr-x 3 root root      0 Oct 21 21:49 net
-r--r--r-- 1 root root   4096 Oct 21 21:53 numa_node
drwxr-xr-x 2 root root      0 Oct 21 21:53 power
--w--w---- 1 root root   4096 Oct 21 21:53 remove
--w--w---- 1 root root   4096 Oct 21 21:53 rescan
--w------- 1 root root   4096 Oct 21 21:53 reset
-r--r--r-- 1 root root   4096 Oct 21 21:53 resource
-rw------- 1 root root 131072 Oct 21 21:53 resource0
-rw------- 1 root root     32 Oct 21 21:53 resource2
-rw------- 1 root root  16384 Oct 21 21:53 resource4
lrwxrwxrwx 1 root root      0 Oct 21 21:49 subsystem -> ../../../../bus/pci
-r--r--r-- 1 root root   4096 Oct 21 21:53 subsystem_device
-r--r--r-- 1 root root   4096 Oct 21 21:53 subsystem_vendor
-rw-r--r-- 1 root root   4096 Oct 21 21:49 uevent
-r--r--r-- 1 root root   4096 Oct 21 21:49 vendor
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn0 -> ../0000:01:10.0
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn1 -> ../0000:01:10.2
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn10 -> ../0000:01:12.4
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn11 -> ../0000:01:12.6
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn12 -> ../0000:01:13.0
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn13 -> ../0000:01:13.2
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn14 -> ../0000:01:13.4
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn15 -> ../0000:01:13.6
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn16 -> ../0000:01:14.0
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn17 -> ../0000:01:14.2
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn18 -> ../0000:01:14.4
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn19 -> ../0000:01:14.6
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn2 -> ../0000:01:10.4
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn20 -> ../0000:01:15.0
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn21 -> ../0000:01:15.2
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn22 -> ../0000:01:15.4
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn23 -> ../0000:01:15.6
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn24 -> ../0000:01:16.0
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn25 -> ../0000:01:16.2
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn26 -> ../0000:01:16.4
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn27 -> ../0000:01:16.6
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn28 -> ../0000:01:17.0
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn29 -> ../0000:01:17.2
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn3 -> ../0000:01:10.6
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn30 -> ../0000:01:17.4
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn31 -> ../0000:01:17.6
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn4 -> ../0000:01:11.0
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn5 -> ../0000:01:11.2
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn6 -> ../0000:01:11.4
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn7 -> ../0000:01:11.6
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn8 -> ../0000:01:12.0
lrwxrwxrwx 1 root root      0 Oct 21 21:49 virtfn9 -> ../0000:01:12.2
-rw------- 1 root root  32768 Oct 21 21:53 vpd
root@oa4-hv-9210:~# ls /sys/devices/pci0000\:00/0000\:00\:01.0/
0000:00:01.0:pcie01  0000:01:10.5  0000:01:11.5  0000:01:12.5  0000:01:13.5  0000:01:14.5  0000:01:15.5  0000:01:16.5  0000:01:17.5              dma_mask_bits  msi_bus    subsystem
0000:01:00.0         0000:01:10.6  0000:01:11.6  0000:01:12.6  0000:01:13.6  0000:01:14.6  0000:01:15.6  0000:01:16.6  0000:01:17.6              driver         numa_node  subsystem_device
0000:01:00.1         0000:01:10.7  0000:01:11.7  0000:01:12.7  0000:01:13.7  0000:01:14.7  0000:01:15.7  0000:01:16.7  0000:01:17.7              enable         pci_bus    subsystem_vendor
0000:01:10.0         0000:01:11.0  0000:01:12.0  0000:01:13.0  0000:01:14.0  0000:01:15.0  0000:01:16.0  0000:01:17.0  broken_parity_status      firmware_node  power      uevent
0000:01:10.1         0000:01:11.1  0000:01:12.1  0000:01:13.1  0000:01:14.1  0000:01:15.1  0000:01:16.1  0000:01:17.1  class                     irq            remove     vendor
0000:01:10.2         0000:01:11.2  0000:01:12.2  0000:01:13.2  0000:01:14.2  0000:01:15.2  0000:01:16.2  0000:01:17.2  config                    local_cpulist  rescan
0000:01:10.3         0000:01:11.3  0000:01:12.3  0000:01:13.3  0000:01:14.3  0000:01:15.3  0000:01:16.3  0000:01:17.3  consistent_dma_mask_bits  local_cpus     reset
0000:01:10.4         0000:01:11.4  0000:01:12.4  0000:01:13.4  0000:01:14.4  0000:01:15.4  0000:01:16.4  0000:01:17.4  device                    modalias       resource

@guyharris
Copy link
Member

Pretty much the same.

The only differences I see are the order of the interfaces and the MAC address of virbr0, so it doesn't appear that the list of interfaces and addresses differs significantly, unless there's something different about the way the netlink socket was providing that list.

@infrastation
Copy link
Member

Could you confirm if this bug still reproduces on the current master branch?

@infrastation
Copy link
Member

Thank you for providing detailed diagnostics, but without current information it is impossible to work on this issue. If it reproduces on modern glibc and the current master branch of libpcap, please tell it (it may still land at glibc though). Otherwise this issue will be closed in 10 days.

@infrastation
Copy link
Member

It is time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
runtime
Milestone
No milestone
Development

No branches or pull requests
3 participants
@infrastation @leonexis @guyharris