Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BPF Filters not working #94

Closed
jimhranicky opened this issue May 9, 2016 · 33 comments
Closed

BPF Filters not working #94

jimhranicky opened this issue May 9, 2016 · 33 comments

Comments

@jimhranicky
Copy link

With normal tcpdump :

% tcpdump -i enp4s0 -nn -c 10 'port 22'
tcpdump: WARNING: enp4s0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp4s0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:50:00.338419 IP XX.XX.XX.XX.22 > XX.XX.XX.XX.60212: Flags [.], seq 2354062218:2354063678, ack 800994694, win 2380, length 1460
17:50:00.338438 IP XX.XX.XX.XX.22 > XX.XX.XX.XX.64833: Flags [P.], seq 652703376:652703456, ack 606406036, win 5657, length 80
17:50:00.338466 IP XX.XX.XX.XX.64833 > XX.XX.XX.XX.22: Flags [.], ack 0, win 255, length 0
17:50:00.338482 IP XX.XX.XX.XX.22 > XX.XX.XX.XX.60212: Flags [.], seq 1460:8760, ack 1, win 2380, length 7300
17:50:00.339772 IP XX.XX.XX.XX.60212 > XX.XX.XX.XX.22: Flags [P.], seq 1:69, ack 32872, win 10519, length 68
17:50:00.339786 IP XX.XX.XX.XX.22 > XX.XX.XX.XX.64833: Flags [P.], seq 480:560, ack 1, win 5657, length 80
17:50:00.339789 IP XX.XX.XX.XX.64833 > XX.XX.XX.XX.22: Flags [.], ack 480, win 253, length 0
17:50:00.339953 IP XX.XX.XX.XX.22 > XX.XX.XX.XX.64833: Flags [P.], seq 560:640, ack 1, win 5657, length 80
17:50:00.340376 IP XX.XX.XX.XX.22 > XX.XX.XX.XX.64833: Flags [P.], seq 640:720, ack 1, win 5657, length 80
17:50:00.340382 IP XX.XX.XX.XX.64833 > XX.XX.XX.XX.22: Flags [.], ack 640, win 252, length 0
10 packets captured
895 packets received by filter
795 packets dropped by kernel

With PF_RING's tcpdump :

% /opt/pf/sbin/tcpdump -i enp4s0 -nn -c 10 'port 22'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp4s0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:50:05.398683938 IP XX.XX.XX.XX.80 > XX.XX.XX.XX.53190: Flags [.], seq 3437247066:3437255826, ack 3263609792, win 513, length 8760
21:50:05.398703325 IP XX.XX.XX.XX.443 > XX.XX.XX.XX.56136: Flags [.], seq 1570714451:1570725683, ack 3907642189, win 273, length 11232
21:50:05.398712933 IP XX.XX.XX.XX.65125 > XX.XX.XX.XX.80: Flags [.], seq 2597100314:2597101774, ack 535663878, win 63855, length 1460
21:50:05.398721319 IP XX.XX.XX.XX.50271 > XX.XX.XX.XX.59307: Flags [.], seq 1379174102:1379181402, ack 3144835430, win 32768, length 7300
21:50:05.398728562 IP XX.XX.XX.XX.443 > XX.XX.XX.XX.57125: UDP, length 1453
21:50:05.398732652 IP XX.XX.XX.XX.443 > XX.XX.XX.XX.57125: UDP, length 1453
21:50:05.398736106 IP XX.XX.XX.XX.80 > XX.XX.XX.XX.36922: Flags [.], seq 308270661:308272109, ack 565323857, win 2050, options [nop,nop,TS val 2804111279 ecr 225559], length 1448
21:50:05.398739251 IP XX.XX.XX.XX.59307 > XX.XX.XX.XX.50271: Flags [.], ack 4294798264, win 12285, length 0
21:50:05.398740596 IP XX.XX.XX.XX.52813 > XX.XX.XX.XX.80: Flags [.], ack 3304701303, win 11946, options [nop,nop,TS val 1567099780 ecr 576135852,nop,nop,sack 1 {1449:60817}], length 0
21:50:05.398743104 IP XX.XX.XX.XX.52813 > XX.XX.XX.XX.80: Flags [.], ack 1, win 11946, options [nop,nop,TS val 1567099780 ecr 576135852,nop,nop,sack 1 {1449:62265}], length 0
10 packets captured
10 packets received by filter
0 packets dropped by kernel

RH Ver : 3.10.0-327.13.1.el7.x86_64
PF_RING Ver :

PF_RING Version          : 6.3.0 (dev:d568ce59908fd0021ec7910b0563db191301e61c)
Total rings              : 1

Standard (non DNA/ZC) Options
Ring slots               : 4096
Slot version             : 16
Capture TX               : Yes [RX+TX]
IP Defragment            : No
Socket Mode              : Standard
Total plugins            : 0
Cluster Fragment Queue   : 0
Cluster Fragment Discard : 0
@zachsis
Copy link

zachsis commented May 11, 2016 •
edited

I am seeing the same issue. I am able to BPF filter non-pfring interfaces, but more specifically I cannot filter on pfring/zc interfaces.

CentOS Linux release 7.2.1511 (Core) 

# cat /proc/net/pf_ring/info 
PF_RING Version          : 6.3.0 (dev:a09d139cb70ae2a82b765630f00189bc55ab3bc5)
Total rings              : 20

Standard (non DNA/ZC) Options
Ring slots               : 4096
Slot version             : 16
Capture TX               : Yes [RX+TX]
IP Defragment            : No
Socket Mode              : Standard
Total plugins            : 0
Cluster Fragment Queue   : 0
Cluster Fragment Discard : 0

strace output:


execve("/usr/local/sbin/tcpdump", ["tcpdump", "-i", "zc:99@17", "host X.X.X.X"], [/* 18 vars */]) = 0
brk(0)                                  = 0xb11000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba67799000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=31961, ...}) = 0
mmap(NULL, 31961, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fba67791000
close(3)                                = 0
open("/lib64/libcrypto.so.10", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\202\6\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2016960, ...}) = 0
mmap(NULL, 4095864, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fba67191000
mprotect(0x7fba6734f000, 2097152, PROT_NONE) = 0
mmap(0x7fba6754f000, 155648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1be000) = 0x7fba6754f000
mmap(0x7fba67575000, 16248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fba67575000
close(3)                                = 0
open("/lib64/librt.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\"\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=44096, ...}) = 0
mmap(NULL, 2128952, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fba66f89000
mprotect(0x7fba66f90000, 2093056, PROT_NONE) = 0
mmap(0x7fba6718f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7fba6718f000
close(3)                                = 0
open("/lib64/libntapi.so", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@<\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=653019, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba67790000
mmap(NULL, 36314320, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fba64ce7000
mprotect(0x7fba64d00000, 2093056, PROT_NONE) = 0
mmap(0x7fba64eff000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7fba64eff000
mmap(0x7fba64f00000, 34114768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fba64f00000
close(3)                                = 0
open("/lib64/libntos.so", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\341\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=337748, ...}) = 0
mmap(NULL, 2230456, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fba64ac6000
mprotect(0x7fba64adf000, 2097152, PROT_NONE) = 0
mmap(0x7fba64cdf000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19000) = 0x7fba64cdf000
mmap(0x7fba64ce6000, 2232, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fba64ce6000
close(3)                                = 0
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240l\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=142304, ...}) = 0
mmap(NULL, 2208864, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fba648aa000
mprotect(0x7fba648c0000, 2097152, PROT_NONE) = 0
mmap(0x7fba64ac0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7fba64ac0000
mmap(0x7fba64ac2000, 13408, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fba64ac2000
close(3)                                = 0
open("/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260T\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1141560, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba6778f000
mmap(NULL, 3150168, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fba645a8000
mprotect(0x7fba646a9000, 2093056, PROT_NONE) = 0
mmap(0x7fba648a8000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x100000) = 0x7fba648a8000
close(3)                                = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \34\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2107816, ...}) = 0
mmap(NULL, 3932736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fba641e7000
mprotect(0x7fba6439d000, 2097152, PROT_NONE) = 0
mmap(0x7fba6459d000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b6000) = 0x7fba6459d000
mmap(0x7fba645a3000, 16960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fba645a3000
close(3)                                = 0
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\16\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19520, ...}) = 0
mmap(NULL, 2109744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fba63fe3000
mprotect(0x7fba63fe6000, 2093056, PROT_NONE) = 0
mmap(0x7fba641e5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fba641e5000
close(3)                                = 0
open("/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p!\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=90632, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba6778e000
mmap(NULL, 2183688, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fba63dcd000
mprotect(0x7fba63de2000, 2093056, PROT_NONE) = 0
mmap(0x7fba63fe1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14000) = 0x7fba63fe1000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba6778d000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba6778b000
arch_prctl(ARCH_SET_FS, 0x7fba6778b740) = 0
mprotect(0x7fba6459d000, 16384, PROT_READ) = 0
mprotect(0x7fba63fe1000, 4096, PROT_READ) = 0
mprotect(0x7fba641e5000, 4096, PROT_READ) = 0
mprotect(0x7fba648a8000, 4096, PROT_READ) = 0
mprotect(0x7fba64ac0000, 4096, PROT_READ) = 0
mprotect(0x7fba6718f000, 4096, PROT_READ) = 0
mprotect(0x7fba6754f000, 106496, PROT_READ) = 0
mprotect(0x747000, 8192, PROT_READ)     = 0
mprotect(0x7fba6779a000, 4096, PROT_READ) = 0
munmap(0x7fba67791000, 31961)           = 0
set_tid_address(0x7fba6778ba10)         = 62612
set_robust_list(0x7fba6778ba20, 24)     = 0
rt_sigaction(SIGRTMIN, {0x7fba648b0780, [], SA_RESTORER|SA_SIGINFO, 0x7fba648b9100}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7fba648b0810, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fba648b9100}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
access("/etc/system-fips", F_OK)        = -1 ENOENT (No such file or directory)
brk(0)                                  = 0xb11000
brk(0xb32000)                           = 0xb32000
brk(0)                                  = 0xb32000
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2427, ...}) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=2427, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba67798000
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\0"..., 4096) = 2427
lseek(3, -1550, SEEK_CUR)               = 877
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\5\0\0\0\0"..., 4096) = 1550
close(3)                                = 0
munmap(0x7fba67798000, 4096)            = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2427, ...}) = 0
socket(PF_PACKET, SOCK_RAW, 768)        = 3
ioctl(3, SIOCGIWMODE, 0x7ffc5ea178c0)   = -1 ENODEV (No such device)
close(3)                                = 0
stat("/etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned", 0x7ffc5ea17500) = -1 ENOENT (No such file or directory)
open("/proc/net/pf_ring/dev/zc:99/info", O_RDONLY) = -1 ENOENT (No such file or directory)
access("/proc/net/pf_ring/dev/99/info", F_OK) = -1 ENOENT (No such file or directory)
socket(0x1b /* PF_??? */, SOCK_RAW, 768) = 3
socket(0x1b /* PF_??? */, SOCK_RAW, 768) = 4
setsockopt(4, SOL_IP, 0x89 /* IP_??? */, "c\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0c\0\0\0", 20) = 0
open("/proc/meminfo", O_RDONLY)         = 5
fstat(5, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba67798000
read(5, "MemTotal:       98796164 kB\nMemF"..., 1024) = 1024
read(5, "Total:     512\nHugePages_Free:  "..., 1024) = 202
close(5)                                = 0
munmap(0x7fba67798000, 4096)            = 0
open("/proc/mounts", O_RDONLY)          = 5
fstat(5, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba67798000
read(5, "rootfs / rootfs rw 0 0\nsysfs /sy"..., 1024) = 1024
read(5, " 0 0\ncgroup /sys/fs/cgroup/devic"..., 1024) = 1024
read(5, "w,seclabel,relatime,attr2,inode6"..., 1024) = 311
open("/mnt/hugepages/pfring_zc_99", O_RDWR|O_CREAT, 0755) = 6
flock(6, LOCK_SH)                       = 0
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_LOCKED, 6, 0) = 0x2aaaaac00000
munmap(0x2aaaaac00000, 2097152)         = 0
flock(6, LOCK_UN)                       = 0
flock(6, LOCK_EX|LOCK_NB)               = -1 EAGAIN (Resource temporarily unavailable)
close(6)                                = 0
open("/mnt/hugepages/pfring_zc_99", O_RDWR|O_CREAT, 0755) = 6
flock(6, LOCK_SH)                       = 0
mmap(NULL, 536870912, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_LOCKED, 6, 0) = 0x2aaaaac00000
mmap(NULL, 2895872, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba63b0a000
setsockopt(4, SOL_IP, 0x89 /* IP_??? */, "c\0\0\0\1\0\0\0\21\0\0\0\1\0\0\0c\0\0\0", 20) = 0
setsockopt(4, SOL_IP, 0x89 /* IP_??? */, "c\0\0\0\2\0\0\0\21\0\0\0\1\0\0\0c\0\0\0", 20) = 0
ioctl(3, SIOCGIFMTU, {ifr_name="99", ???}) = -1 ENODEV (No such device)
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba6774a000
eventfd2(0, 0)                          = 7
mmap(NULL, 8392704, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fba63309000
mprotect(0x7fba63309000, 4096, PROT_NONE) = 0
clone(child_stack=0x7fba63b08fb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7fba63b099d0, tls=0x7fba63b09700, child_tidptr=0x7fba63b099d0) = 62621
getgid()                                = 0
tgkill(62612, 62621, SIGRT_1)           = 0
futex(0x7ffc5ea178e0, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN (Resource temporarily unavailable)
setgid(0)                               = 0
getuid()                                = 0
tgkill(62612, 62621, SIGRT_1)           = 0
setuid(0)                               = 0
open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 8
fstat(8, {st_mode=S_IFREG|0644, st_size=1721, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba67797000
read(8, "#\n# /etc/nsswitch.conf\n#\n# An ex"..., 4096) = 1721
read(8, "", 4096)                       = 0
close(8)                                = 0
munmap(0x7fba67797000, 4096)            = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 8
fstat(8, {st_mode=S_IFREG|0644, st_size=31961, ...}) = 0
mmap(NULL, 31961, PROT_READ, MAP_PRIVATE, 8, 0) = 0x7fba67742000
close(8)                                = 0
open("/lib64/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 8
read(8, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\"\0\0\0\0\0\0"..., 832) = 832
fstat(8, {st_mode=S_IFREG|0755, st_size=61928, ...}) = 0
mmap(NULL, 2173048, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 8, 0) = 0x7fba630f6000
mprotect(0x7fba63102000, 2093056, PROT_NONE) = 0
mmap(0x7fba63301000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 8, 0xb000) = 0x7fba63301000
mmap(0x7fba63303000, 22648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fba63303000
close(8)                                = 0
mprotect(0x7fba63301000, 4096, PROT_READ) = 0
munmap(0x7fba67742000, 31961)           = 0
open("/etc/ethers", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/etc/services", O_RDONLY|O_CLOEXEC) = 8
lseek(8, 0, SEEK_CUR)                   = 0
fstat(8, {st_mode=S_IFREG|0644, st_size=670293, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fba67797000
lseek(8, 0, SEEK_SET)                   = 0
read(8, "# /etc/services:\n# $Id: services"..., 4096) = 4096
read(8, " 74/udp                         "..., 4096) = 4096
read(8, "     # Quick Mail Transfer Proto"..., 4096) = 4096
read(8, "    636/udp                     "..., 4096) = 4096
read(8, "          # Microsoft-SQL-Monito"..., 4096) = 4096
read(8, "                       # discp s"..., 4096) = 4096
read(8, "l Signal Alternate\n\nbprd        "..., 4096) = 4096
read(8, "         3690/udp               "..., 4096) = 4096
read(8, "          # Message Processing M"..., 4096) = 4096
read(8, "o-Point Trans Net\nacr-nema      "..., 4096) = 4096
read(8, "              # SGMP\nsgmp       "..., 4096) = 4096
read(8, "                # Directory Loca"..., 4096) = 4096
read(8, "ervice      267/udp             "..., 4096) = 4096
read(8, "    bh611\nbh611           354/ud"..., 4096) = 4096
read(8, "              # EMBL Nucleic Dat"..., 4096) = 4096
read(8, "    430/tcp                 # UT"..., 4096) = 4096
brk(0)                                  = 0xb32000
brk(0xb53000)                           = 0xb53000
read(8, "login\nljk-login       472/udp   "..., 4096) = 4096
read(8, "#\nulp             522/tcp       "..., 4096) = 4096
read(8, "574/tcp                 # FTP So"..., 4096) = 4096
read(8, "uration Manager\nsco-sysmgr      "..., 4096) = 4096
read(8, " #       AODV\naodv            65"..., 4096) = 4096
read(8, "  692/tcp                 # Hype"..., 4096) = 4096
read(8, "   762/tcp                 #\nquo"..., 4096) = 4096
read(8, " 913/tcp                 # APEX "..., 4096) = 4096
read(8, "  1044/tcp                # Dev "..., 4096) = 4096
read(8, "    1079/tcp                # AS"..., 4096) = 4096
read(8, "             # ARDUS Transfer\nar"..., 4096) = 4096
read(8, "q-repl      1148/udp            "..., 4096) = 4096
read(8, "   # 3Com Net Management\naccelen"..., 4096) = 4096
read(8, "onDCE Gateway\naeroflight-ads  12"..., 4096) = 4096
read(8, "che-query\nde-server       1256/t"..., 4096) = 4096
read(8, "    # PKT-KRB-IPSec\ncmmdriver   "..., 4096) = 4096
read(8, "icy  1333/udp                # P"..., 4096) = 4096
read(8, "      1368/udp                # "..., 4096) = 4096
read(8, "blet   1400/tcp                #"..., 4096) = 4096
read(8, "               # Blueberry Softw"..., 4096) = 4096
read(8, "es           1465/tcp           "..., 4096) = 4096
read(8, "1504/udp                # EVB So"..., 4096) = 4096
read(8, "ridgen-elmd    1542/udp         "..., 4096) = 4096
read(8, "-lm      1578/udp               "..., 4096) = 4096
read(8, "              # NetBill Authoriz"..., 4096) = 4096
read(8, "\nstargatealerts  1654/udp       "..., 4096) = 4096
read(8, "# empire-empuma\nempire-empuma   "..., 4096) = 4096
read(8, "      # SIMS - SIIPAT Protocol f"..., 4096) = 4096
read(8, "       # EssWeb Gateway\nkmscontr"..., 4096) = 4096
read(8, "            # MMPFT\nharp        "..., 4096) = 4096
read(8, "  # VIDS-AVTP\nvids-avtp       18"..., 4096) = 4096
read(8, "ify Web Adapter Service\nunify-ad"..., 4096) = 4096
read(8, "     # XIIP\ndiscovery-port  1925"..., 4096) = 4096
read(8, "            # SIMP Channel\nsimp-"..., 4096) = 4096
brk(0)                                  = 0xb53000
brk(0xb74000)                           = 0xb74000
read(8, "     1994/udp                # c"..., 4096) = 4096
read(8, "           2040/tcp             "..., 4096) = 4096
read(8, "tsrmagt         2077/udp        "..., 4096) = 4096
read(8, "     2116/udp                # C"..., 4096) = 4096
read(8, "-User Plane (3GPP)\ngtp-user     "..., 4096) = 4096
read(8, "VD User\nnvd             2184/udp"..., 4096) = 4096
read(8, "       2220/tcp                #"..., 4096) = 4096
read(8, "       # DTV Channel Request\ndtv"..., 4096) = 4096
read(8, " # NETML\nnetml           2288/ud"..., 4096) = 4096
read(8, "      # ofsd\n3d-nfsd         232"..., 4096) = 4096
read(8, "p                # NexstorIndLtd"..., 4096) = 4096
read(8, "2406/tcp                # JediSe"..., 4096) = 4096
read(8, "pppsvr\nratl            2449/tcp "..., 4096) = 4096
read(8, "dp                # Oracle TTC S"..., 4096) = 4096
read(8, "       2522/udp                #"..., 4096) = 4096
read(8, "p                # labrat\nlabrat"..., 4096) = 4096
read(8, "nt 2598/tcp                # Cit"..., 4096) = 4096
read(8, "-samp        2643/tcp           "..., 4096) = 4096
read(8, "2680/udp                # pxc-sa"..., 4096) = 4096
read(8, "ange\nscan-change     2719/udp   "..., 4096) = 4096
read(8, "status   2758/udp               "..., 4096) = 4096
read(8, "eStats\nac-tech         2796/tcp "..., 4096) = 4096
read(8, "/udp                # EVTP\nevtp-"..., 4096) = 4096
read(8, "port Protocol\nsps-tunnel      28"..., 4096) = 4096
read(8, "Lobby\ngamelobby       2914/udp  "..., 4096) = 4096
read(8, "tcp                # MPFWSAS\nmpf"..., 4096) = 4096
read(8, "V Intelligent Agent Communicatio"..., 4096) = 4096
read(8, "otes\nnds_sso         3024/tcp  n"..., 4096) = 4096
read(8, "cacn-ip-tcp    3062/tcp         "..., 4096) = 4096
read(8, "         # Universal Message Man"..., 4096) = 4096
read(8, "cp             3134/udp         "..., 4096) = 4096
read(8, "erver\nserverview-as   3169/tcp  "..., 4096) = 4096
read(8, "tra\nnetwatcher-mon  3203/tcp    "..., 4096) = 4096
read(8, "ort\nmdap-port       3235/udp    "..., 4096) = 4096
brk(0)                                  = 0xb74000
brk(0xb95000)                           = 0xb95000
read(8, "p                # Microsoft Glo"..., 4096) = 4096
read(8, "3309/tcp                # TNS AD"..., 4096) = 4096
read(8, "viatv       3350/tcp            "..., 4096) = 4096
read(8, " Tapestry Client to Server\nd2k-t"..., 4096) = 4096
read(8, "     # GCSP user port\ngcsp      "..., 4096) = 4096
read(8, "DM ADM Notify\nedm-adm-notify  34"..., 4096) = 4096
read(8, "ether232port  3497/udp          "..., 4096) = 4096
read(8, "Joltid\nraven-rmp       3532/tcp "..., 4096) = 4096
read(8, "68/tcp                # EMIT sec"..., 4096) = 4096
read(8, "i     3601/udp                # "..., 4096) = 4096
read(8, "stributed Objects\nsdo           "..., 4096) = 4096
read(8, "ell Remote Management\ndell-rm-po"..., 4096) = 4096
read(8, "server-5   3705/tcp             "..., 4096) = 4096
read(8, "\nlaunchbird-lm   3739/tcp       "..., 4096) = 4096
read(8, "es     3773/udp                #"..., 4096) = 4096
read(8, "   # SpuGNA Communication Port\ns"..., 4096) = 4096
read(8, "         # Scito Object Server\na"..., 4096) = 4096
read(8, "dp                # Avocent DS A"..., 4096) = 4096
read(8, " Update (MUPDATE) protocol\nmupda"..., 4096) = 4096
read(8, "t po\naamp            3939/tcp   "..., 4096) = 4096
read(8, "       # LANrev Agent\nlanrevagen"..., 4096) = 4096
read(8, "             # pxc-roid\npxc-roid"..., 4096) = 4096
read(8, "yo-main         4040/tcp        "..., 4096) = 4096
read(8, "lly Incremental Backup\naibkup   "..., 4096) = 4096
read(8, "cel           4108/udp          "..., 4096) = 4096
read(8, "s   # Cedros Fraud Detection Sys"..., 4096) = 4096
read(8, "rvices\nsm-disc         4174/udp "..., 4096) = 4096
read(8, " Protocol\ntrim-event      4322/t"..., 4096) = 4096
read(8, "disc        4371/udp            "..., 4096) = 4096
read(8, "           # L-ACOUSTICS managem"..., 4096) = 4096
read(8, "  # MIH Services\nieee-mih       "..., 4096) = 4096
read(8, "ice\nmmaeds          4668/udp    "..., 4096) = 4096
read(8, "ux          4728/tcp            "..., 4096) = 4096
read(8, "         # OPC UA TCP Protocol\nq"..., 4096) = 4096
read(8, "             # Equitrac Office\ne"..., 4096) = 4096
read(8, "onpsocket       5014/udp        "..., 4096) = 4096
brk(0)                                  = 0xb95000
brk(0xbb6000)                           = 0xbb6000
read(8, " # STANAG-5066-SUBNET-INTF\nauthe"..., 4096) = 4096
read(8, "# MyCTS server port\nrmonitor_sec"..., 4096) = 4096
read(8, "t           5234/tcp            "..., 4096) = 4096
read(8, "     # opalis-rbt-ipc\nopalis-rbt"..., 4096) = 4096
read(8, " # Salient User Manager\nsalient-"..., 4096) = 4096
read(8, "ker   5465/tcp                # "..., 4096) = 4096
read(8, "     # A3-SDUNode\na4-sdunode    "..., 4096) = 4096
read(8, "video\nprosharedata    5715/tcp  "..., 4096) = 4096
read(8, "\nnetagent        5771/tcp       "..., 4096) = 4096
read(8, "tcp                # WBEM Export"..., 4096) = 4096
read(8, "uch    6117/tcp                #"..., 4096) = 4096
read(8, "er Interface\ngrid            626"..., 4096) = 4096
read(8, "# Business Objects Enterprise in"..., 4096) = 4096
read(8, "td    # BoKS Clntd\nbadm_priv    "..., 4096) = 4096
read(8, "   # Allied Electronics NeXGen\nn"..., 4096) = 4096
read(8, "b Console Admin\nsmc-http        "..., 4096) = 4096
read(8, " Serve Admin\nctdp            702"..., 4096) = 4096
read(8, " UPP Gateway\ndisplay         723"..., 4096) = 4096
read(8, "cp                # OpenView DM "..., 4096) = 4096
read(8, "penXDAS Wire Protocol\nhawk      "..., 4096) = 4096
read(8, "97/tcp                # Propel C"..., 4096) = 4096
read(8, "            # Intuit Entitlement"..., 4096) = 4096
read(8, "          # Check Point Clusteri"..., 4096) = 4096
read(8, " 8300/tcp                # Trans"..., 4096) = 4096
read(8, "-tunnel      8567/udp           "..., 4096) = 4096
read(8, "  # Desktop Data TCP 4: FARM pro"..., 4096) = 4096
read(8, "/udp                # IBM AURORA"..., 4096) = 4096
read(8, "re\nwap-vcal-s      9207/udp     "..., 4096) = 4096
read(8, "\nmphlpdmc        9344/tcp       "..., 4096) = 4096
read(8, "ervice\ncondor          9618/udp "..., 4096) = 4096
read(8, "  # HaloteC Instrument Network\na"..., 4096) = 4096
read(8, "ssaging\nnetiq-endpoint  10113/tc"..., 4096) = 4096
read(8, "p        11110/tcp              "..., 4096) = 4096
read(8, "mserver1   12005/tcp            "..., 4096) = 4096
read(8, "             # Blackmagic Design"..., 4096) = 4096
read(8, "ate Notification\npduncs         "..., 4096) = 4096
read(8, "    # OPSEC CVP\nopsec-ufp       "..., 4096) = 4096
read(8, "Interdevice Interaction discover"..., 4096) = 4096
brk(0)                                  = 0xbb6000
brk(0xbd7000)                           = 0xbd7000
read(8, "dard\ncaldsoft-backup 22537/tcp  "..., 4096) = 4096
read(8, "trix StorageLink Gateway\nassoc-d"..., 4096) = 4096
read(8, "m      28200/tcp               #"..., 4096) = 4096
read(8, "          # FileNET BPM IOR\nfile"..., 4096) = 4096
read(8, "rt\nsafetynetp      40000/tcp    "..., 4096) = 4096
read(8, "   45678/tcp               # EBA"..., 4096) = 2645
read(8, "", 4096)                       = 0
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=31961, ...}) = 0
mmap(NULL, 31961, PROT_READ, MAP_PRIVATE, 9, 0) = 0x7fba67742000
close(9)                                = 0
open("/lib64/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = 9
read(9, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\30\0\0\0\0\0\0"..., 832) = 832
fstat(9, {st_mode=S_IFREG|0755, st_size=37328, ...}) = 0
mmap(NULL, 2131056, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 9, 0) = 0x7fba62eed000
mprotect(0x7fba62ef5000, 2093056, PROT_NONE) = 0
mmap(0x7fba630f4000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 9, 0x7000) = 0x7fba630f4000
close(9)                                = 0
mprotect(0x7fba630f4000, 4096, PROT_READ) = 0
munmap(0x7fba67742000, 31961)           = 0
futex(0x7fba630f5268, FUTEX_WAKE_PRIVATE, 2147483647) = 0
fstat(-1, 0x7ffc5ea17560)               = -1 EBADF (Bad file descriptor)
socket(PF_LOCAL, SOCK_STREAM, 0)        = 9
fcntl(9, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(9, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
fcntl(9, F_GETFD)                       = 0
fcntl(9, F_SETFD, FD_CLOEXEC)           = 0
connect(9, {sa_family=AF_LOCAL, sun_path="/var/lib/sss/pipes/nss"}, 110) = 0
fstat(9, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0
poll([{fd=9, events=POLLOUT}], 1, 300000) = 1 ([{fd=9, revents=POLLOUT}])
sendto(9, "\24\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0", 16, MSG_NOSIGNAL, NULL, 0) = 16
poll([{fd=9, events=POLLOUT}], 1, 300000) = 1 ([{fd=9, revents=POLLOUT}])
sendto(9, "\1\0\0\0", 4, MSG_NOSIGNAL, NULL, 0) = 4
poll([{fd=9, events=POLLIN}], 1, 300000) = 1 ([{fd=9, revents=POLLIN}])
read(9, "\24\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
poll([{fd=9, events=POLLIN}], 1, 300000) = 1 ([{fd=9, revents=POLLIN}])
read(9, "\1\0\0\0", 4)                  = 4
poll([{fd=9, events=POLLOUT}], 1, 300000) = 1 ([{fd=9, revents=POLLOUT}])
sendto(9, "\20\0\0\0\243\0\0\0\0\0\0\0\0\0\0\0", 16, MSG_NOSIGNAL, NULL, 0) = 16
poll([{fd=9, events=POLLIN}], 1, 300000) = 1 ([{fd=9, revents=POLLIN}])
read(9, "\20\0\0\0\243\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
poll([{fd=9, events=POLLIN|POLLOUT}], 1, 300000) = 1 ([{fd=9, revents=POLLOUT}])
poll([{fd=9, events=POLLOUT}], 1, 300000) = 1 ([{fd=9, revents=POLLOUT}])
sendto(9, "\24\0\0\0\244\0\0\0\0\0\0\0\0\0\0\0", 16, MSG_NOSIGNAL, NULL, 0) = 16
poll([{fd=9, events=POLLOUT}], 1, 300000) = 1 ([{fd=9, revents=POLLOUT}])
sendto(9, "\0\1\0\0", 4, MSG_NOSIGNAL, NULL, 0) = 4
poll([{fd=9, events=POLLIN}], 1, 300000) = 1 ([{fd=9, revents=POLLIN}])
read(9, "\30\0\0\0\244\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
poll([{fd=9, events=POLLIN}], 1, 300000) = 1 ([{fd=9, revents=POLLIN}])
read(9, "\0\0\0\0\0\0\0\0", 8)          = 8
close(8)                                = 0
munmap(0x7fba67797000, 4096)            = 0
poll([{fd=9, events=POLLIN|POLLOUT}], 1, 300000) = 1 ([{fd=9, revents=POLLOUT}])
poll([{fd=9, events=POLLOUT}], 1, 300000) = 1 ([{fd=9, revents=POLLOUT}])
sendto(9, "\20\0\0\0\245\0\0\0\0\0\0\0\0\0\0\0", 16, MSG_NOSIGNAL, NULL, 0) = 16
poll([{fd=9, events=POLLIN}], 1, 300000) = 1 ([{fd=9, revents=POLLIN}])
read(9, "\20\0\0\0\245\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
rt_sigaction(SIGPIPE, {0x406590, [], SA_RESTORER, 0x7fba648b9100}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGTERM, {0x406590, [], SA_RESTORER, 0x7fba648b9100}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGINT, {0x406590, [], SA_RESTORER, 0x7fba648b9100}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGCHLD, {0x406580, [], SA_RESTORER|SA_RESTART, 0x7fba648b9100}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGHUP, {0x406590, [], SA_RESTORER, 0x7fba648b9100}, {SIG_DFL, [], 0}, 8) = 0
getuid()                                = 0
rt_sigaction(SIGUSR1, {0x407010, [], SA_RESTORER, 0x7fba648b9100}, {SIG_DFL, [], 0}, 8) = 0
write(2, "tcpdump: verbose output suppress"..., 75) = 75
write(2, "listening on zc:99@17, link-type"..., 78) = 78
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
nanosleep({0, 1000}, NULL)              = 0
.... 
```(repeated)

@cardigliano
Copy link
Member

Working with both standard and zc interfaces/queues on a similar system in our lab, need to investigate more.. do you have any packet encapsulation perhaps?
CentOS Linux release 7.2.1511 (Core)
Linux Host-001 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 31 16:04:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

@zachsis
Copy link

zachsis commented May 12, 2016

Verified with my network engineer that the traffic i am setting in the BPF would not be encapsulated. Also when i run tcpdump without any filters it dumps to stdout without any issues.

@jimhranicky
Copy link
Author

I'm getting the same feed on the RHEL box sent to another sensor running Ubuntu 12.04 and BPF seems to be working there.

RHEL PF_RING

PF_RING Version : 6.3.0 (dev:d568ce59908fd0021ec7910b0563db191301e61c)
Total rings : 1

Standard (non DNA/ZC) Options
Ring slots : 4096
Slot version : 16
Capture TX : Yes [RX+TX]
IP Defragment : No
Socket Mode : Standard
Total plugins : 0
Cluster Fragment Queue : 0
Cluster Fragment Discard : 0

Ubuntu PF_RING

PF_RING Version : 6.3.0 (dev:6c796df2a032d00202d590485124b67cffea3ef6)
Total rings : 24

Standard (non DNA/ZC) Options
Ring slots : 4096
Slot version : 16
Capture TX : Yes [RX+TX]
IP Defragment : No
Socket Mode : Standard
Total plugins : 0
Cluster Fragment Queue : 0
Cluster Fragment Discard : 0

What would be the best way to go about debugging this?

@zachsis
Copy link

zachsis commented May 20, 2016

@jimhranicky i'm confused are you saying you're seeing the same thing on the RHEL box and not the ubuntu?

@jimhranicky
Copy link
Author

Correct, with taps from the same source RHEL BPF filters are not working while the ubuntu ones are.

@zachsis
Copy link

zachsis commented May 31, 2016

attached are 2 strace's of tcpdump. One on a physical interface that works. The other on a pfring interface.

physical interface (working): physical-int-tcpdump-strace.txt

pfring interface (broken): pfring-int-tcpdump-strace.txt

@cardigliano
Copy link
Member

Hi guys
I am a bit confused, let's recap:

@jimhranicky

  • RH 3.10.0-327.13.1.el7.x86_64, vanilla drivers: NOT working

@zachsis

  • centos 7.2.1511, zc queue: NOT working
  • centos 7.2.1511, vanilla drivers: working

@zachsis
Copy link

zachsis commented Jun 1, 2016

@cardigliano yes that is correct on my end.

@cardigliano
Copy link
Member

A few updates:

  1. there is a new tcpdump version (4.7.4) available on github (worth test it)
  2. unfortunately I am still not able to reproduce any of the issues before on the same system/kernel
    Any chance to let me connect on any of your systems?

@zachsis
Copy link

zachsis commented Jun 2, 2016

@cardigliano I will email you privately and we can arrange a time to do a remote session.

@vbarahona
Copy link

I have the same issue here. Tcpdump is working right, but any bpf filters in nprobe are not working at all. No encapsulation, and traffic is coming from an optic tap.

Ubuntu 16.04.
nProbe v.7.3.160609 (r5255) for x86_64-pc-linux-gnu with native PF_RING acceleration.
PF_RING Version : 6.5.0 (dev:9f358aa8dd5b43bb74f67304c10ff41915e2f562)

When runing "nprobe -f host 127.0.0.1 -i ens1f1 -b 1", I have seen this logs in the initialization:

09/Jun/2016 15:53:44 [pro/pf_ring.c:377] Initializing PF_RING socket on device ens1f1..
09/Jun/2016 15:53:44 [pro/pf_ring.c:419] Dumping traffic statistics on /proc/net/pf_ring/stats/12433-ens1f1.5
09/Jun/2016 15:53:44 [pro/pf_ring.c:452] WARNING: Unable to set PF_RING filter 'host' [rc=-1/Success]
09/Jun/2016 15:53:44 [pro/pf_ring.c:484] PF_RING enabled on ens1f1

I can provide you any needed additional information.

Regards.

@cardigliano
Copy link
Member

@vbarahona please use quotation marks: -f "host 127.0.0.1"

@vbarahona
Copy link

Thanks @cardigliano that works. :-)

Suggestion: there is not mention of needs of quotation marks in the bpf filter nor man page nor "nProbe Users Guide" and since are not need when using tcpdump it should be notice in documentation.

@jimhranicky
Copy link
Author

I just yum updated to 3.10.0-327.18.2.el7.x86_64 and installed the rpms from ntop.org . I'm still getting the same behavior.

There does seem to be a difference in the behavior of the libraries as reported by ltrace :

Preload the pf_ring libpcap:

% LD_PRELOAD=/usr/local/lib/libpcap.so.1 ltrace -e '@libpcap.so' tcpdump -i enp4s0 -nn -c 10 'port 22' 2>&1 | grep bpf
libpcap.so.1->bpf_optimize(0x7f75b77fa368, 0, 0, 1 <unfinished ...>
<... bpf_optimize resumed> ) = 0xc5c1
libpcap.so.1->install_bpf_program(0x1affb50, 0x7ffd98068b10, 0, -1 <unfinished ...>
libpcap.so.1->bpf_validate(0x1b13a50, 24, 0, -1) = 1
<... install_bpf_program resumed> ) = 0

Normal tcpdump:

% ltrace -e '@libpcap.so' tcpdump -i enp4s0 -nn -c 10 'port 22' 2>&1 | grep bpf
libpcap.so.1->bpf_optimize(0x7f067217fb68, 0, 0, 1 <unfinished ...>
<... bpf_optimize resumed> ) = 0xc8d1
libpcap.so.1->install_bpf_program(0x199bb50, 0x7ffd728c4f80, 1, -1 <unfinished ...>
libpcap.so.1->bpf_validate(0x19af740, 24, 1, -1) = 1
<... install_bpf_program resumed> ) = 0

  • libpcap.so.1->bpf_filter_with_aux_data(0x19ac400, 0x7f0671384046, 1022, 1022) = 0
  • [...bpf_filter_with_aux_data repeats 31 times...]

It doesn't seem that the pf_ring libpcap calls bpf_filter_with_aux_data . Does
this help?

@cardigliano
Copy link
Member

@jimhranicky thank you for the hint. Do you experience the same issue using the tcpdump in pf_ring, compiled with static libpcap (not the system tcpdump with LD_PRELOAD)?

@jimhranicky
Copy link
Author

Yes:

% ldd /opt/pf/sbin/tcpdump
linux-vdso.so.1 => (0x00007ffd43be3000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007ff1222d1000)
librt.so.1 => /lib64/librt.so.1 (0x00007ff1220c8000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff121eac000)
libm.so.6 => /lib64/libm.so.6 (0x00007ff121baa000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff1217e7000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007ff1215e3000)
libz.so.1 => /lib64/libz.so.1 (0x00007ff1213cd000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff1226c8000)

% strings - /opt/pf/sbin/tcpdump | grep pfring | head -20
[PF_RING] Wrong RING version: kernel is %i, libpfring was compiled with %i
[PF_RING] ring failure (pfring_get_slot_header_len)
pfring_get_mapped_dna_device() failed [rc=%d]
pfring_map_dna_device
[ERROR] with libpfring.a. Please update the DNA library.
pfring-zc-%d-%s
pfring-zc-cluster-%d
/dev/virtio-ports/pfring-zc-vport-%u
pfring_zc_%u
pfring-zc-kvm-shmem-sock-
pfring-zc-kvm-vport-sock-
pfring-zc-vport-
pfring.h
pfring.h
pfring.h
pfring.h
pfring.h
pfring.h
pfring.h
pfring.h

% /opt/pf/sbin/tcpdump -i enp4s0 -nn -c 10 'port 22' | perl -pe 's,\d+.\d+.\d+.\d+,XX.XX.XX.XX,g'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp4s0, link-type EN10MB (Ethernet), capture size 262144 bytes
10 packets captured
10 packets received by filter
0 packets dropped by kernel
15:42:32.542244213 IP XX.XX.XX.XX.443 > XX.XX.XX.XX.62781: UDP, length 1350
15:42:32.542246410 IP XX.XX.XX.XX.443 > XX.XX.XX.XX.59144: Flags [.], seq 2296792287:2296793747, ack 4071331999, win 62, length 1460
15:42:32.542248541 IP XX.XX.XX.XX.80 > XX.XX.XX.XX.53980: Flags [.], ack 3618022251, win 5584, length 0
15:42:32.542249006 IP XX.XX.XX.XX.443 > XX.XX.XX.XX.55598: Flags [.], ack 593798320, win 350, options [nop,nop,TS val 416574482 ecr 913401700], length 0
15:42:32.542248031 IP XX.XX.XX.XX.443 > XX.XX.XX.XX.62781: UDP, length 1350
15:42:32.542253427 IP XX.XX.XX.XX.37626 > XX.XX.XX.XX.64520: UDP, length 41
15:42:32.542253706 IP XX.XX.XX.XX.53966 > XX.XX.XX.XX.443: Flags [P.], seq 2020710854:2020712284, ack 2696003636, win 252, length 1430
15:42:32.542253597 IP XX.XX.XX.XX.443 > XX.XX.XX.XX.55598: Flags [P.], seq 0:38, ack 1, win 350, options [nop,nop,TS val 416574482 ecr 913401700], length 38
15:42:32.542256035 IP XX.XX.XX.XX.56722 > XX.XX.XX.XX.80: Flags [.], ack 403565785, win 181, options [nop,nop,TS val 1674918449 ecr 1796252449], length 0
15:42:32.542259211 IP XX.XX.XX.XX.49787 > XX.XX.XX.XX.443: Flags [.], ack 1679445011, win 4113, options [nop,nop,TS val 357118138 ecr 1433921874], length 0

@zachsis
Copy link

zachsis commented Jul 12, 2016

Rebuiling PF_RING to version 6.5.0 resolved this issue for me. Many thanks @cardigliano for the assistance.

@cardigliano
Copy link
Member

@jimhranicky could you also try updating to latest dev code, or providing me access to the machine? Thank you

@jimhranicky
Copy link
Author

Sorry, this got sent to my spam folder, will try updating soon.

Jim

On 07/13/2016 04:28 AM, Alfredo Cardigliano wrote:

@jimhranicky https://github.com/jimhranicky could you also try
updating to latest dev code, or providing me access to the machine?
Thank you

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#94 (comment), or
mute the thread
https://github.com/notifications/unsubscribe/AAO25Mobb0NQ6VmJxwPHYVbK1e7meg2Hks5qVKGdgaJpZM4IaVwd.

@jimhranicky
Copy link
Author

Same behavior for 6.5.0:

Jul 22 14:07:51 pdump kernel: [PF_RING] Welcome to PF_RING 6.5.0 ($Revision: dev:8f298fcf9e4ccc33c2bf46bb94f00d0d46dd
36f6$)#12(C) 2004-16 ntop.org

Do you want to try a webex or something?

@cardigliano
Copy link
Member

@jimhranicky please send to cardigliano@ntop.org the details for connecting to your machine (preferably ssh, webex also works, send me your availability in the latter case). Thank you.

@jimhranicky
Copy link
Author

jimhranicky commented Jul 27, 2016 •
edited

I ran gdb on both the pf tcpdump and the latest tcpdump from tcpdump.org and it seems that
the 'handlep->filter_in_userland' is not set when using pf tcpdump.

tcpdump-4.7.4/pcap-linux:4536

    if (handlep->filter_in_userland && handle->fcode.bf_insns) {
            struct bpf_aux_data aux_data;

            aux_data.vlan_tag = tp_vlan_tci & 0x0fff;
            aux_data.vlan_tag_present = tp_vlan_tci_valid;

            if (bpf_filter_with_aux_data(handle->fcode.bf_insns, bp,
                tp_len, tp_snaplen, &aux_data) == 0)
                    return 0;
    }

Tracing through the pf_ring tcpdump, I see that variable getting unset:

libpcap-1.7.4/pcap-linux.c : 
      if (can_filter_in_kernel) {
              if ((err = set_kernel_filter(handle, &fcode)) == 0)
              {
                      /*                                                                                         
                       * Installation succeded - using kernel filter,                                            
                       * so userland filtering not needed.                                                       
                       */
  =>                  handlep->filter_in_userland = 0;
              }

I don't know if that's what's causing the problem.

@jimhranicky
Copy link
Author

Ok, seeing as how this may be a problem with the kernel filters, I set enable_debug = 1 in pf_ring.c .
Here's the output from a run of the pf-ring-enabled tcpdump:

Jul 28 10:27:18 pdump kernel: [PF_RING] ring_create() [pid=8699]
Jul 28 10:27:18 pdump kernel: [PF_RING] ring_insert
Jul 28 10:27:18 pdump kernel: [PF_RING] -> BEGIN lockless_list_add() [total=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] -> END lockless_list_add() [total=1][id=0][top_element_id=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] -> lockless_list_add() [slot 0 is full]
Jul 28 10:27:18 pdump kernel: [PF_RING] Added /proc/net/pf_ring/8699-none.2
Jul 28 10:27:18 pdump kernel: [PF_RING] ring_create(): created
Jul 28 10:27:18 pdump kernel: [PF_RING] --> ring_setsockopt(optname=107)
Jul 28 10:27:18 pdump kernel: [PF_RING] --> SO_RING_BUCKET_LEN=65535
Jul 28 10:27:18 pdump kernel: [PF_RING] --> ring_setsockopt(optname=127)
Jul 28 10:27:18 pdump kernel: [PF_RING] ring_bind() called
Jul 28 10:27:18 pdump kernel: [PF_RING] searching device enp4s0
Jul 28 10:27:18 pdump kernel: [PF_RING] packet_ring_bind(enp4s0, bucket_len=65535) called
Jul 28 10:27:18 pdump kernel: [PF_RING] Removing /proc/net/pf_ring/8699-none.2
Jul 28 10:27:18 pdump kernel: [PF_RING] Removed /proc/net/pf_ring/8699-none.2
Jul 28 10:27:18 pdump kernel: [PF_RING] Added /proc/net/pf_ring/8699-enp4s0.2
Jul 28 10:27:18 pdump kernel: [PF_RING] --> ring_setsockopt(optname=108)
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 0
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 1
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 2
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 3
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 4
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 5
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 6
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 7
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 8
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 9
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 10
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 11
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 12
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 13
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 14
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 15
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 16
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 17
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 18
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 19
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 20
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 21
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 22
Jul 28 10:27:18 pdump kernel: [PF_RING] Setting channel 23
Jul 28 10:27:18 pdump kernel: [PF_RING] [pfr->channel_id_mask=FFFFFFFFFFFFFFFF][channel_id_mask=FFFFFFFFFFFFFFFF]
Jul 28 10:27:18 pdump kernel: [PF_RING] ring_mmap() called
Jul 28 10:27:18 pdump kernel: [PF_RING] ring_mmap() called, size: 4096 bytes [bucket_len=65535]
Jul 28 10:27:18 pdump kernel: [PF_RING] ring_alloc_mem(bucket_len=65535)
Jul 28 10:27:18 pdump kernel: [PF_RING] successfully allocated 268677120 bytes at 0xffffc9001374c000
Jul 28 10:27:18 pdump kernel: [PF_RING] allocated 4096 slots [slot_len=65592][tot_mem=268677120]
Jul 28 10:27:18 pdump kernel: [PF_RING] mmap [slot_len=65592][tot_slots=4096] for ring on device enp4s0
Jul 28 10:27:18 pdump kernel: [PF_RING] do_memory_mmap(mode=0, size=4096, ptr=ffffc9001374c000)
Jul 28 10:27:18 pdump kernel: [PF_RING] ring_mmap succeeded
Jul 28 10:27:18 pdump kernel: [PF_RING] ring_mmap() called
Jul 28 10:27:18 pdump kernel: [PF_RING] ring_mmap() called, size: 268677120 bytes [bucket_len=65535]
Jul 28 10:27:18 pdump kernel: [PF_RING] mmap [slot_len=65592][tot_slots=4096] for ring on device enp4s0
Jul 28 10:27:18 pdump kernel: [PF_RING] do_memory_mmap(mode=0, size=268677120, ptr=ffffc9001374c000)
Jul 28 10:27:18 pdump kernel: [PF_RING] ring_mmap succeeded
Jul 28 10:27:18 pdump kernel: [PF_RING] --> ring_setsockopt(optname=140)
Jul 28 10:27:18 pdump kernel: [PF_RING] --> getsockopt(179)
Jul 28 10:27:18 pdump kernel: [PF_RING] --> getsockopt(182)
Jul 28 10:27:18 pdump kernel: [PF_RING] --> getsockopt(184)
Jul 28 10:27:18 pdump kernel: [PF_RING] --> ring_setsockopt(optname=117)
Jul 28 10:27:18 pdump kernel: [PF_RING] --> SO_SET_POLL_WATERMARK=1
Jul 28 10:27:18 pdump kernel: [PF_RING] --> ring_setsockopt(optname=106)
Jul 28 10:27:18 pdump kernel: [PF_RING] * SO_ACTIVATE_RING *
Jul 28 10:27:18 pdump kernel: [PF_RING] --> getsockopt(184)
Jul 28 10:27:18 pdump kernel: [PF_RING] --> ring_setsockopt(optname=26)
Jul 28 10:27:18 pdump kernel: [PF_RING] BPF filter
Jul 28 10:27:18 pdump kernel: [PF_RING] --> ring_setsockopt(optname=26)
Jul 28 10:27:18 pdump kernel: [PF_RING] BPF filter
Jul 28 10:27:18 pdump kernel: [PF_RING] BPF filter (len = 24)
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1472][tot=158][insert_off=212312][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1416][tot=160][insert_off=213960][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=137][tot=160][insert_off=213960][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=599][tot=160][insert_off=213960][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1416][tot=160][insert_off=213960][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=2900][tot=160][insert_off=213960][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=11133][tot=161][insert_off=214072][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=121][tot=161][insert_off=214072][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=161][insert_off=214072][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=161][insert_off=214072][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1388][tot=161][insert_off=214072][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=216][tot=161][insert_off=214072][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=161][insert_off=214072][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=239][tot=161][insert_off=214072][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=121][tot=161][insert_off=214072][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=60][tot=161][insert_off=214072][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1454][tot=161][insert_off=214072][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=7152][tot=161][insert_off=214072][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=162][insert_off=214184][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=16452][tot=162][insert_off=214184][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1318][tot=163][insert_off=214296][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=163][insert_off=214296][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=4312][tot=163][insert_off=214296][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1472][tot=163][insert_off=214296][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=164][insert_off=214296][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=164][insert_off=214296][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1500][tot=164][insert_off=229000][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1317][tot=164][insert_off=229000][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=164][insert_off=229000][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1454][tot=164][insert_off=229000][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=164][insert_off=229000][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=164][insert_off=229000][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=52][tot=164][insert_off=229000][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=52][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=121][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=2960][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=165][insert_off=229112][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=52][tot=166][insert_off=229224][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1470][tot=166][insert_off=229224][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1653][tot=166][insert_off=229224][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=128][tot=166][insert_off=229224][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=65][tot=166][insert_off=229224][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=166][insert_off=229224][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=166][insert_off=229224][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=166][insert_off=229224][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=166][insert_off=229224][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=166][insert_off=229224][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=64][tot=166][insert_off=229224][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=166][insert_off=229224][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1766][tot=166][insert_off=229224][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=167][insert_off=229336][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1064][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=147][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1323][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=290][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=93][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=60][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=52][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=142][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1470][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=52][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1885][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=52][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=168][insert_off=229448][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=52][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=64][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=149][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1040][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1317][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=194][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1040][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=174][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=196][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1038][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=169][insert_off=229560][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=46][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1472][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=2760][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1472][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1500][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1454][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=1480][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=52][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] --> ring_setsockopt(optname=124)
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=69][tot=171][insert_off=229784][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] bpf_filter_skb(skb): Filter failed [len=52][tot=172][insert_off=229896][pkt_type=3][cloned=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] called ring_release(enp4s0)
Jul 28 10:27:18 pdump kernel: [PF_RING] Removing /proc/net/pf_ring/8699-enp4s0.2
Jul 28 10:27:18 pdump kernel: [PF_RING] Removed /proc/net/pf_ring/8699-enp4s0.2
Jul 28 10:27:18 pdump kernel: [PF_RING] ring_remove()
Jul 28 10:27:18 pdump kernel: [PF_RING] Found socket to remove
Jul 28 10:27:18 pdump kernel: [PF_RING] -> BEGIN lockless_list_remove() [total=1]
Jul 28 10:27:18 pdump kernel: [PF_RING] -> END lockless_list_remove() [total=0][top_element_id=0]
Jul 28 10:27:18 pdump kernel: [PF_RING] leaving ring_remove()
Jul 28 10:27:18 pdump kernel: [PF_RING] ring_release: done

@cardigliano
Copy link
Member

"Filter failed" actually is not a failure, it means that the bpf filter is filtering out packets (I will change the message). However this means that the filter is somehow working (at least it is filtering something). What is strange is that vanilla tcpdump is not using kernel filtering on your OS..

@jimhranicky
Copy link
Author

I don't know if this is useful, but there's a difference between using -i and -i zc: :

Without BPF :

/opt/pf/sbin/tcpdump -i enp4s0 -nns0 -c 1000000 -w /dev/null

Warning: Kernel filter failed: Bad address
tcpdump: listening on enp4s0, link-type EN10MB (Ethernet), capture size 262144 bytes
1000000 packets captured
1000000 packets received by filter
0 packets dropped by kernel

/opt/pf/sbin/tcpdump -i zc:enp4s0 -nns0 -c 1000000 -w /dev/null

tcpdump: listening on zc:enp4s0, link-type EN10MB (Ethernet), capture size 262144 bytes
1000000 packets captured
1000000 packets received by filter
89969 packets dropped by kernel

With BPF:

(filter fails)

/opt/pf/sbin/tcpdump -i enp4s0 -nns0 -c 5 'port 22' | perl -pe 's,(\d+.\d+.\d+.\d+),XX.XX.XX.XX,g'

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp4s0, link-type EN10MB (Ethernet), capture size 262144 bytes
5 packets captured
5 packets received by filter
0 packets dropped by kernel
10:40:34.378003 IP XX.XX.XX.XX.80 > XX.XX.XX.XX.53047: Flags [.], seq 3886447579:3886449027, ack 1052229497, win 1093, options [nop,nop,TS val 77511972 ecr 235889190], length 1448: HTTP
10:40:34.378008 IP XX.XX.XX.XX.443 > XX.XX.XX.XX.51794: Flags [P.], seq 3876659856:3876659902, ack 1673482180, win 2038, options [nop,nop,TS val 236226646 ecr 1284609527], length 46
10:40:34.378010 IP XX.XX.XX.XX.80 > XX.XX.XX.XX.53047: Flags [.], seq 1448:2896, ack 1, win 1093, options [nop,nop,TS val 77511972 ecr 235889190], length 1448: HTTP
10:40:34.378014 IP XX.XX.XX.XX.443 > XX.XX.XX.XX.12513: Flags [P.], seq 2983083818:2983085266, ack 1080453012, win 51793, options [nop,nop,TS val 2738819215 ecr 1149054174], length 1448
10:40:34.378018 IP XX.XX.XX.XX.80 > XX.XX.XX.XX.50157: Flags [.], seq 1336300696:1336302144, ack 222815138, win 63, options [nop,nop,TS val 131892602 ecr 373107311], length 1448: HTTP

(filter succeeds)

/opt/pf/sbin/tcpdump -i zc:enp4s0 -nns0 -c 5 'port 22' | perl -pe 's,(\d+.\d+.\d+.\d+),XX.XX.XX.XX,g'

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on zc:enp4s0, link-type EN10MB (Ethernet), capture size 262144 bytes
5 packets captured
645 packets received by filter
896 packets dropped by kernel
10:40:25.467296 IP XX.XX.XX.XX.52138 > XX.XX.XX.XX.22: Flags [.], ack 805655556, win 24576, length 0
10:40:25.467618 IP XX.XX.XX.XX.36470 > XX.XX.XX.XX.22: Flags [S], seq 2440718091, win 14600, options [mss 1460,sackOK,TS val 266060178 ecr 0,nop,wscale 6], length 0
10:40:25.467659 IP XX.XX.XX.XX.22 > XX.XX.XX.XX.52138: Flags [.], seq 81761:83221, ack 0, win 2272, length 1460
10:40:25.467669 IP XX.XX.XX.XX.22 > XX.XX.XX.XX.52138: Flags [.], seq 83221:84681, ack 0, win 2272, length 1460
10:40:25.467681 IP XX.XX.XX.XX.22 > XX.XX.XX.XX.52138: Flags [P.], seq 84681:86141, ack 0, win 2272, length 1460

@jasontant
Copy link

@jimhranicky @cardigliano Have there been any further developments on this issue? We are encountering the same problem (BPF filters not working) with 6.4.1 and 6.5.0 on Ubuntu 16.04.1.

@cardigliano
Copy link
Member

@sonysouthbeach do you have the same issue (filter not working) all the time during a capture session, or just at the beginning of the session (let's say for a couple of seconds)?

@jasontant
Copy link

@cardigliano First couple of seconds only.

@cardigliano
Copy link
Member

This was due to libpcap activating the socket before setting the filter. Please update the code from github dev (or wait for the nightly build) and let us know. Thank you.

@jasontant
Copy link

Will do! Thanks so much for the quick response.

@jasontant
Copy link

@cardigliano We were able to confirm that this fixed the BPF filtering problem. Thanks again.

@cardigliano
Copy link
Member

Great, thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jimhranicky @zachsis @vbarahona @cardigliano @jasontant