Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Buffer overflow when reading lots of PCAPs #627

Open
philrz opened this issue Oct 8, 2019 · 3 comments

Comments

@philrz
Copy link

commented Oct 8, 2019

I'm easily able to repro this with an out-of-the-box Zeek v3.0.0 compiled on Ubuntu 18.04.1.0.

First I cook up a tiny dummy PCAP.

# tcpdump -i enp0s3 -c 1 -w foo.pcap

Now I make lots of copies of it and try to read them into Zeek all at once.

# mkdir foo && cd foo
# ulimit -n 1000000
# NUM=1
# while [ $NUM -le 5000 ] ; do   cp ../foo.pcap $NUM.pcap;   NUM=`expr $NUM + 1`; done
# zeek $(for file in $(ls -v1 *.pcap); do   /bin/echo -n "-r $file "; done) local
WARNING: No Site::local_nets have been defined.  It's usually a good idea to define your local networks.
*** buffer overflow detected ***: zeek terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f55271b27e5]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f552725415c]
/lib/x86_64-linux-gnu/libc.so.6(+0x117160)[0x7f5527252160]
/lib/x86_64-linux-gnu/libc.so.6(+0x1190a7)[0x7f55272540a7]
zeek(_ZNK8iosource7Manager6Source6SetFdsEP6fd_setS3_S3_Pi+0x3c)[0x95276c]
zeek(_ZN8iosource7Manager11FindSoonestEPd+0x28e)[0x95316e]
zeek(_Z7net_runv+0x72)[0x6d4c82]
zeek(main+0x1a06)[0x5852b6]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f552715b830]
zeek(_start+0x29)[0x590b59]
======= Memory map: ========
00400000-00b26000 r-xp 00000000 fc:00 16706                              /usr/local/zeek/bin/zeek
00d25000-00d27000 r--p 00725000 fc:00 16706                              /usr/local/zeek/bin/zeek
00d27000-00d2b000 rw-p 00727000 fc:00 16706                              /usr/local/zeek/bin/zeek
00d2b000-00d40000 rw-p 00000000 00:00 0 
01cc0000-083da000 rw-p 00000000 00:00 0                                  [heap]
7f54b0000000-7f54b0021000 rw-p 00000000 00:00 0 
7f54b0021000-7f54b4000000 ---p 00000000 00:00 0 
7f54b4000000-7f54b4021000 rw-p 00000000 00:00 0 
7f54b4021000-7f54b8000000 ---p 00000000 00:00 0 
7f54b8000000-7f54b8021000 rw-p 00000000 00:00 0 
7f54b8021000-7f54bc000000 ---p 00000000 00:00 0 
7f54be250000-7f54be251000 ---p 00000000 00:00 0 
7f54be251000-7f54bea51000 rw-p 00000000 00:00 0 
7f54bea51000-7f54bea52000 ---p 00000000 00:00 0 
7f54bea52000-7f54bf252000 rw-p 00000000 00:00 0 
7f54bf252000-7f54bf253000 ---p 00000000 00:00 0 
7f54bf253000-7f54bfa53000 rw-p 00000000 00:00 0 
7f54bfa53000-7f54bfa54000 ---p 00000000 00:00 0 
7f54bfa54000-7f550c000000 rw-p 00000000 00:00 0 
7f550c000000-7f550c021000 rw-p 00000000 00:00 0 
7f550c021000-7f5510000000 ---p 00000000 00:00 0 
7f5510000000-7f5510021000 rw-p 00000000 00:00 0 
7f5510021000-7f5514000000 ---p 00000000 00:00 0 
7f5514000000-7f5514021000 rw-p 00000000 00:00 0 
7f5514021000-7f5518000000 ---p 00000000 00:00 0 
7f5518000000-7f5518021000 rw-p 00000000 00:00 0 
7f5518021000-7f551c000000 ---p 00000000 00:00 0 
7f551c019000-7f551f21e000 rw-p 00000000 00:00 0 
7f551f5ed000-7f551f5f8000 r-xp 00000000 fc:00 576                        /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f551f5f8000-7f551f7f7000 ---p 0000b000 fc:00 576                        /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f551f7f7000-7f551f7f8000 r--p 0000a000 fc:00 576                        /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f551f7f8000-7f551f7f9000 rw-p 0000b000 fc:00 576                        /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f551f7f9000-7f551f7ff000 rw-p 00000000 00:00 0 
7f551f7ff000-7f551f800000 ---p 00000000 00:00 0 
7f551f800000-7f5520000000 rw-p 00000000 00:00 0 
7f5520000000-7f5520021000 rw-p 00000000 00:00 0 
7f5520021000-7f5524000000 ---p 00000000 00:00 0 
7f552404a000-7f552404b000 ---p 00000000 00:00 0 
7f552404b000-7f552484b000 rw-p 00000000 00:00 0 
7f552484b000-7f552484c000 ---p 00000000 00:00 0 
7f552484c000-7f552504c000 rw-p 00000000 00:00 0 
7f552504c000-7f552504d000 ---p 00000000 00:00 0 
7f552504d000-7f552584d000 rw-p 00000000 00:00 0 
7f552584d000-7f552584e000 ---p 00000000 00:00 0 
7f552584e000-7f552604e000 rw-p 00000000 00:00 0 
7f552604e000-7f552604f000 ---p 00000000 00:00 0 
7f552604f000-7f5526a50000 rw-p 00000000 00:00 0 
7f5526a50000-7f5526ab4000 r-xp 00000000 fc:00 16607                      /usr/local/zeek/lib/libcaf_openssl.so.0.16.4
7f5526ab4000-7f5526cb4000 ---p 00064000 fc:00 16607                      /usr/local/zeek/lib/libcaf_openssl.so.0.16.4
7f5526cb4000-7f5526cb6000 r--p 00064000 fc:00 16607                      /usr/local/zeek/lib/libcaf_openssl.so.0.16.4
7f5526cb6000-7f5526cb7000 rw-p 00066000 fc:00 16607                      /usr/local/zeek/lib/libcaf_openssl.so.0.16.4
7f5526cb7000-7f5526f28000 r-xp 00000000 fc:00 16534                      /usr/local/zeek/lib/libcaf_io.so.0.16.4
7f5526f28000-7f5527128000 ---p 00271000 fc:00 16534                      /usr/local/zeek/lib/libcaf_io.so.0.16.4
7f5527128000-7f5527139000 r--p 00271000 fc:00 16534                      /usr/local/zeek/lib/libcaf_io.so.0.16.4
7f5527139000-7f552713b000 rw-p 00282000 fc:00 16534                      /usr/local/zeek/lib/libcaf_io.so.0.16.4
7f552713b000-7f55272fb000 r-xp 00000000 fc:00 512                        /lib/x86_64-linux-gnu/libc-2.23.so
7f55272fb000-7f55274fb000 ---p 001c0000 fc:00 512                        /lib/x86_64-linux-gnu/libc-2.23.so
7f55274fb000-7f55274ff000 r--p 001c0000 fc:00 512                        /lib/x86_64-linux-gnu/libc-2.23.so
7f55274ff000-7f5527501000 rw-p 001c4000 fc:00 512                        /lib/x86_64-linux-gnu/libc-2.23.so
7f5527501000-7f5527505000 rw-p 00000000 00:00 0 
7f5527505000-7f552751b000 r-xp 00000000 fc:00 538                        /lib/x86_64-linux-gnu/libgcc_s.so.1
7f552751b000-7f552771a000 ---p 00016000 fc:00 538                        /lib/x86_64-linux-gnu/libgcc_s.so.1
7f552771a000-7f552771b000 rw-p 00015000 fc:00 538                        /lib/x86_64-linux-gnu/libgcc_s.so.1
7f552771b000-7f5527823000 r-xp 00000000 fc:00 553                        /lib/x86_64-linux-gnu/libm-2.23.so
7f5527823000-7f5527a22000 ---p 00108000 fc:00 553                        /lib/x86_64-linux-gnu/libm-2.23.so
7f5527a22000-7f5527a23000 r--p 00107000 fc:00 553                        /lib/x86_64-linux-gnu/libm-2.23.so
7f5527a23000-7f5527a24000 rw-p 00108000 fc:00 553                        /lib/x86_64-linux-gnu/libm-2.23.so
7f5527a24000-7f5527b96000 r-xp 00000000 fc:00 132819                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f5527b96000-7f5527d96000 ---p 00172000 fc:00 132819                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f5527d96000-7f5527da0000 r--p 00172000 fc:00 132819                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f5527da0000-7f5527da2000 rw-p 0017c000 fc:00 132819                     /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f5527da2000-7f5527da6000 rw-p 00000000 00:00 0 
7f5527da6000-7f5527fdb000 r-xp 00000000 fc:00 16169                      /usr/local/zeek/lib/libcaf_core.so.0.16.4
7f5527fdb000-7f55281da000 ---p 00235000 fc:00 16169                      /usr/local/zeek/lib/libcaf_core.so.0.16.4
7f55281da000-7f55281e9000 r--p 00234000 fc:00 16169                      /usr/local/zeek/lib/libcaf_core.so.0.16.4
7f55281e9000-7f55281ec000 rw-p 00243000 fc:00 16169                      /usr/local/zeek/lib/libcaf_core.so.0.16.4
7f55281ec000-7f55281ed000 rw-p 00000000 00:00 0 
7f55281ed000-7f55281f0000 r-xp 00000000 fc:00 526                        /lib/x86_64-linux-gnu/libdl-2.23.so
7f55281f0000-7f55283ef000 ---p 00003000 fc:00 526                        /lib/x86_64-linux-gnu/libdl-2.23.so
7f55283ef000-7f55283f0000 r--p 00002000 fc:00 526                        /lib/x86_64-linux-gnu/libdl-2.23.so
7f55283f0000-7f55283f1000 rw-p 00003000 fc:00 526                        /lib/x86_64-linux-gnu/libdl-2.23.so
7f55283f1000-7f5528409000 r-xp 00000000 fc:00 599                        /lib/x86_64-linux-gnu/libpthread-2.23.so
7f5528409000-7f5528608000 ---p 00018000 fc:00 599                        /lib/x86_64-linux-gnu/libpthread-2.23.so
7f5528608000-7f5528609000 r--p 00017000 fc:00 599                        /lib/x86_64-linux-gnu/libpthread-2.23.so
7f5528609000-7f552860a000 rw-p 00018000 fc:00 599                        /lib/x86_64-linux-gnu/libpthread-2.23.so
7f552860a000-7f552860e000 rw-p 00000000 00:00 0 
7f552860e000-7f55289b2000 r-xp 00000000 fc:00 16694                      /usr/local/zeek/lib/libbroker.so.1.2
7f55289b2000-7f5528bb2000 ---p 003a4000 fc:00 16694                      /usr/local/zeek/lib/libbroker.so.1.2
7f5528bb2000-7f5528bc9000 r--p 003a4000 fc:00 16694                      /usr/local/zeek/lib/libbroker.so.1.2
7f5528bc9000-7f5528bce000 rw-p 003bb000 fc:00 16694                      /usr/local/zeek/lib/libbroker.so.1.2
7f5528bce000-7f5528bcf000 rw-p 00000000 00:00 0 
7f5528bcf000-7f5528be8000 r-xp 00000000 fc:00 635                        /lib/x86_64-linux-gnu/libz.so.1.2.8
7f5528be8000-7f5528de7000 ---p 00019000 fc:00 635                        /lib/x86_64-linux-gnu/libz.so.1.2.8
7f5528de7000-7f5528de8000 r--p 00018000 fc:00 635                        /lib/x86_64-linux-gnu/libz.so.1.2.8
7f5528de8000-7f5528de9000 rw-p 00019000 fc:00 635                        /lib/x86_64-linux-gnu/libz.so.1.2.8
7f5528de9000-7f5529004000 r-xp 00000000 fc:00 14251                      /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f5529004000-7f5529203000 ---p 0021b000 fc:00 14251                      /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f5529203000-7f552921f000 r--p 0021a000 fc:00 14251                      /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f552921f000-7f552922b000 rw-p 00236000 fc:00 14251                      /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f552922b000-7f552922e000 rw-p 00000000 00:00 0 
7f552922e000-7f552928c000 r-xp 00000000 fc:00 14252                      /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f552928c000-7f552948c000 ---p 0005e000 fc:00 14252                      /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f552948c000-7f5529490000 r--p 0005e000 fc:00 14252                      /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f5529490000-7f5529497000 rw-p 00062000 fc:00 14252                      /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f5529497000-7f55294d5000 r-xp 00000000 fc:00 146305                     /usr/lib/x86_64-linux-gnu/libpcap.so.1.7.4
7f55294d5000-7f55296d5000 ---p 0003e000 fc:00 146305                     /usr/lib/x86_64-linux-gnu/libpcap.so.1.7.4
7f55296d5000-7f55296d7000 r--p 0003e000 fc:00 146305                     /usr/lib/x86_64-linux-gnu/libpcap.so.1.7.4
7f55296d7000-7f55296d8000 rw-p 00040000 fc:00 146305                     /usr/lib/x86_64-linux-gnu/libpcap.so.1.7.4
7f55296d8000-7f55296d9000 rw-p 00000000 00:00 0 
7f55296d9000-7f55296dd000 r-xp 00000000 fc:00 16149                      /usr/local/zeek/lib/libbinpac.so.0.54
7f55296dd000-7f55298dc000 ---p 00004000 fc:00 16149                      /usr/local/zeek/lib/libbinpac.so.0.54
7f55298dc000-7f55298dd000 r--p 00003000 fc:00 16149                      /usr/local/zeek/lib/libbinpac.so.0.54
7f55298dd000-7f55298de000 rw-p 00004000 fc:00 16149                      /usr/local/zeek/lib/libbinpac.so.0.54
7f55298de000-7f5529904000 r-xp 00000000 fc:00 488                        /lib/x86_64-linux-gnu/ld-2.23.so
7f5529aef000-7f5529afb000 rw-p 00000000 00:00 0 
7f5529b01000-7f5529b03000 rw-p 00000000 00:00 0 
7f5529b03000-7f5529b04000 r--p 00025000 fc:00 488                        /lib/x86_64-linux-gnu/ld-2.23.so
7f5529b04000-7f5529b05000 rw-p 00026000 fc:00 488                        /lib/x86_64-linux-gnu/ld-2.23.so
7f5529b05000-7f5529b06000 rw-p 00000000 00:00 0 
7ffde5c75000-7ffde5ca6000 rw-p 00000000 00:00 0                          [stack]
7ffde5d78000-7ffde5d7b000 r--p 00000000 00:00 0                          [vvar]
7ffde5d7b000-7ffde5d7d000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)

By repeating this exercise, I was able to narrow it down to a magic max number of 1008 PCAP files. Once I add PCAP file #1009, it starts crashing like this.

Interestingly, I was able to complete this same exercise with 5000+ PCAP files just fine on MacOS without it crashing.

@jsiwek

This comment has been minimized.

Copy link
Member

commented Oct 8, 2019

Looks like a general/known limitation of select() used in Zeek's main loop -- most platforms are going to have a FD_SETSIZE of 1024 (and pcap file descriptors aren't the only ones used in the main select() loop, which is why you're having problems at 1009). We might have a better error message in cases where we approach the limits, but generally the problem ought to be solved by #264 since we'll be using other polling mechanisms that don't have the same limitations as select().

@jsiwek

This comment has been minimized.

Copy link
Member

commented Oct 8, 2019

Or for this particular use-case of just reading many pcaps, we really might not even have to add the FDs in the select() sets and instead always consider them read-ready.

But think it's less likely any changes to help this issue will make it into 3.0.x and instead the overhauled main loop aimed at 3.1 will just happen to fix it.

@philrz

This comment has been minimized.

Copy link
Author

commented Oct 10, 2019

@jsiwek: Thanks for the info. Your response led me to educate myself on FD_SETSIZE a bit. In case anyone else stumbles onto this issue and is looking for an interim workaround, it seems I've found one. While the articles online imply that in Linux FD_SETSIZE is a static, immovable object, in FreeBSD I was able to increase the value in /usr/include/sys/select.h before compiling Zeek and now it's gnawing on my 5000+ PCAPs without complaint. I'll definitely plan to retest this on Linux if/when I see it's been addressed for the general case in Zeek v3.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.