Segmentation fault #3

rookierks · 2015-12-27T15:46:07Z

I'm having trouble with the current release and current git versions.

It runs and displays the state once but then I get a segmentation fault, it seems to be when the screen is about to be refreshed the first time.

I get the following with the current git version:
*** Error in `./iptstate': munmap_chunk(): invalid ======= Backtrace: =========
/usr/lib/libc.so.6(+0x72055)[0x7fc03e3c8055]
/usr/lib/libc.so.6(+0x779a6)[0x7fc03e3cd9a6]
./iptstate[0x406198]
./iptstate[0x403850]
/usr/lib/libc.so.6(__libc_start_main+0xf0)[0x7fc03e37 ./iptstate[0x404a79]
======= Memory map: ========
00400000-00412000 r-xp 00000000 00:28 1450709 00612000-00613000 rw-p 00012000 00:28 1450709 023ca000-04770000 rw-p 00000000 00:00 0 7fc03dd37000-7fc03dd42000 r-xp 00000000 fe:00 1887597 7fc03dd42000-7fc03df41000 ---p 0000b000 fe:00 1887597 7fc03df41000-7fc03df42000 r--p 0000a000 fe:00 1887597 7fc03df42000-7fc03df43000 rw-p 0000b000 fe:00 1887597 7fc03df43000-7fc03df49000 rw-p 00000000 00:00 0
7fc03df49000-7fc03df4d000 r-xp 00000000 fe:00 1886093 7fc03df4d000-7fc03e14d000 ---p 00004000 fe:00 1886093 7fc03e14d000-7fc03e14e000 r--p 00004000 fe:00 1886093 7fc03e14e000-7fc03e14f000 rw-p 00005000 fe:00 1886093 7fc03e14f000-7fc03e155000 r-xp 00000000 fe:00 1886027 7fc03e155000-7fc03e354000 ---p 00006000 fe:00 1886027 7fc03e354000-7fc03e355000 r--p 00005000 fe:00 1886027 7fc03e355000-7fc03e356000 rw-p 00006000 fe:00 1886027 7fc03e356000-7fc03e4f1000 r-xp 00000000 fe:00 1886852 7fc03e4f1000-7fc03e6f0000 ---p 0019b000 fe:00 1886852 7fc03e6f0000-7fc03e6f4000 r--p 0019a000 fe:00 1886852 7fc03e6f4000-7fc03e6f6000 rw-p 0019e000 fe:00 1886852 7fc03e6f6000-7fc03e6fa000 rw-p 00000000 00:00 0
7fc03e6fa000-7fc03e710000 r-xp 00000000 fe:00 1835152 7fc03e710000-7fc03e90f000 ---p 00016000 fe:00 1835152 7fc03e90f000-7fc03e910000 rw-p 00015000 fe:00 1835152 7fc03e910000-7fc03ea0d000 r-xp 00000000 fe:00 1887703 7fc03ea0d000-7fc03ec0c000 ---p 000fd000 fe:00 1887703 7fc03ec0c000-7fc03ec0d000 r--p 000fc000 fe:00 1887703 7fc03ec0d000-7fc03ec0e000 rw-p 000fd000 fe:00 1887703 7fc03ec0e000-7fc03ed80000 r-xp 00000000 fe:00 1835189 7fc03ed80000-7fc03ef80000 ---p 00172000 fe:00 1835189 7fc03ef80000-7fc03ef8a000 r--p 00172000 fe:00 1835189 7fc03ef8a000-7fc03ef8c000 rw-p 0017c000 fe:00 1835189 7fc03ef8c000-7fc03ef90000 rw-p 00000000 00:00 0
7fc03ef90000-7fc03efab000 r-xp 00000000 fe:00 1877341 7fc03efab000-7fc03f1aa000 ---p 0001b000 fe:00 1877341 7fc03f1aa000-7fc03f1ac000 r--p 0001a000 fe:00 1877341 7fc03f1ac000-7fc03f1ad000 rw-p 0001c000 fe:00 1877341 7fc03f1ad000-7fc03f214000 r-xp 00000000 fe:00 1835230 7fc03f214000-7fc03f414000 ---p 00067000 fe:00 1835230 7fc03f414000-7fc03f418000 r--p 00067000 fe:00 1835230 7fc03f418000-7fc03f41a000 rw-p 0006b000 fe:00 1835230 7fc03f41a000-7fc03f43c000 r-xp 00000000 fe:00 1886851 7fc03f5b4000-7fc03f5fc000 rw-p 00000000 00:00 0
7fc03f63a000-7fc03f63b000 rw-p 00000000 00:00 0
7fc03f63b000-7fc03f63c000 r--p 00021000 fe:00 1886851 7fc03f63c000-7fc03f63d000 rw-p 00022000 fe:00 1886851 7fc03f63d000-7fc03f63e000 rw-p 00000000 00:00 0
7ffd1d4a4000-7ffd1d4c5000 rw-p 00000000 00:00 0 7ffd1d524000-7ffd1d526000 r--p 00000000 00:00 0 7ffd1d526000-7ffd1d528000 r-xp 00000000 00:00 0 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 Aborted pointer: 0x000000000474dec0 ***
6610]
/tmpfs/iptstate/iptstate
/tmpfs/iptstate/iptstate
[heap]
/usr/lib/libnss_files-2.22.so
/usr/lib/libnss_files-2.22.so
/usr/lib/libnss_files-2.22.so
/usr/lib/libnss_files-2.22.so
/usr/lib/libmnl.so.0.1.0
/usr/lib/libmnl.so.0.1.0
/usr/lib/libmnl.so.0.1.0
/usr/lib/libmnl.so.0.1.0
/usr/lib/libnfnetlink.so.0.2.0
/usr/lib/libnfnetlink.so.0.2.0
/usr/lib/libnfnetlink.so.0.2.0
/usr/lib/libnfnetlink.so.0.2.0
/usr/lib/libc-2.22.so
/usr/lib/libc-2.22.so
/usr/lib/libc-2.22.so
/usr/lib/libc-2.22.so
/usr/lib/libgcc_s.so.1
/usr/lib/libgcc_s.so.1
/usr/lib/libgcc_s.so.1
/usr/lib/libm-2.22.so
/usr/lib/libm-2.22.so
/usr/lib/libm-2.22.so
/usr/lib/libm-2.22.so
/usr/lib/libstdc++.so.6.0.21
/usr/lib/libstdc++.so.6.0.21
/usr/lib/libstdc++.so.6.0.21
/usr/lib/libstdc++.so.6.0.21
/usr/lib/libnetfilter_conntrack.so.3.5.0
/usr/lib/libnetfilter_conntrack.so.3.5.0
/usr/lib/libnetfilter_conntrack.so.3.5.0
/usr/lib/libnetfilter_conntrack.so.3.5.0
/usr/lib/libncursesw.so.6.0
/usr/lib/libncursesw.so.6.0
/usr/lib/libncursesw.so.6.0
/usr/lib/libncursesw.so.6.0
/usr/lib/ld-2.22.so
/usr/lib/ld-2.22.so
/usr/lib/ld-2.22.so
[stack]
[vvar]
[vdso]
[vsyscall]

More info:
distribution and distribution version: Arch linux fully up-to-date
kernel version: 4.2.5
g++ version: 5.3.0
make version: 4.1
glibc version: 2.22
ncurses version: 6.0
libnetfilter_conntrack version: 1.0.5

jaymzh · 2016-01-02T21:36:01Z

Hmmm I can't repro this. What options are you running it with?

rookierks · 2016-01-02T22:36:20Z

I'm not passing any options, just running it as root.

I'm not very familiar with debugging programs but if there is anything I can do to help track this down I can try it.

I've tried running iptstate with gdb and this is what I got (although no idea of what I'm looking at):

(gdb) backtrace
#0 0x00007ffff6d4a5f8 in raise () from /usr/lib/libc.so.6
#1 0x00007ffff6d4ba7a in abort () from /usr/lib/libc.so.6
#2 0x00007ffff6d8905a in __libc_message () from /usr/lib/libc.so.6
#3 0x00007ffff6d8e9a6 in malloc_printerr () from /usr/lib/libc.so.6
#4 0x00007ffff6d8f18e in _int_free () from /usr/lib/libc.so.6
#5 0x0000000000406198 in __gnu_cxx::new_allocator::deallocate (this=0x44ec790, __p=)
at /usr/include/c++/5.3.0/ext/new_allocator.h:110
#6 __gnu_cxx::__alloc_traitsstd::allocator::deallocate (__a=..., __n=, __p=)
at /usr/include/c++/5.3.0/ext/alloc_traits.h:185
#7 std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_destroy (__size=, this=0x44ec790)
at /usr/include/c++/5.3.0/bits/basic_string.h:185
#8 std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_dispose (this=0x44ec790)
at /usr/include/c++/5.3.0/bits/basic_string.h:180
#9 std::__cxx11::basic_string<char, std::char_traits, std::allocator >::~basic_string (this=0x44ec790,
__in_chrg=) at /usr/include/c++/5.3.0/bits/basic_string.h:544
#10 table_t::~table_t (this=0x44ec6d0, __in_chrg=) at iptstate.cc:138
#11 std::_Destroy<table_t> (__pointer=) at /usr/include/c++/5.3.0/bits/stl_construct.h:93
#12 std::_Destroy_aux::__destroy<table_t*> (__last=, __first=0x44ec6d0)
at /usr/include/c++/5.3.0/bits/stl_construct.h:103
#13 std::_Destroy<table_t*> (__last=, __first=) at /usr/include/c++/5.3.0/bits/stl_construct.h:126
#14 std::_Destroy<table_t*, table_t> (__last=0x44ed8a0, __first=0x44ec6d0) at /usr/include/c++/5.3.0/bits/stl_construct.h:151
#15 std::vector<table_t, std::allocator<table_t> >::_M_erase_at_end (this=0x7fffffffdd60, __pos=0x44ec6d0)
at /usr/include/c++/5.3.0/bits/stl_vector.h:1438
#16 std::vector<table_t, std::allocator<table_t> >::clear (this=0x7fffffffdd60) at /usr/include/c++/5.3.0/bits/stl_vector.h:1212
#17 build_table (flags=..., filters=..., stable=std::vector of length 15, capacity 16 = {...}, counts=..., max=...) at iptstate.cc:1175
#18 0x0000000000403850 in main (argc=, argv=) at iptstate.cc:2389

jbglaw · 2016-01-06T13:59:35Z

I'm seeing this issue as well: Crashes just by starting it. It'll display the initial table of packets/connections once, then crash (probably during updating internal state.)

Reading the error message, that's probably a problem with malloc()/free() or new/delete, and indeed running it with valgrind shows many messages like:

==13911== Invalid free() / delete / delete[] / realloc()
==13911== at 0x4C2964C: operator delete(void_) (vg_replace_malloc.c:480)
==13911== by 0x40D9EB: std::vector<table_t, std::allocator<table_t> >::~vector() (new_allocator.h:110)
==13911== by 0x40403F: main (iptstate.cc:2143)
==13911== Address 0x77a5cf0 is 10,720 bytes inside a block of size 19,456 alloc'd
==13911== at 0x4C2A857: operator new(unsigned long) (vg_replace_malloc.c:298)
==13911== by 0x40E106: std::vector<table_t, std::allocator<table_t> >::_M_insert_aux(_gnu_cxx::normal_iterator<table_t, std::vector<table_t, std::allocator<table_t> > >, table_t const&) (new_allocator.h:104)
==13911== by 0x40D5C7: conntrack_hook(nf_conntrack_msg_type, nf_conntrack, void) (stl_vector.h:925)
==13911== by 0x5285451: __callback (in /usr/lib/x86_64-linux-gnu/libnetfilter_conntrack.so.3.5.0)
==13911== by 0x62DDF6E: ??? (in /usr/lib/x86_64-linux-gnu/libnfnetlink.so.0.2.0)
==13911== by 0x62DE712: nfnl_process (in /usr/lib/x86_64-linux-gnu/libnfnetlink.so.0.2.0)
==13911== by 0x62DEA85: nfnl_catch (in /usr/lib/x86_64-linux-gnu/libnfnetlink.so.0.2.0)
==13911== by 0x528627C: nfct_query (in /usr/lib/x86_64-linux-gnu/libnetfilter_conntrack.so.3.5.0)
==13911== by 0x4062AC: build_table(flags_t&, filters_t const&, std::vector<table_t, std::allocator<table_t> >&, counters_t&, max_t&) (iptstate.cc:1184)
==13911== by 0x403887: main (iptstate.cc:2386)

jaymzh · 2016-01-06T17:35:01Z

Yeah - but we don't do any memory management directly - we literally just use vectors, clear them, push things on, and then re-clear them (for some history on valgrind and iptstate, see https://www.phildev.net/iptstate/memleak.shtml) - you can see in the gdb backtrace that it was literally in clear method on a vector... which makes me wonder if there's a bad gcc version or something out there. @jbglaw - what versions of libc/g++/libstdc++/libnetfilter do you have?

jbglaw · 2016-01-06T18:39:36Z

First of all, I'm not a C++ guru. However, I looked at the sources and (with current git), I wonder about the conntrack_hook() function.

My reading is, that in line 974, an automatic (only on stack!) variable is created:
973 // our table entry
974 table_t entry;

That's being worked on, until the end of conntrack_hook():
1136 /*
1137 * Add this to the array
1138 */
1139 stable->push_back(entry);
1140
1141 return NFCT_CB_CONTINUE;
1142 }

...where entry (which lifecycle ends when conntrack_hook() exits its scope) is pushed into the stable vector. Thus, the vector contains elements of former on-stack variables. I guess fixing iptstate will involve making entry a dynamically allocated object (ie. created with new or malloc as per preference) and pushing that into stable? (As said, I'm into C and from a C perspective, this looks fishy to me...)

jaymzh · 2016-01-06T19:30:02Z

Hmmm... that would explain it... we can certainly try that. It's odd that that would actually work in the vast majority of cases though. I suspect a more-strict version of g++ has made it's way into the world.

jbglaw · 2016-01-06T21:17:56Z

Had another look at it. Using qsort() on a std::vector doesn't look too healthy to me as well.

martynjy · 2016-04-19T19:02:12Z

Arch Linux
linux-lts 4.4.7-1
linux 4.5.1-1
glibc 2.23-1

Same issue. Start it and it shows the table for a few seconds then crashes.
The gdb debug below. I can run 'bt full' if required but I would have to paste it to pastebin because it is very long.

#0 0xb7fdbc11 in __kernel_vsyscall ()
#1 0xb7babc39 in raise () from /usr/lib/libc.so.6
#2 0xb7bad197 in abort () from /usr/lib/libc.so.6
#3 0xb7be6d3d in __libc_message () from /usr/lib/libc.so.6
#4 0xb7bed097 in malloc_printerr () from /usr/lib/libc.so.6
#5 0xb7bed851 in _int_free () from /usr/lib/libc.so.6
#6 0xb7e16d18 in operator delete (ptr=0xbea5898) at /build/gcc/src/gcc-5-20160209/libstdc++-v3/libsupc++/del_op.cc:46
#7 0x0804d854 in __gnu_cxx::new_allocator::deallocate (this=0x90, __p=)
at /usr/include/c++/5.3.0/ext/new_allocator.h:110
#8 __gnu_cxx::__alloc_traitsstd::allocator::deallocate (__a=..., __n=, __p=)
at /usr/include/c++/5.3.0/ext/alloc_traits.h:185
#9 std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_destroy (__size=, this=0x90)
at /usr/include/c++/5.3.0/bits/basic_string.h:185
#10 std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_dispose (this=0x90)
at /usr/include/c++/5.3.0/bits/basic_string.h:180
#11 std::__cxx11::basic_string<char, std::char_traits, std::allocator >::~basic_string (this=0x90, __in_chrg=)
at /usr/include/c++/5.3.0/bits/basic_string.h:543
#12 table_t::~table_t (this=0x0, __in_chrg=) at iptstate.cc:138
#13 std::_Destroy<table_t> (__pointer=) at /usr/include/c++/5.3.0/bits/stl_construct.h:93
#14 std::_Destroy_aux::__destroy<table_t*> (__last=, __first=0x0) at /usr/include/c++/5.3.0/bits/stl_construct.h:103
#15 std::_Destroy<table_t*> (__last=, __first=) at /usr/include/c++/5.3.0/bits/stl_construct.h:126
#16 std::_Destroy<table_t*, table_t> (__last=0xbeabd80, __first=0xbffff5c0) at /usr/include/c++/5.3.0/bits/stl_construct.h:151
#17 std::vector<table_t, std::allocator<table_t> >::_M_erase_at_end (this=0xbe9d418, __pos=0xbffff5c0)
at /usr/include/c++/5.3.0/bits/stl_vector.h:1438
#18 std::vector<table_t, std::allocator<table_t> >::clear (this=0xbe9d418) at /usr/include/c++/5.3.0/bits/stl_vector.h:1212
#19 build_table (flags=..., filters=..., stable=std::vector of length 123, capacity 128 = {...}, counts=..., max=...) at iptstate.cc:1174
#20 0x0804ac40 in main (argc=1, argv=0xbffff9e4) at iptstate.cc:2386

bug · 2016-06-30T19:25:35Z

Same issue, ArchLinux, kernel 4.6.3, glibc 2.23. Starts (without any option), shows the table for half a second and segfaults.

tmberg · 2016-07-19T20:22:17Z

Yepp. Same here. :(

iptstate[20826]: segfault at 0 ip 00007f05dadbd9c4 sp 00007fff17331a68 error 4 in libc-2.23.so[7f05dad45000+197000]

Debian unstable/sid

k0ste · 2016-07-20T14:15:00Z

Same issue on Arch.

jaymzh · 2016-07-22T05:37:43Z

Sorry - I've been traveling nonstop... I'll try to have a look at this soon.

jaymzh · 2016-07-22T07:33:46Z

I pushed a branch called segfault that may fix this. Since I can't repro it, I'm not sure, can one of you try it?

That branch has several broken features: reverse sorting doesn't work, sorting by IP is now much more naive... I'll fix all that before I merge if this works, I just don't want to do that before I know if those worked.

rookierks · 2016-07-22T13:05:43Z

That seems to have fixed the segfault for me.

jaymzh · 2016-07-23T18:53:36Z

Thanks @rookierks - I'll try to clean this up and get it merged soon.

jaymzh · 2016-08-14T06:06:54Z

OK @rookierks - I've pushed another update to the segfault branch which fixes sorting and several other things. Can you test? Also @tmberg @k0ste @bug @martynjy @jbglaw. I think it's ready for merge, but I'd like someone to check who's actually running into this bug.

k0ste · 2016-08-14T06:15:10Z

segfault branch works_for_me

rookierks · 2016-08-14T12:48:20Z

Latest segfault branch code works fine for me.

The refactor for #3 introduced a minor bug that broke `std::sort()` on DstIP sorting. Fixed. Signed-off-by: Phil Dibowitz <phil@ipom.com>

jaymzh · 2016-08-14T22:37:19Z

iptstate 2.2.6 has been released with this fix.

jaymzh self-assigned this Jul 22, 2016

jaymzh closed this as completed in 50a0e17 Aug 14, 2016

jaymzh added a commit that referenced this issue Aug 14, 2016

Fix crashing bug on DstIP sort

22a6641

The refactor for #3 introduced a minor bug that broke `std::sort()` on DstIP sorting. Fixed. Signed-off-by: Phil Dibowitz <phil@ipom.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segmentation fault #3

Segmentation fault #3

rookierks commented Dec 27, 2015

jaymzh commented Jan 2, 2016

rookierks commented Jan 2, 2016

jbglaw commented Jan 6, 2016

jaymzh commented Jan 6, 2016

jbglaw commented Jan 6, 2016

jaymzh commented Jan 6, 2016

jbglaw commented Jan 6, 2016

martynjy commented Apr 19, 2016

bug commented Jun 30, 2016

tmberg commented Jul 19, 2016

k0ste commented Jul 20, 2016

jaymzh commented Jul 22, 2016

jaymzh commented Jul 22, 2016

rookierks commented Jul 22, 2016

jaymzh commented Jul 23, 2016

jaymzh commented Aug 14, 2016

k0ste commented Aug 14, 2016

rookierks commented Aug 14, 2016

jaymzh commented Aug 14, 2016

Segmentation fault #3

Segmentation fault #3

Comments

rookierks commented Dec 27, 2015

jaymzh commented Jan 2, 2016

rookierks commented Jan 2, 2016

jbglaw commented Jan 6, 2016

jaymzh commented Jan 6, 2016

jbglaw commented Jan 6, 2016

jaymzh commented Jan 6, 2016

jbglaw commented Jan 6, 2016

martynjy commented Apr 19, 2016

bug commented Jun 30, 2016

tmberg commented Jul 19, 2016

k0ste commented Jul 20, 2016

jaymzh commented Jul 22, 2016

jaymzh commented Jul 22, 2016

rookierks commented Jul 22, 2016

jaymzh commented Jul 23, 2016

jaymzh commented Aug 14, 2016

k0ste commented Aug 14, 2016

rookierks commented Aug 14, 2016

jaymzh commented Aug 14, 2016