Memory corruption with testAttackBatch on 32 bit system (i386) with SSE enabled

[rhertzog]

We tried to update pyrit for Kali but the build fails on i386, we can reproduce this with 0.4.0 (patched for new scapy) and 0.5.1 (as released). Here's the log of the problem:

make[1]: Entering directory '/<<BUILDDIR>>/pyrit-0.5.1+git20160815'
PYBUILD_SYSTEM=custom \
PYBUILD_TEST_ARGS="cd /<<BUILDDIR>>/pyrit-0.5.1+git20160815/test/ && {interpreter} test_pyrit.py" \
PYRIT_CONFIG_FILE="/<<BUILDDIR>>/pyrit-0.5.1+git20160815/debian/config.build" \
dh_auto_test
I: pybuild base:184: cd /<<BUILDDIR>>/pyrit-0.5.1+git20160815/test/ && python2.7 test_pyrit.py
testAnalyze (__main__.FilesystemTestCase) ... ok
testAttackBatch (__main__.FilesystemTestCase) ... *** Error in `python2.7': malloc(): memory corruption: 0xf999e1c0 ***
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(+0x6737a)[0xf714c37a]
/lib/i386-linux-gnu/libc.so.6(+0x6dfb7)[0xf7152fb7]
/lib/i386-linux-gnu/libc.so.6(+0x6fe52)[0xf7154e52]
/lib/i386-linux-gnu/libc.so.6(__libc_malloc+0xc5)[0xf7156a25]
python2.7(PyMem_Malloc+0x26)[0xf741f416]
/<<BUILDDIR>>/pyrit-0.5.1+git20160815/.pybuild/pythonX.Y_2.7/build/cpyrit/_cpyrit_cpu.so(+0x3d30)[0xf6be4d30]
/<<BUILDDIR>>/pyrit-0.5.1+git20160815/.pybuild/pythonX.Y_2.7/build/cpyrit/_cpyrit_cpu.so(+0x37fe)[0xf6be47fe]
python2.7(+0x14c599)[0xf74ad599]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(PyEval_CallObjectWithKeywords+0x40)[0xf7441600]
python2.7(+0x184201)[0xf74e5201]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(PyEval_EvalFrameEx+0x5c71)[0xf743d101]
python2.7(PyEval_EvalCodeEx+0x1fb)[0xf7435cfb]
python2.7(+0xf0e02)[0xf7451e02]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(+0x1076d5)[0xf74686d5]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(+0x106e9e)[0xf7467e9e]
python2.7(+0xc9477)[0xf742a477]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(PyEval_EvalFrameEx+0x5c71)[0xf743d101]
python2.7(PyEval_EvalCodeEx+0x1fb)[0xf7435cfb]
python2.7(+0xf0e02)[0xf7451e02]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(+0x1076d5)[0xf74686d5]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(+0x106e9e)[0xf7467e9e]
python2.7(+0xc9477)[0xf742a477]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(PyEval_EvalFrameEx+0x5c71)[0xf743d101]
python2.7(PyEval_EvalCodeEx+0x1fb)[0xf7435cfb]
python2.7(+0xf0fee)[0xf7451fee]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(PyEval_EvalFrameEx+0x2db3)[0xf743a243]
python2.7(PyEval_EvalCodeEx+0x1fb)[0xf7435cfb]
python2.7(PyEval_EvalFrameEx+0x6310)[0xf743d7a0]
python2.7(PyEval_EvalCodeEx+0x1fb)[0xf7435cfb]
python2.7(+0xf0fee)[0xf7451fee]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(PyEval_EvalFrameEx+0x2db3)[0xf743a243]
python2.7(PyEval_EvalCodeEx+0x1fb)[0xf7435cfb]
python2.7(PyEval_EvalFrameEx+0x6310)[0xf743d7a0]
python2.7(PyEval_EvalCodeEx+0x1fb)[0xf7435cfb]
python2.7(+0xf0fee)[0xf7451fee]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(PyEval_EvalFrameEx+0x2db3)[0xf743a243]
python2.7(PyEval_EvalCodeEx+0x1fb)[0xf7435cfb]
python2.7(+0xf0e02)[0xf7451e02]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(+0x1076d5)[0xf74686d5]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(+0x166ede)[0xf74c7ede]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(PyEval_EvalFrameEx+0x5c71)[0xf743d101]
python2.7(PyEval_EvalCodeEx+0x1fb)[0xf7435cfb]
python2.7(+0xf0fee)[0xf7451fee]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(PyEval_EvalFrameEx+0x2db3)[0xf743a243]
python2.7(PyEval_EvalCodeEx+0x1fb)[0xf7435cfb]
python2.7(+0xf0e02)[0xf7451e02]
python2.7(PyObject_Call+0x49)[0xf7424f59]
python2.7(+0x1076d5)[0xf74686d5]
======= Memory map: ========
f5800000-f5821000 rw-p 00000000 00:00 0
f5821000-f5900000 ---p 00000000 00:00 0
f5a00000-f5a21000 rw-p 00000000 00:00 0
f5a21000-f5b00000 ---p 00000000 00:00 0
f5b3c000-f5b3d000 ---p 00000000 00:00 0
f5b3d000-f657d000 rw-p 00000000 00:00 0                                  [stack:6772]
f6597000-f65b9000 r-xp 00000000 00:22 2600                               /usr/lib/python2.7/lib-dynload/_ctypes.i386-linux-gnu.so
f65b9000-f65ba000 r--p 00021000 00:22 2600                               /usr/lib/python2.7/lib-dynload/_ctypes.i386-linux-gnu.so
f65ba000-f65bd000 rw-p 00022000 00:22 2600                               /usr/lib/python2.7/lib-dynload/_ctypes.i386-linux-gnu.so
f65bd000-f65fd000 rw-p 00000000 00:00 0
f65fd000-f6640000 r-xp 00000000 00:22 6307                               /usr/lib/i386-linux-gnu/libpcap.so.1.8.1
f6640000-f6641000 ---p 00043000 00:22 6307                               /usr/lib/i386-linux-gnu/libpcap.so.1.8.1
f6641000-f6642000 r--p 00043000 00:22 6307                               /usr/lib/i386-linux-gnu/libpcap.so.1.8.1
 f6642000-f6643000 rw-p 00044000 00:22 6307                               /usr/lib/i386-linux-gnu/libpcap.so.1.8.1
f6643000-f6a43000 rw-p 00000000 00:00 0
f6a49000-f6a65000 r-xp 00000000 00:22 208                                /lib/i386-linux-gnu/libgcc_s.so.1
f6a65000-f6a66000 r--p 0001b000 00:22 208                                /lib/i386-linux-gnu/libgcc_s.so.1
f6a66000-f6a67000 rw-p 0001c000 00:22 208                                /lib/i386-linux-gnu/libgcc_s.so.1
f6a67000-f6b27000 rw-p 00000000 00:00 0
f6b27000-f6b4d000 r-xp 00000000 00:22 2288                               /lib/i386-linux-gnu/libexpat.so.1.6.2
f6b4d000-f6b4e000 ---p 00026000 00:22 2288                               /lib/i386-linux-gnu/libexpat.so.1.6.2
f6b4e000-f6b50000 r--p 00026000 00:22 2288                               /lib/i386-linux-gnu/libexpat.so.1.6.2
f6b50000-f6b51000 rw-p 00028000 00:22 2288                               /lib/i386-linux-gnu/libexpat.so.1.6.2
f6b51000-f6bd1000 rw-p 00000000 00:00 0
f6bdb000-f6bdf000 r-xp 00000000 00:22 878                                /lib/i386-linux-gnu/libuuid.so.1.3.0
f6bdf000-f6be0000 r--p 00003000 00:22 878                                /lib/i386-linux-gnu/libuuid.so.1.3.0
f6be0000-f6be1000 rw-p 00004000 00:22 878                                /lib/i386-linux-gnu/libuuid.so.1.3.0
f6be1000-f6bed000 r-xp 00000000 09:02 18874672                           /<<BUILDDIR>>/pyrit-0.5.1+git20160815/.pybuild/pythonX.Y_2.7/build/cpyrit/_cpyrit_cpu.so
f6bed000-f6bee000 r--p 0000b000 09:02 18874672                           /<<BUILDDIR>>/pyrit-0.5.1+git20160815/.pybuild/pythonX.Y_2.7/build/cpyrit/_cpyrit_cpu.so
f6bee000-f6bef000 rw-p 0000c000 09:02 18874672                           /<<BUILDDIR>>/pyrit-0.5.1+git20160815/.pybuild/pythonX.Y_2.7/build/cpyrit/_cpyrit_cpu.so
f6bef000-f6c91000 rw-p 00000000 00:00 0
f6c91000-f6ed9000 r-xp 00000000 00:22 2380                               /usr/lib/i386-linux-gnu/libcrypto.so.1.1
f6ed9000-f6eda000 ---p 00248000 00:22 2380                               /usr/lib/i386-linux-gnu/libcrypto.so.1.1
f6eda000-f6eeb000 r--p 00248000 00:22 2380                               /usr/lib/i386-linux-gnu/libcrypto.so.1.1
f6eeb000-f6ef2000 rw-p 00259000 00:22 2380                               /usr/lib/i386-linux-gnu/libcrypto.so.1.1
f6ef2000-f6ef5000 rw-p 00000000 00:00 0
f6ef5000-f6f5b000 r-xp 00000000 00:22 2381                               /usr/lib/i386-linux-gnu/libssl.so.1.1
f6f5b000-f6f5c000 ---p 00066000 00:22 2381                               /usr/lib/i386-linux-gnu/libssl.so.1.1
f6f5c000-f6f5f000 r--p 00066000 00:22 2381                               /usr/lib/i386-linux-gnu/libssl.so.1.1
f6f5f000-f6f63000 rw-p 00069000 00:22 2381                               /usr/lib/i386-linux-gnu/libssl.so.1.1
f6f63000-f6fa3000 rw-p 00000000 00:00 0
f6fa4000-f6fab000 r-xp 00000000 00:22 2304                               /usr/lib/i386-linux-gnu/libffi.so.6.0.4
f6fab000-f6fac000 r--p 00006000 00:22 2304                               /usr/lib/i386-linux-gnu/libffi.so.6.0.4
f6fac000-f6fad000 rw-p 00007000 00:22 2304                               /usr/lib/i386-linux-gnu/libffi.so.6.0.4
f6fad000-f6fbf000 r-xp 00000000 00:22 2627                               /usr/lib/python2.7/lib-dynload/pyexpat.i386-linux-gnu.so
f6fbf000-f6fc0000 r--p 00011000 00:22 2627                               /usr/lib/python2.7/lib-dynload/pyexpat.i386-linux-gnu.so
f6fc0000-f6fc1000 rw-p 00012000 00:22 2627                               /usr/lib/python2.7/lib-dynload/pyexpat.i386-linux-gnu.so
f6fc1000-f6fc2000 rw-p 00000000 00:00 0
f6fc2000-f6fd8000 r-xp 00000000 00:22 2612                               /usr/lib/python2.7/lib-dynload/_ssl.i386-linux-gnu.so
f6fd8000-f6fd9000 r--p 00015000 00:22 2612                               /usr/lib/python2.7/lib-dynload/_ssl.i386-linux-gnu.so
f6fd9000-f6fdc000 rw-p 00016000 00:22 2612                               /usr/lib/python2.7/lib-dynload/_ssl.i386-linux-gnu.so
f6fdc000-f6fe1000 r-xp 00000000 00:22 2605                               /usr/lib/python2.7/lib-dynload/_hashlib.i386-linux-gnu.so
f6fe1000-f6fe2000 r--p 00004000 00:22 2605                               /usr/lib/python2.7/lib-dynload/_hashlib.i386-linux-gnu.so
f6fe2000-f6fe3000 rw-p 00005000 00:22 2605                               /usr/lib/python2.7/lib-dynload/_hashlib.i386-linux-gnu.so
f6fe3000-f70e5000 rw-p 00000000 00:00 0
f70e5000-f7296000 r-xp 00000000 00:22 37                                 /lib/i386-linux-gnu/libc-2.24.so
f7296000-f7298000 r--p 001b0000 00:22 37                                 /lib/i386-linux-gnu/libc-2.24.so
f7298000-f7299000 rw-p 001b2000 00:22 37                                 /lib/i386-linux-gnu/libc-2.24.so
f7299000-f729c000 rw-p 00000000 00:00 0
f729c000-f72ef000 r-xp 00000000 00:22 65                                 /lib/i386-linux-gnu/libm-2.24.so
f72ef000-f72f0000 r--p 00052000 00:22 65                                 /lib/i386-linux-gnu/libm-2.24.so
f72f0000-f72f1000 rw-p 00053000 00:22 65                                 /lib/i386-linux-gnu/libm-2.24.so
f72f1000-f730a000 r-xp 00000000 00:22 212                                /lib/i386-linux-gnu/libz.so.1.2.8
f730a000-f730b000 r--p 00018000 00:22 212                                /lib/i386-linux-gnu/libz.so.1.2.8
f730b000-f730c000 rw-p 00019000 00:22 212                                /lib/i386-linux-gnu/libz.so.1.2.8
f730c000-f730e000 r-xp 00000000 00:22 873                                /lib/i386-linux-gnu/libutil-2.24.so
f730e000-f730f000 r--p 00001000 00:22 873                                /lib/i386-linux-gnu/libutil-2.24.so
f730f000-f7310000 rw-p 00002000 00:22 873                                /lib/i386-linux-gnu/libutil-2.24.so
f7310000-f7313000 r-xp 00000000 00:22 63                                 /lib/i386-linux-gnu/libdl-2.24.so
f7313000-f7314000 r--p 00002000 00:22 63                                 /lib/i386-linux-gnu/libdl-2.24.so
f7314000-f7315000 rw-p 00003000 00:22 63                                 /lib/i386-linux-gnu/libdl-2.24.so
f7315000-f732e000 r-xp 00000000 00:22 67                                 /lib/i386-linux-gnu/libpthread-2.24.so
f732e000-f732f000 r--p 00018000 00:22 67                                 /lib/i386-linux-gnu/libpthread-2.24.so
f732f000-f7330000 rw-p 00019000 00:22 67                                 /lib/i386-linux-gnu/libpthread-2.24.so
f7330000-f7332000 rw-p 00000000 00:00 0
f7336000-f7339000 rw-p 00000000 00:00 0
f7339000-f733a000 r-xp 00000000 00:00 0                                  [vdso]
f733a000-f733c000 r--p 00000000 00:00 0                                  [vvar]
f733c000-f735e000 r-xp 00000000 00:22 34                                 /lib/i386-linux-gnu/ld-2.24.so
f735e000-f735f000 rw-p 00000000 00:00 0
f735f000-f7360000 r--p 00022000 00:22 34                                 /lib/i386-linux-gnu/ld-2.24.so
f7360000-f7361000 rw-p 00023000 00:22 34                                 /lib/i386-linux-gnu/ld-2.24.so
f7361000-f76bc000 r-xp 00000000 00:22 2205                               /usr/bin/python2.7
f76bc000-f76bd000 rwxp 00000000 00:00 0
f76bd000-f76be000 r--p 0035b000 00:22 2205                               /usr/bin/python2.7
f76be000-f771e000 rw-p 0035c000 00:22 2205                               /usr/bin/python2.7
f771e000-f7733000 rw-p 00000000 00:00 0
f95ca000-f99ea000 rw-p 00000000 00:00 0                                  [heap]
ffb64000-ffb85000 rw-p 00000000 00:00 0                                  [stack]
Testing with filesystem-storage...
Aborted
E: pybuild pybuild:276: test: plugin custom failed with: exit code=134: cd /<<BUILDDIR>>/pyrit-0.5.1+git20160815/test/ && python2.7 test_pyrit.py
dh_auto_test: pybuild --test -i python{version} -p 2.7 returned exit code 13
debian/rules:25: recipe for target 'override_dh_auto_test' failed

This only happens when SSE2 is enabled. When I build with a cpyrit/_cpyrit_cpu.h edited to not define COMPILE_PADLOCK and COMPILE_SSE2, then the problem goes away and the test succeeds.

This also seems to be processor specific as I can't reproduce the problem on my laptop but I can reproduce it on the build server.

My laptop CPU is:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 78
model name      : Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
stepping        : 3
microcode       : 0x9e
cpu MHz         : 500.048
cache size      : 4096 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs            :
bogomips        : 5616.00
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

The build server CPU is:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 58
model name      : Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz
stepping        : 9
microcode       : 0x15
cpu MHz         : 1605.437
cache size      : 8192 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips        : 6784.85
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

In both cases, the build environment is mostly identical as the build happens in a minimal chroot with only the latest version of each required dependency available.

The full build log is here if you want to check it:
http://buildd-amd64.kali.org/build-logs/pyrit_0.5.1+git20160815-0kali1-kali-experimental-i386-20170110-150459.3114.log

[sandrotosi]

hello,
this bug is currently preventing pyrit (as of version 0.4.0) to be released with the upcoming debian stable release - could you have a look at this somehow urgently? let us know if there is anything Debian can do to help you investigating and fixing this bug

thanks!

[LocutusOfBorg]

Seems to be working now after a rebuild