Numpy crashes during tests with a "Illegal instruction (core dumped)".

Hello,

I've run into a really cryptic problem with numpy's functionality on a virtual machine running Ubuntu 16.04 with ~2GB of RAM (just a tad less) and 100MB of video memory on Python2.7. The virtual machine was done on my laptop and then exported. It was tested on another laptop where it worked. When we tried running it on third laptop it kept crashing. The laptop configuration is:

OS Name	Microsoft Windows 10 Pro
Version	10.0.14393 Build 14393
Other OS Description 	Not Available
OS Manufacturer	Microsoft Corporation
System Name	DESKTOP-N693E6G
System Manufacturer	LENOVO
System Model	80EW
System Type	x64-based PC
System SKU	LENOVO_MT_80EW_BU_idea_FM_Lenovo B50-80
Processor	Intel(R) Core(TM) i3-5005U CPU @ 2.00GHz, 2000 Mhz, 2 Core(s), 4 Logical Processor(s)
BIOS Version/Date	LENOVO A8CN51WW(V3.04), 26.2.2016.
SMBIOS Version	2.7
Embedded Controller Version	3.51
BIOS Mode	Legacy
BaseBoard Manufacturer	LENOVO
BaseBoard Model	Not Available
BaseBoard Name	Base Board
Platform Role	Mobile
Secure Boot State	Unsupported
PCR7 Configuration	Binding Not Possible
Windows Directory	C:\WINDOWS
System Directory	C:\WINDOWS\system32
Boot Device	\Device\HarddiskVolume1
Locale	Hrvatska
Hardware Abstraction Layer	Version = "10.0.14393.1378"
User Name	DESKTOP-N693E6G\Korisnik
Time Zone	Central European Daylight Time
Installed Physical Memory (RAM)	4,00 GB
Total Physical Memory	3,92 GB
Available Physical Memory	1,30 GB
Total Virtual Memory	5,79 GB
Available Virtual Memory	1,95 GB
Page File Space	1,88 GB
Page File	C:\pagefile.sys
Hyper-V - VM Monitor Mode Extensions	Yes
Hyper-V - Second Level Address Translation Extensions	Yes
Hyper-V - Virtualization Enabled in Firmware	No
Hyper-V - Data Execution Protection	Yes

The error manifests itself as an Illegal instruction (core dumped) and the shell restarts, or if you ran the program from the terminal you get back to a clean terminal line. Eventually the error was tracked to a np.where command which seems to reproduce the behavior consistently. I've used the following code to test it:

student@vsa:~/Desktop/PracticeExamples/6_contours_fitObj$ cat test.py
import numpy as np

a = np.arange(0, 100, 1)
np.where(a>90)

When I run it with gdb I get:

student@vsa:~/Desktop/PracticeExamples/6_contours_fitObj$ gdb -ex r --args python test.py
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...(no debugging symbols found)...done.
Starting program: /usr/bin/python test.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
0xb61338a8 in LONG_greater_avx2 (args=args@entry=0xbfffe52c,
    dimensions=dimensions@entry=0xbfffe538, steps=steps@entry=0xbfffe544,
    __NPY_UNUSED_TAGGEDfunc=__NPY_UNUSED_TAGGEDfunc@entry=0x0)
    at numpy/core/src/umath/loops.c.src:936
936    numpy/core/src/umath/loops.c.src: No such file or directory.
(gdb) bt
#0  0xb61338a8 in LONG_greater_avx2 (args=args@entry=0xbfffe52c,
    dimensions=dimensions@entry=0xbfffe538, steps=steps@entry=0xbfffe544,
    __NPY_UNUSED_TAGGEDfunc=__NPY_UNUSED_TAGGEDfunc@entry=0x0)
    at numpy/core/src/umath/loops.c.src:936
#1  0xb617d5a6 in trivial_three_operand_loop (op=op@entry=0xbfffe8f0,
    innerloop=0xb61337ac <LONG_greater_avx2>, innerloopdata=0x0)
    at numpy/core/src/umath/ufunc_object.c:1339
#2  0xb61856d7 in execute_legacy_ufunc_loop (arr_prep_args=0x0, arr_prep=0xbfffe680,
    buffersize=8192, order=NPY_KEEPORDER, dtypes=0xbfffe600, op=0xbfffe8f0,
    trivial_loop_ok=<optimized out>, ufunc=0x84eadb8) at numpy/core/src/umath/ufunc_object.c:1700
#3  PyUFunc_GenericFunction (ufunc=ufunc@entry=0x84eadb8, args=args@entry=0xb7cda78c,
    kwds=kwds@entry=0x0, op=op@entry=0xbfffe8f0) at numpy/core/src/umath/ufunc_object.c:2749
#4  0xb61869c0 in ufunc_generic_call (ufunc=0x84eadb8, args=0xb7cda78c, kwds=0x0)
    at numpy/core/src/umath/ufunc_object.c:4411
#5  0x080f90d9 in PyObject_Call ()
#6  0x081017e9 in PyObject_CallFunctionObjArgs ()
#7  0xb79a2dd7 in PyArray_GenericBinaryFunction (m1=0xb7a73a98, m2=0x8412c3c, op=0x84eadb8)
    at numpy/core/src/multiarray/number.c:260
#8  0x080f4f1c in PyObject_RichCompare ()
#9  0x08111a00 in PyEval_EvalFrameEx ()
#10 0x0810a47b in PyEval_EvalCodeEx ()
#11 0x0810a26e in PyEval_EvalCode ()
#12 0x08139418 in ?? ()
#13 0x08133ea5 in PyRun_FileExFlags ()
#14 0x08132838 in PyRun_SimpleFileExFlags ()
#15 0x08132684 in PyRun_AnyFileExFlags ()
#16 0x080e65bc in Py_Main ()
#17 0x080e5ec6 in main ()
(gdb) p a
No symbol "a" in current context.
(gdb) u

Program terminated with signal SIGILL, Illegal instruction.
The program no longer exists.
(gdb) 

not sure if relevant but numpy also consistently fails to survive the tests on the buss dates:

student@vsa:~/Desktop/PracticeExamples/6_contours_fitObj$ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.13.1'
>>> numpy.__file__
'/usr/local/lib/python2.7/dist-packages/numpy/__init__.pyc'
>>> numpy.test(verbose=10)
Running unit tests for numpy
NumPy version 1.13.1
NumPy relaxed strides checking option: True
NumPy is installed in /usr/local/lib/python2.7/dist-packages/numpy
Python version 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
nose version 1.3.7
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
test_complex (test_abc.ABC) ... ok
test_floats (test_abc.ABC) ... ok
test_int (test_abc.ABC) ... ok
test_uint (test_abc.ABC) ... ok
test_api.test_array_array ... ok
test_api.test_fastCopyAndTranspose ... ok
test_api.test_array_astype ... ok
test_api.test_copyto_fromscalar ... ok
test_api.test_copyto ... ok
test_api.test_copyto_permut ... ok
test_api.test_copy_order ... ok
test_api.test_contiguous_flags ... ok
test_api.test_broadcast_arrays ... ok
Basic test of array2string. ... ok
Test custom format function for each element in array. ... ok
test_structure_format (test_arrayprint.TestArray2String) ... ok
This should only apply to 0-D arrays. See #1218. ... ok
test_arrayprint.TestArrayRepr.test_containing_list ... ok
test_arrayprint.TestArrayRepr.test_nan_inf ... ok
test_arrayprint.TestArrayRepr.test_self_containing ... ok
test_arrayprint.TestArrayRepr.test_subclass ... ok
test_str (test_arrayprint.TestComplexArray) ... ok
test_arrayprint.TestPrintOptions.test_basic ... ok
test_arrayprint.TestPrintOptions.test_formatter ... ok
test_arrayprint.TestPrintOptions.test_formatter_reset ... ok
test_arrayprint.TestPrintOptions.test_precision_zero ... ok
test_arrayprint.test_unicode_object_array ... ok
test_cast_overflow (test_datetime.TestDateTime) ... ok
test_compare_generic_nat (test_datetime.TestDateTime) ... ok
test_creation_overflow (test_datetime.TestDateTime) ... ok
test_datetime_add (test_datetime.TestDateTime) ... ok
test_datetime_arange (test_datetime.TestDateTime) ... ok
test_datetime_arange_no_dtype (test_datetime.TestDateTime) ... ok
test_datetime_array_find_type (test_datetime.TestDateTime) ... ok
test_datetime_array_str (test_datetime.TestDateTime) ... ok
test_datetime_as_string (test_datetime.TestDateTime) ... ok
test_datetime_as_string_timezone (test_datetime.TestDateTime) ... ok
test_datetime_busday_holidays_count (test_datetime.TestDateTime) ... Illegal instruction (core dumped)
student@vsa:~/Desktop/PracticeExamples/6_contours_fitObj$ 

The plan is to use the virtual machines on a summer school to avoid having to setup the whole machine for everyone individually which would include setting up the virtual machine, installing Ubuntu on it, installing numpy, scipy, fitsio and then compiling OpenCV 3.0. Some help would be greatly appreciated.

Thanks!

[juliantaylor]

the virtual machine probably exports cpu capabilities your real cpu does not have.
what is /proc/cpuinfo on the host and the guest?

[juliantaylor]

hm the host is windows? that cpu should have avx2 so maybe the vm is configured to not support it but in a way numpy does not detect (numpy essentially uses cpuid)

The host OS is Win and the guest is Ubuntu (32bit just in case). The host cpu is Processor Intel(R) Core(TM) i3-5005U CPU @ 2.00GHz, 2000 Mhz, 2 Core(s), 4 Logical Processor(s) as stated in the config file and the cpuinfo on that laptop reports:

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 61
model name	: Intel(R) Core(TM) i3-5005U CPU @ 2.00GHz
stepping	: 4
cpu MHz		: 1995.379
cache size	: 3072 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fdiv_bug	: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 rdtscp constant_tsc xtopology nonstop_tsc eagerfpu pni monitor ssse3 sse4_1 sse4_2 x2apic hypervisor lahf_lm
bugs		:
bogomips	: 3990.75
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

Which seems to be in order, that is - it matches. The cpuinfo reported for the laptop on which the virtual machine was created on is:

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 61
model name	: Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz
stepping	: 4
cpu MHz		: 2196.816
cache size	: 3072 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fdiv_bug	: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 20
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 nx rdtscp constant_tsc xtopology nonstop_tsc pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx rdrand hypervisor lahf_lm abm 3dnowprefetch rdseed
bugs		:
bogomips	: 4393.63
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

[juliantaylor]

In the guest avx2 is not listed in the flags, that is a misconfiguration when cpuid still returns it.
Might make sense to put an extra check into numpy for wrongly configured vms

I'm not quite sure I completely understand what that implies. What would be a fix to get the VM working?

EDIT: Numpy was completely uninstalled and then installed via pip on the host machine because I thought the compiled files might not match from my VM to his. It had no effect.

[juliantaylor]

depends on the vm software used, there should be a way to change the cpu features the guest has somewhere in it.

[njsmith]

I wonder how Linux is detecting avx2 if not via cpuid...

I'm doing the fixes over TeamViewer. The laptop doesn't seem to be up currently. I'll install CPU-Z and double check if the flags match the instructions in host and guest. If the flags match - why would numpy be affected by it exactly (especially after reinstalling it)? If the VM was created on a CPU that supports AVX, but then exported and imported on a CPU that doesn't, it shouldn't matter a lot as long as the new CPU is detected correctly. If the VM detected the new environment correctly and registered that the CPU on the new machine does not support AVX - shouldn't reinstalling numpy solve the issue?

I'll get back to you with the results as soon as possible. Thanks for the help.

[juliantaylor]

Reinstalling numpy won't change anything, numpy tries to detect avx2 at runtime via the cpuid instruction.
It is strange that the VM sigills the instruction but still returns essentially wrong results with cpuid. Maybe the used VM software just doesn't support avx2 at all.

[njsmith]

@ljetibo: @juliantaylor's theory is that numpy is like "hey, virtual CPU, do you support AVX2?" and it's like "yep, I sure do!" and then numpy's like "okay, here's some AVX2 instructions to execute!" and the virtual CPU is like "omgwtf are these I never heard of this AVX2 thing". This can happen if the VM hypervisor is buggy. (What hypervisor are you using btw?)

@juliantaylor: I guess one way to eliminate this class of bugs entirely would be to make our AVX2 detection logic be: (1) try to execute some AVX2, (2) if we get SIGILL, then the CPU doesn't support AVX2. Making this actually work would require some gross stuff with signal handlers and longjmp though, plus I don't know, SEH or something on Windows? And we'd get some very confused bug reports by people trying to debug something else using gdb and seeing a SIGILL and freaking out. OTOH, "actually works" is a nice feature.

@njsmith Heh, I understood that much. Thanks for the dumb-down though, I'm really not in familiar territory here.

Ok, So I have the flags that CPU-Z and the flags reporter for the host OS (win) on that laptop are as follows: MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, EM64T, VT-x, AES, AVX, AVX2, FMA3

The VM software used is VirtualBox, so there seems to be an error on the part of the VirtualBox not registering avx2 for that particular cpu. Should I close the issue and ask VirtualBox people?

Just for redundancy and explicitness sake these are the flags reported from the guest OS (Ubuntu) inside the VM:
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 rdtscp constant_tsc xtopology nonstop_tsc eagerfpu pni monitor ssse3 sse4_1 sse4_2 x2apic hypervisor lahf_lm

Thanks for all the help! If there are any suggestions on how to quick-fix the issue - please suggest.

[bashtage]

One solution would be to create your VM on a old CPU that doesn't have AVX2 (or possibly AVX, if you want maximum portability).

[bashtage]

Also, that the Intel(R) Core(TM) i3-5005U CPU @ 2.00GHz laptop doesn't report AVX support, even though it should (http://ark.intel.com/products/84695/Intel-Core-i3-5005U-Processor-3M-Cache-2_00-GHz), suggests that laptop's OS is mis-configured, since OS support is required to use AVX.

Yes, I have posted the issue on virtualbox.org forums if someone hits the same error later on. Maybe it will develop into something, maybe not.

The issue doesn't seem to be that Intel i3 CPU doesn't support AVX or AVX2 (because it does - at least that's what CPU-Z app says in the host OS) it's that the VB doesn't seem to recognize the AVX flag is present for that particular PC when migrated. It's there, but it doesn't exist inside the guest OS.

Reinstalling a new Ubuntu VM might fix, it, or it might not. In the case it does not I'm afraid that there is no solution for the issue. In the lucky accident that it does recognize the host OS CPU flags correctly it's going to be a bit bothersome compiling OpenCV (especially if it runs very slow) and installing the rest of required packages but not impossible.

Unless there's a way to hotfix this mess by i.e. compiling monkeypatched numpy there with special flags? Is that a smaller compile time, is it less CPU intense than OpenCV? I'm open to suggestions like that to get a working copy on everyone's PC rather than risking a complete install of everything for half the PCs there. [I don't currently know how many of the PCs might exhibit the same issues]

[bashtage]

I don't think setting up student PCs using Anaconda or Miniconda is particularly hard. You could easily provide a script that would do all of the work, even pinning packages if you need to.