[C++][Python] installing pyarrow breaks the conda environment

[asfimport]

I am working with pyarrow for parquet and feather IO. My environment has the following requirements python=3.8, pandas, geopandas, pyarrow, matplotlib, seaborn, jupyter

You can reproduce the env using

$ conda create -n testpyarrow python=3.8 pandas geopandas matplotlib seaborn jupyter notebook pyarrow 

$ conda activate testpyarrow

you can skip pyarrow in the above command and then install it separately from conda-forge as instructed in the pyarrow github (doesn't make a difference) page using

$ conda install -c conda-forge pyarrow

After the env is created and activated, opening a python terminal and try importing to import anything

$ python
>>>import pandas

yields

Illegal instruction (core dumped)

If you uninstall pyarrow, all the imports start working. So I have zeroed down that pyarrow is breaking the imports.

As it is core dumped I can't trace the issue further. Illegal instruction is raised irrespective of the library I try to import  (i.e. seaborn, matplotlib, geopandas, pandas, etc)

Environment: Conda version: conda 4.10.3
The issue occurs on RedHat Linux Server running on AMD Opteron Processor 4180. (Works with intel processor)
Reporter: Naman

Related issues:

• [C++] Provide an initialization and/or compatibility check function (is related to)

_{Note: This issue was originally created as ARROW-14526. Please see the migration documentation for further details.}

[asfimport]

Weston Pace / @westonpace:
If you have gdb installed can you figure out a bit more information for us? It should be something like:

gdb --args python -c "import pandas"
...
(gdb) run
...
Program received signal SIGSEGV, Segmentation fault.
...
(gdb) disassemble
...
=> 0x00007ffff7d616a7 <+71>:	???
...

Alternatively, if you can capture and attach a core file that would show the same. A common cause is an attempt to run some kind of extension (e.g. AVX2) that isn't supported by the processor. Generally Arrow should detect this and not make such calls but I don't know if we have an Opteron in our CI infrastructure.

[asfimport]

Naman:
run:

 

Starting program: somewhere/miniconda3/envs/testpyarrow/bin/python -c import\ pandas
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for somewhere/miniconda3/envs/testpyarrow/lib/python3.8/site-packages/mkl/../../../libiomp5.so
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/47/5ceda1e8d10732daed7c831a3e2568c761e6d5.debug
[Detaching after fork from child process 9140]
Program received signal SIGILL, Illegal instruction.
0x00007fffe08f1aeb in arrow::compute::ArraySortOptions::ArraySortOptions(arrow::compute::SortOrder, arrow::compute::NullPlacement) ()
 from somewhere/miniconda3/envs/testpyarrow/lib/python3.8/site-packages/pyarrow/libarrow.so.600
Missing separate debuginfos, use: debuginfo-install glibc-2.17-325.el7_9.x86_64

disassemble:

Dump of assembler code for function _ZN5arrow7compute16ArraySortOptionsC2ENS0_9SortOrderENS0_13NullPlacementE:
 0x00007fffe08f1ae0 <+0>: mov 0x11e3e09(%rip),%rax # 0x7fffe1ad58f0 <_ZN5arrow7compute8internal12_GLOBAL__N_1L21kArraySortOptionsTypeE>
 0x00007fffe08f1ae7 <+7>: movd %esi,%xmm0
=> 0x00007fffe08f1aeb <+11>: pinsrd $0x1,%edx,%xmm0
 0x00007fffe08f1af1 <+17>: mov %rax,0x8(%rdi)
 0x00007fffe08f1af5 <+21>: mov 0x11c9034(%rip),%rax # 0x7fffe1abab30
 0x00007fffe08f1afc <+28>: movq %xmm0,0x10(%rdi)
 0x00007fffe08f1b01 <+33>: add $0x10,%rax
 0x00007fffe08f1b05 <+37>: mov %rax,(%rdi)
 0x00007fffe08f1b08 <+40>: retq
End of assembler dump.

Does this help?

[asfimport]

Weston Pace / @westonpace:
Yes, that's great. pinsrd requires (I think) >= SSE4.1 (although Arrow's threshold is SSE4.2). Opteron 4180 supports only SSE4a.

I don't know enough about conda, packaging, or simd to say what the correct solution / workaround would be.

CC @pitrou @kou

[asfimport]

Antoine Pitrou / @pitrou:
Right, the binaries we publish require SSE4.2. If you compile Arrow C++ and PyArrow from source, you are of course free to choose your own instruction set baseline. However, we also require the presence of a POPCNT instruction, because we use the corresponding intrinsics unconditionally. Luckily, it seems the Opteron 4180 should support it.

We could also provide an initialization function to bail out with a clear error message: ARROW-6033