[Python] parquet.read_table causes crashes on Windows Server 2016 w/ Xeon Processor

[asfimport]

Call to read_all() crashes immediately with a windows fatal error on version 0.16.0, 0.17.0, 0.17.1.  Downgrading to 0.15.1 fixes the problem.

Python scripts work fine on all other PCs, but on Server w/ Windows Server 2016 and Xeon processor, it crashes immediately.

ERROR:

Windows fatal exception: code 0xc000001d

Current thread 0x00001950 (most recent call first):
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pyarrow\parquet.py", line 253 in read
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pyarrow\parquet.py", line 605 in read
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pyarrow\parquet.py", line 1137 in read
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pyarrow\parquet.py", line 1281 in read_table
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pycelonis\objects_ibc.py", line 888 in upload_file
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pycelonis\objects_ibc.py", line 938 in upload_df_chunk
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib\parallel.py", line 253 in
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib\parallel.py", line 253 in call
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib_parallel_backends.py", line 572 in init
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib_parallel_backends.py", line 208 in apply_async
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib\parallel.py", line 765 in _dispatch
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib\parallel.py", line 847 in dispatch_one_batch
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib\parallel.py", line 1029 in call
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pycelonis\utils\api_utils.py", line 32 in threaded
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pycelonis\objects_ibc.py", line 944 in
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pycelonis\objects_ibc.py", line 944 in push_table
  File "pythonscript.py", line 152 in main
  File "pythonscript.py", line 171 in
Fatal Python error: Illegal instruction

Current thread 0x00001950 (most recent call first):
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pyarrow\parquet.py", line 253 in read
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pyarrow\parquet.py", line 605 in read
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pyarrow\parquet.py", line 1137 in read
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pyarrow\parquet.py", line 1281 in read_table
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pycelonis\objects_ibc.py", line 888 in upload_file
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pycelonis\objects_ibc.py", line 938 in upload_df_chunk
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib\parallel.py", line 253 in
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib\parallel.py", line 253 in call
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib_parallel_backends.py", line 572 in init
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib_parallel_backends.py", line 208 in apply_async
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib\parallel.py", line 765 in _dispatch
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib\parallel.py", line 847 in dispatch_one_batch
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\joblib\parallel.py", line 1029 in call
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pycelonis\utils\api_utils.py", line 32 in threaded
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pycelonis\objects_ibc.py", line 944 in
  File "D:\XXXXXXXXXXXXXXXXXXXXXXXXX\lib\site-packages\pycelonis\objects_ibc.py", line 944 in push_table
  File "pythonscript.py", line 152 in main
  File "pythonscript.py", line 171 in

Environment: OS Name Microsoft Windows Server 2016 Standard
Version 10.0.14393 Build 14393
Other OS Description Not Available
OS Manufacturer Microsoft Corporation
System Name XXXXXXXXXXXXXXXXXXXXXXXXXXX
System Manufacturer VMware, Inc.
System Model VMware7,1
System Type x64-based PC
System SKU Unsupported
Processor Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz, 2295 Mhz, 1 Core(s), 1 Logical Processor(s)
Processor Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz, 2295 Mhz, 1 Core(s), 1 Logical Processor(s)
Processor Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz, 2295 Mhz, 1 Core(s), 1 Logical Processor(s)
Processor Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz, 2295 Mhz, 1 Core(s), 1 Logical Processor(s)
BIOS Version/Date VMware, Inc. VMW71.00V.0.B64.1704120155, 4/12/2017
SMBIOS Version 2.7
BIOS Mode UEFI
BaseBoard Manufacturer Intel Corporation
BaseBoard Model Not Available
BaseBoard Name Base Board
Platform Role Desktop
Secure Boot State Unsupported
PCR7 Configuration Not Available
Windows Directory C:\Windows
System Directory C:\Windows\system32
Boot Device \Device\HarddiskVolume2
Locale United States
Hardware Abstraction Layer Version = "10.0.14393.3297"
User Name Not Available
Time Zone Eastern Daylight Time
Installed Physical Memory (RAM) 32.0 GB
Total Physical Memory 32.0 GB
Available Physical Memory 29.7 GB
Total Virtual Memory 40.0 GB
Available Virtual Memory 37.9 GB
Page File Space 8.00 GB
Page File C:\pagefile.sys
Device Guard Virtualization based security Not enabled
A hypervisor has been detected. Features required for Hyper-V will not be displayed.
Reporter: Kristopher Jong

_{Note: This issue was originally created as ARROW-9349. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Can you let us know what the processor ID is? This is almost certainly ARROW-7939

[asfimport]

Wes McKinney / @wesm:
Sorry I missed that it's E5-2699. This processor has AVX2 so I don't think it's ARROW-7939. If you can find out any information to determine what is the illegal instruction that is crashing the application that would help us figure out what's wrong

[asfimport]

Wes McKinney / @wesm:
Could you try using a nightly build pip install --extra-index-url https://pypi.fury.io/arrow-nightlies/ --pre pyarrow. That would help us rule out the BMI2 issue

[asfimport]

Kristopher Jong:
I installed pyarrow-0.18.0.dev551 build and it has the same behavior.

[asfimport]

Kristopher Jong:
I can't figure out what the illegal instruction is, it appears to be the point when it switches into the DLL compiled instructions which is why my python script isn't picking up the exception stack.  The original behavior was no errors at all, it just causes the python script to fail with no exceptions.  Once I turned on the faulthandler, I was able to find it was a windows fatal error that was happening causing the failure.

[asfimport]

Wes McKinney / @wesm:
[~mparry] can you offer any guidance to [~kmj1104213] about how to determine what illegal instruction is causing the problem like you did in ARROW-9114?

[asfimport]

Morgan Parry:
We just attached the Visual Studio debugger to the Python process. It trapped the illegal instruction and it was then apparent from the disassembly what the offender was.

I see from the env reported that this is running under VMWare. Note that this can mask CPU features, depending on the software version, configuration, specs of other machines in the cluster (i.e. it may mask to the lowest common denominator), etc. This is exactly what was happening in our case, which took a while to figure out - i.e. the physical CPU had AVX2 but the virtual one didn't.

[asfimport]

Charles Surett:
Can someone make a debug build? I've been attempting to get a build with debug symbols but haven't had any luck.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
[~kmj1104213] do you know if you still run into this issue with the latest pyarrow release?