[Python] Creating csv.ParseOptions() causes "Windows fatal exception: access violation" with Visual Studio 2017

[asfimport]

I encountered this when trying to verify the release with MSVC 2017. It may be particular to this machine or build (though it's 100% reproducible for me). I will check the Windows wheels to see if it occurs there, too

(C:\tmp\arrow-verify-release\conda-env) λ python
Python 3.7.3 | packaged by conda-forge | (default, Jul  1 2019, 22:01:29) [MSC v.1900 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow.csv as pc
>>> pc.ParseOptions()

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

Related issues:

• [Python] Default PYARROW_CMAKE_GENERATOR can yield broken libraries with MSVC if the C++ toolchain is different (relates to)

PRs and other links:

• GitHub Pull Request #6350

_{Note: This issue was originally created as ARROW-6757. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Does the debugger tell you something?

[asfimport]

Wes McKinney / @wesm:
I haven't had a chance to investigate further yet. Will report back once I learn something

[asfimport]

Wes McKinney / @wesm:
I was having other problems with my Miniconda – I installed a new Miniconda, re-bootstrapped the dev environment, and then was not able to reproduce

[asfimport]

Wes McKinney / @wesm:
This happened again while trying to verify 0.16.0 RC2. I wish I knew what was wrong

[asfimport]

Wes McKinney / @wesm:
I've been able to reproduce this with a fresh Miniconda using the release verification script:

C:\tmp\arrow-verify-release\conda-env\Library\bin\cmake.exe -DPYTHON_EXECUTABLE=C:\tmp\arrow-verify-release\conda-env\python.exe  -G "Visual Studio 14 2015 Win64" -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=on -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=on -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release C:\tmp\arrow-verify-release\apache-arrow-0.16.0\python
-- Selecting Windows SDK version 10.0.14393.0 to target Windows 10.0.17763.
-- The C compiler identification is MSVC 19.0.24234.1
-- The CXX compiler identification is MSVC 19.0.24234.1

It looks like it could be a conflict between the MSVC (VS 2017) used to build libarrow and the configuration "Visual Studio 14 2015 Win64" being passed by the pyarrow build.

I'm guessing we should not be hardcoding this in setup.py

https://github.com/apache/arrow/blob/master/python/setup.py#L130

I'll see if setting the environment variable PYARROW_CMAKE_GENERATOR fixes the problem

[asfimport]

Antoine Pitrou / @pitrou:
Ha, that could very well be the explanation. In any case, it looks wrong for it to be hard-coded like this.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 6350
#6350