ARROW-18339: [Python][CI] Add DYLD_LIBRARY_PATH to avoid requiring PYARROW_BUNDLE_ARROW_CPP on macOS job #14643

raulcd · 2022-11-15T10:14:35Z

No description provided.

…ARROW_CPP on macOS job

raulcd · 2022-11-15T13:53:20Z

@kou @AlenkaF I tested removing the skip for the cython test and also adding to the test the following, as seen on the intermediate commits, but did not solve the issue for the cython test, that's why I opted to continue skipping it:

        elif sys.platform == 'darwin':
            delim, var = ':', 'LD_LIBRARY_PATH'
            subprocess_env['DYLD_LIBRARY_PATH'] = delim.join(
                pa.get_library_dirs() + [
                    subprocess_env.get('DYLD_LIBRARY_PATH', '')
                ]
            )

Let me know if you want me to create a JIRA for this change or if you are ok with it being MINOR

AlenkaF · 2022-11-15T17:59:48Z

I agree with skipping the test for now. We can change that when XCode gets upgraded on the GitHub runner.
Regarding the change, not sure where is the line for the MINOR label 🤷‍♀️

kou · 2022-11-15T21:52:37Z

Could you open a Jira issue because this pull request needs more changes?

The following log shows that import _json in the json library imports pyarrow's _json not the json library's one:

https://github.com/apache/arrow/actions/runs/3470117791/jobs/5797917201#step:6:481

  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/scanner.py", line 5, in <module>
    from _json import make_scanner as c_make_scanner
  File "pyarrow/_json.pyx", line 1, in init pyarrow._json

I think that we can fix it by preparing a proper path. I'll take a look at this on my local M1 Mac. Please wait for a while.

github-actions · 2022-11-16T08:16:57Z

https://issues.apache.org/jira/browse/ARROW-18339

raulcd

Thanks for the fix @kou ! LGTM

jorisvandenbossche · 2022-11-16T12:16:54Z

python/pyarrow/tests/test_cython.py

+        if path_var:
+            paths = sys.path
+            paths += pa.get_library_dirs()
+            paths += [test_ld_path]


This path is probably not needed? (it seems we are not using PYARROW_TEST_LD_PATH anywhere)

I needed to use PYARROW_TEST_LD_PATH to reproduce this problem on my local M1 Mac.
(I did it without reading documentations. So my approach may not be a recommended approach. Sorry.)

If PYARROW_TEST_LD_PATH isn't recommended by pyarrow developers, I'm OK with removing it.

I have no idea about PYARROW_TEST_LD_PATH or what it is supposed to do. I just notices that it has not a single mention in our repo outside of the line above test_ld_path = os.environ.get('PYARROW_TEST_LD_PATH', ''). So therefore I was assuming this is some old left-over.

I needed to use PYARROW_TEST_LD_PATH to reproduce this problem on my local M1 Mac.

Can you clarify how you "used" it? I suppose you mean setting it manually to some path, but which?

Sure:

$ rm -rf cpp.build $ make -S cpp -B cpp.build -G "Ninja" -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DCMAKE_INSTALL_PREFIX=/tmp/local -DCMAKE_BUILD_TYPE=Debug -DARROW_FILESYSTEM=ON -DARROW_GANDIVA=ON -DARROW_COMPUTE=ON -DARROW_CSV=ON -DARROW_DATASET=ON -DARROW_ORC=ON -DARROW_JSON=ON $ cmake --build cpp.build --config Debug --target install $ MACOSX_DEPLOYMENT_TARGET=13.0 CMAKE_PREFIX_PATH=/tmp/local CMAKE_BUILD_PARALLEL_LEVEL=$(sysctl -n hw.ncpu) python3 -m pip install -vv ./python $ MACOSX_DEPLOYMENT_TARGET=13.0 PYARROW_TEST_LD_PATH=/tmp/local/lib DYLD_LIBRARY_PATH=/tmp/local/lib python3 -m pytest -r s -k 'test_cython_api' --pyargs pyarrow

If I don't set PYARROW_TEST_LD_PATH, the following error is reported:

clang++ -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -g build/temp.macosx-10.9-universal2-cpython-310/pyarrow_cython_example.o -L/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyarrow -larrow -larrow_python -o build/lib.macosx-10.9-universal2-cpython-310/pyarrow_cython_example.cpython-310-darwin.so ld: library not found for -larrow

================================================ test session starts ================================================= platform darwin -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 rootdir: /Users/kou/work/cpp/arrow plugins: hypothesis-6.57.1, lazy-fixture-0.6.3 collected 4447 items / 4446 deselected / 3 skipped / 1 selected tests/test_cython.py F [100%] ====================================================== FAILURES ====================================================== __________________________________________________ test_cython_api ___________________________________________________ tmpdir = local('/private/var/folders/6b/s397zyc57g9gvqm8cj6gzq0r0000gn/T/pytest-of-kou/pytest-3/test_cython_api0') @pytest.mark.cython def test_cython_api(tmpdir): """ Basic test for the Cython API. """ # Fail early if cython is not found import cython # noqa with tmpdir.as_cwd(): # Set up temporary workspace pyx_file = 'pyarrow_cython_example.pyx' shutil.copyfile(os.path.join(here, pyx_file), os.path.join(str(tmpdir), pyx_file)) # Create setup.py file setup_code = setup_template.format(pyx_file=pyx_file, compiler_opts=compiler_opts, test_ld_path=test_ld_path) with open('setup.py', 'w') as f: f.write(setup_code) # ARROW-2263: Make environment with this pyarrow/ package first on the # PYTHONPATH, for local dev environments subprocess_env = test_util.get_modified_env_with_pythonpath() # Compile extension module > subprocess.check_call([sys.executable, 'setup.py', 'build_ext', '--inplace'], env=subprocess_env) /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyarrow/tests/test_cython.py:109: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ popenargs = (['/Library/Frameworks/Python.framework/Versions/3.10/bin/python3', 'setup.py', 'build_ext', '--inplace'],) kwargs = {'env': {'AWS_CONFIG_FILE': '/dev/null', 'AWS_EC2_METADATA_DISABLED': 'true', 'DYLD_LIBRARY_PATH': '/tmp/local/lib', 'EDITOR': 'vim', ...}} retcode = 1 cmd = ['/Library/Frameworks/Python.framework/Versions/3.10/bin/python3', 'setup.py', 'build_ext', '--inplace'] def check_call(*popenargs, **kwargs): """Run command with arguments. Wait for command to complete. If the exit code was zero then return, otherwise raise CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute. The arguments are the same as for the call function. Example: check_call(["ls", "-l"]) """ retcode = call(*popenargs, **kwargs) if retcode: cmd = kwargs.get("args") if cmd is None: cmd = popenargs[0] > raise CalledProcessError(retcode, cmd) E subprocess.CalledProcessError: Command '['/Library/Frameworks/Python.framework/Versions/3.10/bin/python3', 'setup.py', 'build_ext', '--inplace']' returned non-zero exit status 1. /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py:369: CalledProcessError ------------------------------------------------ Captured stdout call ------------------------------------------------ Compiling pyarrow_cython_example.pyx because it changed. [1/1] Cythonizing pyarrow_cython_example.pyx Extension module: <setuptools.extension.Extension('pyarrow_cython_example') at 0x119c86620> ['/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/core/include', '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyarrow/include'] ['arrow', 'arrow_python'] ['/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyarrow'] running build_ext building 'pyarrow_cython_example' extension creating build creating build/temp.macosx-10.9-universal2-cpython-310 clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/numpy/core/include -I/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyarrow/include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c pyarrow_cython_example.cpp -o build/temp.macosx-10.9-universal2-cpython-310/pyarrow_cython_example.o -std=c++17 creating build/lib.macosx-10.9-universal2-cpython-310 clang++ -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -g build/temp.macosx-10.9-universal2-cpython-310/pyarrow_cython_example.o -L/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyarrow -larrow -larrow_python -o build/lib.macosx-10.9-universal2-cpython-310/pyarrow_cython_example.cpython-310-darwin.so ------------------------------------------------ Captured stderr call ------------------------------------------------ pyarrow_cython_example.cpp:3756:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_buffer' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_buffer)(std::shared_ptr< arrow::Buffer> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3757:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_resizable_buffer' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_resizable_buffer)(std::shared_ptr< arrow::ResizableBuffer> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3758:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_data_type' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_data_type)(std::shared_ptr< arrow::DataType> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3759:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_field' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_field)(std::shared_ptr< arrow::Field> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3760:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_schema' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_schema)(std::shared_ptr< arrow::Schema> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3763:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_chunked_array' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_chunked_array)(std::shared_ptr< arrow::ChunkedArray> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3764:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_coo_tensor' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_coo_tensor)(std::shared_ptr< arrow::SparseCOOTensor> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3765:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_csc_matrix' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_csc_matrix)(std::shared_ptr< arrow::SparseCSCMatrix> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3766:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_csf_tensor' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_csf_tensor)(std::shared_ptr< arrow::SparseCSFTensor> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3767:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_csr_matrix' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_csr_matrix)(std::shared_ptr< arrow::SparseCSRMatrix> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3768:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_tensor' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_tensor)(std::shared_ptr< arrow::Tensor> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3769:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_batch' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_batch)(std::shared_ptr< arrow::RecordBatch> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3770:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_table' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_table)(std::shared_ptr< arrow::Table> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3771:43: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_buffer' [-Wunused-variable] static std::shared_ptr< arrow::Buffer> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_buffer)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3773:42: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_field' [-Wunused-variable] static std::shared_ptr< arrow::Field> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_field)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3774:43: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_schema' [-Wunused-variable] static std::shared_ptr< arrow::Schema> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_schema)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3777:49: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_chunked_array' [-Wunused-variable] static std::shared_ptr< arrow::ChunkedArray> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_chunked_array)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3778:52: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_coo_tensor' [-Wunused-variable] static std::shared_ptr< arrow::SparseCOOTensor> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_coo_tensor)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3779:52: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_csc_matrix' [-Wunused-variable] static std::shared_ptr< arrow::SparseCSCMatrix> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_csc_matrix)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3780:52: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_csf_tensor' [-Wunused-variable] static std::shared_ptr< arrow::SparseCSFTensor> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_csf_tensor)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3781:52: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_csr_matrix' [-Wunused-variable] static std::shared_ptr< arrow::SparseCSRMatrix> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_csr_matrix)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3782:43: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_tensor' [-Wunused-variable] static std::shared_ptr< arrow::Tensor> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_tensor)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3783:48: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_batch' [-Wunused-variable] static std::shared_ptr< arrow::RecordBatch> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_batch)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3784:42: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_table' [-Wunused-variable] static std::shared_ptr< arrow::Table> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_table)(PyObject *); /*proto*/ ^ 24 warnings generated. pyarrow_cython_example.cpp:3756:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_buffer' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_buffer)(std::shared_ptr< arrow::Buffer> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3757:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_resizable_buffer' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_resizable_buffer)(std::shared_ptr< arrow::ResizableBuffer> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3758:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_data_type' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_data_type)(std::shared_ptr< arrow::DataType> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3759:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_field' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_field)(std::shared_ptr< arrow::Field> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3760:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_schema' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_schema)(std::shared_ptr< arrow::Schema> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3763:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_chunked_array' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_chunked_array)(std::shared_ptr< arrow::ChunkedArray> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3764:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_coo_tensor' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_coo_tensor)(std::shared_ptr< arrow::SparseCOOTensor> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3765:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_csc_matrix' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_csc_matrix)(std::shared_ptr< arrow::SparseCSCMatrix> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3766:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_csf_tensor' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_csf_tensor)(std::shared_ptr< arrow::SparseCSFTensor> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3767:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_csr_matrix' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_sparse_csr_matrix)(std::shared_ptr< arrow::SparseCSRMatrix> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3768:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_tensor' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_tensor)(std::shared_ptr< arrow::Tensor> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3769:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_batch' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_batch)(std::shared_ptr< arrow::RecordBatch> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3770:20: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_wrap_table' [-Wunused-variable] static PyObject *(*__pyx_f_7pyarrow_3lib_pyarrow_wrap_table)(std::shared_ptr< arrow::Table> const &); /*proto*/ ^ pyarrow_cython_example.cpp:3771:43: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_buffer' [-Wunused-variable] static std::shared_ptr< arrow::Buffer> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_buffer)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3773:42: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_field' [-Wunused-variable] static std::shared_ptr< arrow::Field> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_field)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3774:43: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_schema' [-Wunused-variable] static std::shared_ptr< arrow::Schema> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_schema)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3777:49: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_chunked_array' [-Wunused-variable] static std::shared_ptr< arrow::ChunkedArray> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_chunked_array)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3778:52: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_coo_tensor' [-Wunused-variable] static std::shared_ptr< arrow::SparseCOOTensor> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_coo_tensor)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3779:52: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_csc_matrix' [-Wunused-variable] static std::shared_ptr< arrow::SparseCSCMatrix> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_csc_matrix)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3780:52: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_csf_tensor' [-Wunused-variable] static std::shared_ptr< arrow::SparseCSFTensor> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_csf_tensor)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3781:52: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_csr_matrix' [-Wunused-variable] static std::shared_ptr< arrow::SparseCSRMatrix> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_sparse_csr_matrix)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3782:43: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_tensor' [-Wunused-variable] static std::shared_ptr< arrow::Tensor> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_tensor)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3783:48: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_batch' [-Wunused-variable] static std::shared_ptr< arrow::RecordBatch> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_batch)(PyObject *); /*proto*/ ^ pyarrow_cython_example.cpp:3784:42: warning: unused variable '__pyx_f_7pyarrow_3lib_pyarrow_unwrap_table' [-Wunused-variable] static std::shared_ptr< arrow::Table> (*__pyx_f_7pyarrow_3lib_pyarrow_unwrap_table)(PyObject *); /*proto*/ ^ 24 warnings generated. ld: library not found for -larrow clang: error: linker command failed with exit code 1 (use -v to see invocation) error: command '/usr/bin/clang++' failed with exit code 1 ============================================== short test summary info =============================================== SKIPPED [1] ../../../../../Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyarrow/tests/test_cuda.py:32: could not import 'pyarrow.cuda': No module named 'pyarrow._cuda' SKIPPED [1] ../../../../../Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyarrow/tests/test_cuda_numba_interop.py:23: could not import 'pyarrow.cuda': No module named 'pyarrow._cuda' SKIPPED [1] ../../../../../Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pyarrow/tests/test_jvm.py:27: could not import 'jpype': No module named 'jpype' =================================== 1 failed, 3 skipped, 4446 deselected in 6.48s ====================================

Note that we can remove paths += [test_ld_path] with my instruction because my instruction specifies DYLD_LIBRARY_PATH .

Note that we can remove paths += [test_ld_path] with my instruction because my instruction specifies DYLD_LIBRARY_PATH .

Then I would do that?

OK. I've removed it.

raulcd

Thanks @kou for taking this over! Looks good to me +1

raulcd · 2022-12-14T10:00:17Z

@jorisvandenbossche are you ok with the latest changes here?

jorisvandenbossche

Yes! Sorry for the slow response

ursabot · 2022-12-16T05:22:28Z

Benchmark runs are scheduled for baseline = 794aefd and contender = 6724fc1. 6724fc1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.56% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.27% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.34% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 6724fc1d ec2-t3-xlarge-us-east-2
[Failed] 6724fc1d test-mac-arm
[Finished] 6724fc1d ursa-i9-9960x
[Finished] 6724fc1d ursa-thinkcentre-m75q
[Finished] 794aefd6 ec2-t3-xlarge-us-east-2
[Failed] 794aefd6 test-mac-arm
[Finished] 794aefd6 ursa-i9-9960x
[Finished] 794aefd6 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

raulcd added 2 commits November 15, 2022 11:13

[Python][CI] Add DYLD_LIBRARY_PATH to avoid requiring PYARROW_BUNDLE_…

6546d3b

…ARROW_CPP on macOS job

Add DYLD_LIBRARY_PATH to test_cython too

dc81844

github-actions bot added the Component: Python label Nov 15, 2022

raulcd added 2 commits November 15, 2022 12:42

Fix unbound variable

cb77d6c

Add back the skip for cython test

4332e22

github-actions bot removed the Component: Python label Nov 15, 2022

raulcd marked this pull request as ready for review November 15, 2022 13:46

Use sys.path

28e17e1

github-actions bot added the Component: Python label Nov 16, 2022

raulcd changed the title ~~MINOR: [Python][CI] Add DYLD_LIBRARY_PATH to avoid requiring PYARROW_BUNDLE_ARROW_CPP on macOS job~~ ARROW-18339: [Python][CI] Add DYLD_LIBRARY_PATH to avoid requiring PYARROW_BUNDLE_ARROW_CPP on macOS job Nov 16, 2022

raulcd commented Nov 16, 2022

View reviewed changes

jorisvandenbossche reviewed Nov 16, 2022

View reviewed changes

Remove test_ld_path

c32a9b0

raulcd commented Nov 25, 2022

View reviewed changes

raulcd requested a review from jorisvandenbossche November 25, 2022 12:12

jorisvandenbossche approved these changes Dec 14, 2022

View reviewed changes

kou merged commit 6724fc1 into apache:master Dec 14, 2022

asfimport mentioned this pull request Dec 16, 2022

[CI][Python] Add DYLD_LIBRARY_PATH to avoid requiring PYARROW_BUNDLE_ARROW_CPP on macOS job #33508

Closed

raulcd mentioned this pull request Jan 11, 2023

[Python] test_cython failing on macOS #33344

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-18339: [Python][CI] Add DYLD_LIBRARY_PATH to avoid requiring PYARROW_BUNDLE_ARROW_CPP on macOS job #14643

ARROW-18339: [Python][CI] Add DYLD_LIBRARY_PATH to avoid requiring PYARROW_BUNDLE_ARROW_CPP on macOS job #14643

raulcd commented Nov 15, 2022

raulcd commented Nov 15, 2022

AlenkaF commented Nov 15, 2022

kou commented Nov 15, 2022

github-actions bot commented Nov 16, 2022

raulcd left a comment

jorisvandenbossche Nov 16, 2022

kou Nov 16, 2022

jorisvandenbossche Nov 18, 2022

kou Nov 18, 2022 •

edited

kou Nov 18, 2022

jorisvandenbossche Nov 22, 2022

kou Nov 22, 2022

raulcd left a comment

raulcd commented Dec 14, 2022

jorisvandenbossche left a comment

ursabot commented Dec 16, 2022

ARROW-18339: [Python][CI] Add DYLD_LIBRARY_PATH to avoid requiring PYARROW_BUNDLE_ARROW_CPP on macOS job #14643

ARROW-18339: [Python][CI] Add DYLD_LIBRARY_PATH to avoid requiring PYARROW_BUNDLE_ARROW_CPP on macOS job #14643

Conversation

raulcd commented Nov 15, 2022

raulcd commented Nov 15, 2022

AlenkaF commented Nov 15, 2022

kou commented Nov 15, 2022

github-actions bot commented Nov 16, 2022

raulcd left a comment

Choose a reason for hiding this comment

jorisvandenbossche Nov 16, 2022

Choose a reason for hiding this comment

kou Nov 16, 2022

Choose a reason for hiding this comment

jorisvandenbossche Nov 18, 2022

Choose a reason for hiding this comment

kou Nov 18, 2022 • edited

Choose a reason for hiding this comment

kou Nov 18, 2022

Choose a reason for hiding this comment

jorisvandenbossche Nov 22, 2022

Choose a reason for hiding this comment

kou Nov 22, 2022

Choose a reason for hiding this comment

raulcd left a comment

Choose a reason for hiding this comment

raulcd commented Dec 14, 2022

jorisvandenbossche left a comment

Choose a reason for hiding this comment

ursabot commented Dec 16, 2022

kou Nov 18, 2022 •

edited