segfault with Python 3.9 MacOS conda

[cdeil]

Description

The simple example below segfaults:

% python crash2.py
zsh: segmentation fault  python crash2.py

Reproducible example

import lightgbm as lgb
from sklearn.datasets import fetch_california_housing
X_train, y_train = fetch_california_housing(return_X_y=True)
model = lgb.LGBMRegressor()
model.fit(X_train, y_train)

Environment info

LightGBM version or commit hash: lightgbm-3.3.0-py3-none-macosx_10_14_x86_64.macosx_10_15_x86_64.macosx_11_0_x86_64.whl

Command(s) you used to install LightGBM

conda create -c conda-forge -n lgb-crash python=3.9 scikit-learn=1.0
conda activate lgb-crash
pip install lightgbm

% conda list # packages in environment at /Users/cdeil/opt/anaconda3/envs/lgb-crash: # # Name Version Build Channel ca-certificates 2021.10.8 h033912b_0 conda-forge joblib 1.1.0 pyhd8ed1ab_0 conda-forge libblas 3.9.0 11_osx64_openblas conda-forge libcblas 3.9.0 11_osx64_openblas conda-forge libcxx 12.0.1 habf9029_0 conda-forge libffi 3.4.2 he49afe7_4 conda-forge libgfortran 5.0.0 9_3_0_h6c81a4c_23 conda-forge libgfortran5 9.3.0 h6c81a4c_23 conda-forge liblapack 3.9.0 11_osx64_openblas conda-forge libopenblas 0.3.17 openmp_h3351f45_1 conda-forge libzlib 1.2.11 h9173be1_1013 conda-forge lightgbm 3.3.0 pypi_0 pypi llvm-openmp 12.0.1 hda6cdc1_1 conda-forge ncurses 6.2 h2e338ed_4 conda-forge numpy 1.21.2 py39h7eed0ac_0 conda-forge openssl 3.0.0 h0d85af4_1 conda-forge pip 21.2.4 pyhd8ed1ab_0 conda-forge python 3.9.7 h38b4d05_3_cpython conda-forge python_abi 3.9 2_cp39 conda-forge readline 8.1 h05e3726_0 conda-forge scikit-learn 1.0 py39hd2caeff_1 conda-forge scipy 1.7.1 py39h056f1c0_0 conda-forge setuptools 58.2.0 py39h6e9494a_0 conda-forge sqlite 3.36.0 h23a322b_2 conda-forge threadpoolctl 3.0.0 pyh8a188c0_0 conda-forge tk 8.6.11 h5dbffcc_1 conda-forge tzdata 2021c he74cb21_0 conda-forge wheel 0.37.0 pyhd8ed1ab_1 conda-forge xz 5.2.5 haf1e3a3_1 conda-forge zlib 1.2.11 h9173be1_1013 conda-forge

Additional Comments

Originally reported in microsoft/FLAML#243 but as mentioned in microsoft/FLAML#243 (comment) it's a lightgbm issue, has nothing to do with flaml.

[cdeil]

It looks like this is an issue with the wheel, or an incompatibility of the wheel with conda.

This resolved the issue for me:

pip uninstall lightgbm
conda install -c conda-forge lightgbm

which results in

% conda list | grep lightgbm
lightgbm                  3.2.1            py39h9fcab8e_0    conda-forge

@sonichi - the reason I ran into this issue in the first place is that I ran pip install flaml since there was no conda-forge package for it, and the pip resolver installed the lightgbm wheel as part of the flaml install.

So this issue can be avoided, but IMO would still be nice to identify the root cause and fix it if possible.

[StrikerRUS]

Hey @cdeil, thanks for reporting this issue!

This is a duplicate of #4632 and the root cause is the upstream bug in libomp >= 12 on macOS #4229 (comment).

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.