Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ci] indicate support of Big Sur and drop High Sierra #3749

Merged
merged 2 commits into from
Jan 13, 2021
Merged

[ci] indicate support of Big Sur and drop High Sierra #3749

merged 2 commits into from
Jan 13, 2021

Conversation

StrikerRUS
Copy link
Collaborator

Requires recent pip version: pypa/packaging#319.

pytest doesn't work on Big Sur:

============================= test session starts ==============================
platform darwin -- Python 3.6.12, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /Users/runner/work/LightGBM/LightGBM
collected 210 items / 1 skipped / 209 selected

Fatal Python error: Aborted

../tests/c_api_test/test_.py 
Current thread 0x0000000108d48e00 (most recent call first):
  File "/Users/runner/work/LightGBM/LightGBM/tests/c_api_test/test_.py", line 72 in load_from_file
  File "/Users/runner/work/LightGBM/LightGBM/tests/c_api_test/test_.py", line 200 in test_dataset
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/python.py", line 1641 in runtest
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 311 in from_call
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 255 in call_runtest_hook
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/main.py", line 323 in _main
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/config/__init__.py", line 163 in main
  File "/Users/runner/miniconda/envs/test-env/lib/python3.6/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "/Users/runner/miniconda/envs/test-env/bin/pytest", line 11 in <module>
/Users/runner/work/LightGBM/LightGBM/.ci/test.sh: line 172:  4119 Abort trap: 6           pytest $BUILD_DIRECTORY/tests

https://github.com/microsoft/LightGBM/runs/1681244674?check_suite_focus=true#step:3:637

But all our examples work OK: https://github.com/microsoft/LightGBM/runs/1681660188?check_suite_focus=true

Compilation with both gcc and AppleClang works fine.

-- The C compiler identification is AppleClang 12.0.0.12000032
-- The CXX compiler identification is AppleClang 12.0.0.12000032
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode_12.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode_12.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -Xclang -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -Xclang -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/runner/work/LightGBM/LightGBM/build
Scanning dependencies of target _lightgbm
[  2%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
[  5%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
[  8%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
[ 11%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 14%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
[ 17%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
[ 20%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.o
[ 23%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config_auto.cpp.o
[ 26%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.o
[ 29%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.o
[ 32%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/file_io.cpp.o
[ 35%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/json11.cpp.o
[ 38%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.o
[ 41%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.o
[ 44%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/train_share_states.cpp.o
[ 47%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.o
[ 50%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.o
[ 52%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.o
[ 55%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/ifaddrs_patch.cpp.o
[ 58%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.o
[ 61%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.o
[ 64%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.o
[ 67%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.o
[ 70%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.o
[ 73%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/cuda_tree_learner.cpp.o
[ 76%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 79%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 82%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 85%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/linear_tree_learner.cpp.o
[ 88%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
[ 91%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
[ 94%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[ 97%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.o
[100%] Linking CXX shared library ../lib_lightgbm.so
[100%] Built target _lightgbm

-- The C compiler identification is GNU 10.2.0
-- The CXX compiler identification is GNU 10.2.0
-- Checking whether C compiler has -isysroot
-- Checking whether C compiler has -isysroot - yes
-- Checking whether C compiler supports OSX deployment target flag
-- Checking whether C compiler supports OSX deployment target flag - yes
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/local/bin/gcc-10 - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Checking whether CXX compiler has -isysroot
-- Checking whether CXX compiler has -isysroot - yes
-- Checking whether CXX compiler supports OSX deployment target flag
-- Checking whether CXX compiler supports OSX deployment target flag - yes
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/local/bin/g++-10 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -fopenmp (found version "4.5") 
-- Found OpenMP_CXX: -fopenmp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5")  
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/runner/work/LightGBM/LightGBM/build
Scanning dependencies of target _lightgbm
[  2%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
[  5%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
[  8%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
[ 11%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 14%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
[ 17%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
[ 20%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.o
[ 23%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config_auto.cpp.o
[ 26%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.o
[ 29%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.o
[ 32%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/file_io.cpp.o
[ 35%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/json11.cpp.o
[ 38%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.o
[ 41%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.o
[ 44%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/train_share_states.cpp.o
[ 47%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.o
[ 50%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.o
[ 52%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.o
[ 55%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/ifaddrs_patch.cpp.o
[ 58%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.o
[ 61%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.o
[ 64%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.o
[ 67%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.o
[ 70%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.o
[ 73%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/cuda_tree_learner.cpp.o
[ 76%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 79%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 82%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 85%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/linear_tree_learner.cpp.o
[ 88%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
[ 91%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
[ 94%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[ 97%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.o
[100%] Linking CXX shared library ../lib_lightgbm.so
[100%] Built target _lightgbm

But running examples with gcc results in

OMP: Error #13: Assertion failure at kmp_csupport.cpp(522).
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see https://bugs.llvm.org/.
/Users/runner/work/LightGBM/LightGBM/.ci/test.sh: line 192:  4167 Abort trap: 6           python $f

error. I believe it is due to some conflicts between multiple OpenMP installations again. For example, one excess Intel OpneMP comes from conda deps intel-openmp pkgs/main/osx-64::intel-openmp-2019.4-233. And probably requires something like

LightGBM/.ci/test.sh

Lines 73 to 76 in 1abc2e0

if [[ $OS_NAME == "macos" ]] && [[ $COMPILER == "clang" ]]; then
# fix "OMP: Error #15: Initializing libiomp5.dylib, but found libomp.dylib already initialized." (OpenMP library conflict due to conda's MKL)
for LIBOMP_ALIAS in libgomp.dylib libiomp5.dylib libomp.dylib; do sudo ln -sf "$(brew --cellar libomp)"/*/lib/libomp.dylib $CONDA_PREFIX/lib/$LIBOMP_ALIAS || exit -1; done
fi

but for gcc now. I'm not trying to find a workaround right now. macos-11 is a preview image now on GitHub Actions, therefore environment might be changed later. Also, builds on macos-11 are queried for 45+ mins before starting to run, so debugging process is is extremely annoying now. It is better to wait for full release of the image and then test gcc again.

jameslamb
jameslamb reviewed Jan 11, 2021
View reviewed changes
Copy link
Collaborator

@jameslamb jameslamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the implication of this change? Does it mean that users on anything older than Big Sur will be building from source (instead of wheel) when they pip install lightgbm==3.2.0 once that's released?

Big Sur only came out about 2 months ago, so I'm worried that it's too soon to make a change like that.

@StrikerRUS
Copy link
Collaborator Author

StrikerRUS commented Jan 11, 2021 •
edited

@jameslamb

Does it mean that users on anything older than Big Sur will be building from source (instead of wheel) ...

On anything older than Mojave (10.14) according to the policy #2691.

@jameslamb
Copy link
Collaborator

@jameslamb

Does it mean that users on anything older than Big Sur will be building from source (instead of wheel) ...

On anything older than Mojave (10.14) according to the policy #2691.

ok great, thanks

@StrikerRUS StrikerRUS merged commit 577b2f9 into master Jan 13, 2021
@StrikerRUS StrikerRUS deleted the big_sur branch January 13, 2021 18:44
@github-actions
Copy link

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@jameslamb jameslamb jameslamb approved these changes

@Laurae2 Laurae2 Awaiting requested review from Laurae2

@guolinke guolinke Awaiting requested review from guolinke guolinke is a code owner

Assignees
No one assigned
Labels
breaking
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@StrikerRUS @jameslamb