Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation Fault on MacOS with pytorch > 2.2.0 #3524

Open
connortann opened this issue Feb 26, 2024 · 4 comments · May be fixed by #3568
Open

Segmentation Fault on MacOS with pytorch > 2.2.0 #3524

connortann opened this issue Feb 26, 2024 · 4 comments · May be fixed by #3568
Labels
bug Indicates an unexpected problem or unintended behaviour ci Relating to Continuous Integration / GitHub Actions dependencies Pull requests that update a dependency file
Milestone
0.46.0

Comments

@connortann
Copy link
Collaborator

connortann commented Feb 26, 2024 •
edited

EDIT: relevant issue on pytorch: pytorch/pytorch#121101

The test suite recently began failing on MacOS.

Example failing run:

https://github.com/shap/shap/actions/runs/8021717954/job/21914432162

Fatal Python error: Segmentation fault

Thread 0x000070000ca96000 (most recent call first):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/pool.py", line 579 in _handle_results
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 982 in run
  File Fatal Python error: Segmentation fault

"/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1002 in _bootstrap

When the python version is pinned to 3.11.7, we seemingly get a different error relating to lightgbm:

/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ctypes/__init__.py:376: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: dlopen(/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/lightgbm/lib/lib_lightgbm.so, 6): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
E     Referenced from: /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/lightgbm/lib/lib_lightgbm.so
E     Reason: image not found

Related:
@connortann connortann added bug Indicates an unexpected problem or unintended behaviour ci Relating to Continuous Integration / GitHub Actions labels Feb 26, 2024
@connortann connortann changed the title CI failing on MacOS: Segmentation Fault Segmentation Fault on MacOS in test suite Feb 26, 2024
This was referenced Feb 26, 2024
Merged
Merged
@connortann
Copy link
Collaborator Author

This may be unrelated, but on the topic of MacOS issues I noticed there was a failure in the mac-os job for the latest proposed release on conda-forge: conda-forge/shap-feedstock#76

@CloseChoice
Copy link
Collaborator

Here are a couple notes:
The last successful run of the macos pipeline on master was this: https://github.com/shap/shap/actions/runs/7972563240/job/21765144274.
I debugged the macos pipeline using https://github.com/mxschmitt/action-tmate and found that the segmentation faults happen in the pytorch tests, in the lines where one calls the model on data, e.g. here. From the successful run I found that we used torch version 2.2.0 there instead of 2.2.1. Will check if it works if I pin the version.

@connortann connortann changed the title Segmentation Fault on MacOS in test suite Segmentation Fault on MacOS with pytorch > 2.2.0 Mar 4, 2024
@connortann
Copy link
Collaborator Author

connortann commented Mar 4, 2024 •
edited

#3518 fixed the original issue with the tests by pinning pytorch; let's keep this issue open until the full test suite passes with the latest pytorch.

@connortann connortann added the dependencies Pull requests that update a dependency file label Mar 4, 2024
@CloseChoice
Copy link
Collaborator

CloseChoice commented Mar 4, 2024 •
edited

The pytorch issue is documented in: pytorch/pytorch#121101.

@connortann connortann added this to the 0.46.0 milestone Mar 8, 2024
@connortann connortann linked a pull request Mar 11, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Indicates an unexpected problem or unintended behaviour ci Relating to Continuous Integration / GitHub Actions dependencies Pull requests that update a dependency file
Projects
None yet
Milestone
0.46.0
Development

Successfully merging a pull request may close this issue.

Update macos tests to latest pytorch shap/shap
2 participants
@CloseChoice @connortann