Update setup.py #1802
Update setup.py #1802
Conversation
… stable release of numpy at release time.
Codecov Report
@@ Coverage Diff @@
## master #1802 +/- ##
==========================================
+ Coverage 50.99% 51.01% +0.01%
==========================================
Files 88 88
Lines 12222 12258 +36
==========================================
+ Hits 6233 6253 +20
- Misses 5989 6005 +16
Continue to review full report at Codecov.
|
|
Thanks for digging into this! I am hesitant to lock in the numpy version since that will also effect people who build from source (in which case the cext will not be tied to the wrong numpy version). So I don't want to stop SHAP from working on the latest numpy when just a rebuild will do it. However, this is clearly something that would be good to clear up. What about building a very informative error message for this case? That way people know to either rebuild SHAP or lower their numpy version. |
|
I believe that this issue is related to pypa/pip#9542 (and many other packages are similarly affected). I haven't been able to completely confirm it, but it sounds like perhaps the correct fix is to use Also note that there are other ways that this issue can manifest itself:
Another note: this only currently results in an error when SHAP is installed via a source distribution (so by default Windows users are spared from this, but there are no binary releases of SHAP for macOS or Linux), because the binary wheels for SHAP were built against numpy 1.19 (which is also forward compatible with numpy 1.20), while the source distribution will be built against the latest version of numpy even if the user has some other version of numpy installed locally already. However, if the binary wheels for a future release of SHAP are built against numpy 1.20, then I believe that even users of binary distributions will see this issue if they try to run while using numpy 1.19 locally (which again may be forced by some other package like tensorflow that the user has installed). |
|
Thanks @kbattocchi ! I hadn't been aware of @slundberg wdyt? I've update the PR to reflect this. I think there's loads of value in an informative warning as well if leaving setup.py as is, I think this would have to be raised wherever |
|
@mkretsch327 I couldn't repro your issue exactly as filed, but almost. After running import shap
import numpy as np
from sklearn.ensemble import RandomForestRegressor
shap.TreeExplainer(RandomForestRegressor(max_depth=4, n_estimators=10).fit(
np.random.normal(size=(30, 6)), np.random.normal(size=(30,))))fails with That same code succeeds on all platforms when using a python 3.6 environment (where numpy 1.20 wasn't released, so the version of numpy that pip builds shap from source against is also 1.19 and everything is fine), and under Windows for python 3.7 and 3.8 (where binary wheels for shap are available). The same code also succeeds when running |
I saw this |
|
The current |
|
@slundberg Please let me know your thoughts on using |
|
We also ran into this issue while trying to install shap in the same environment as tensorflow (which pins numpy at 1.19.5), similar to what @jklaise described above. While we wait for this PR to be finalized, are there any suggested workarounds that can help? |
|
@menewman the workaround is to install |
|
@jklaise - Ahhh, thank you. I will give that a shot! 🙏 EDIT: It looks like this works! Much appreciated. |
This is what we needed to do as well. |
|
✨ This is an old work account. Please reference @brandonchinn178 for all future communication ✨ Any updates on this? This started failing again on Dec 31 2021, seemingly when numpy released 1.22 (although we have CI pinned to numpy 1.20, so I'm not sure why it started failing). This comment is also relevant to this discussion: #1895 (comment) |
I too have the same issue today and using numpy 1.20.3 and shap 0.39.0. |
Me too am facing the exact same error. C extension was not built during install! During handling of the above exception, another exception occurred: Traceback (most recent call last): |
Steps that I have done, I have installed Shap==0.39 seperately and then installed my requirement.txt packages. 2022-01-07T08:20:04.9291747Z Step 4/7 : RUN pip install shap==0.39.0 |
Hello, I managed to fix the issue by using the latest version of shap :) |
|
✨ This is an old work account. Please reference @brandonchinn178 for all future communication ✨
Thanks, but unfortunately, we need to keep shap pinned to 0.39.0, so this won't work for us. It would be nice if this PR could be backported and have a 0.39.1 released. |
Hi @slundberg,
First, I want to thank you for maintaining this project. It’s been incredibly useful over the last few years!
The initial motivation for this PR was the recent version bump of Numpy to v1.20.0, which coincided with a change to their C API (table of versions in numpy's source code).
A reproducible version of this issue is observed with the following:
pip install -r requirements.txt, with a requirements.txt of:This problem has manifested itself in the past when numpy's C api version has changed( link to mapping), and often manifests with the below error (taken from my stack trace), also seen in #382 and #381:
While it often is resolved by uninstalling and re-installing
shap, I don't believe this prevents the issue from occurring in the first place, as such an installation is broken when leveraging arequirements.txtfile.The culprit appears to be the specification of
numpywithin the setup_requires argument of setup insetup.py.The fix proposed here is to bound the
numpyversion to the latest stable release available atshaprelease time. In addition to fixing the above mentioned observed error, this will also help with the current behavior observed when installing an older version ofshapin isolation. As an example, I'd expectshap==0.36to be installed and setup againstnumpy 1.19.2, but as of today, it's done againstnumpy1.20.0`.Look forward to hearing your thoughts and comments!