-
Notifications
You must be signed in to change notification settings - Fork 896
Make S3 dependencies optional #404
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report
@@ Coverage Diff @@
## master #404 +/- ##
==========================================
+ Coverage 95.99% 96.07% +0.07%
==========================================
Files 92 92
Lines 7996 8024 +28
==========================================
+ Hits 7676 7709 +33
+ Misses 320 315 -5
Continue to review full report at Codecov.
|
* change method var variable_types to prevent overloading * add and pass allowed_variable_types var to _run_dfs and _build_forward_features * allowed_variable_types can be passed to dfs * correct docstrings for allowed_variable_types in dfs and deep_feature_synthesis.build_features * fix linting issues. * fix linting errors * fix another linting error * fix F632 linter error * add test. rename build_features return_variable_types arg. * remove redundant return_variable_types attr from deep_feature_synthesis._run_dfs() * add datetime tests. minor formatting change. * remove bad line break * Update deep_feature_synthesis.py
import s3fs | ||
from botocore.exceptions import NoCredentialsError | ||
except ImportError: | ||
raise ImportError("The s3fs library is required to handle s3 files") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s3fs is actually only used as a fallback since smart_open
cannot handle the anonymous clients which leads to the NoCredentialsError
case. smart open can open s3 files in other cases.
see this issue: piskvorky/smart_open#250
perhaps move this try/except into the NoCredentialsError
block which is the only place s3fs is used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. I was assuming that smart_open
also uses s3fs
under the hood. I've now placed the s3fs
import as you suggested.
do we need to add the |
Both |
Looks good merging |
Based on the discussion in #340, this PR makes all S3-related dependencies optional, considering that S3-usage is presumably limited to a smaller range of users. However, the dependencies were moved to test-requirements.txt so that unit tests still run.
One corner case I left out is S3 buckets which are not supported right now. Adding such support would (IMHO) require adding some regular expressions which would unnecessarily clutter the code. If preferred, I can add those nonetheless.