Make S3 dependencies optional #404
Conversation
Codecov Report
@@ Coverage Diff @@
## master #404 +/- ##
==========================================
+ Coverage 95.99% 96.07% +0.07%
==========================================
Files 92 92
Lines 7996 8024 +28
==========================================
+ Hits 7676 7709 +33
+ Misses 320 315 -5
Continue to review full report at Codecov.
|
* change method var variable_types to prevent overloading * add and pass allowed_variable_types var to _run_dfs and _build_forward_features * allowed_variable_types can be passed to dfs * correct docstrings for allowed_variable_types in dfs and deep_feature_synthesis.build_features * fix linting issues. * fix linting errors * fix another linting error * fix F632 linter error * add test. rename build_features return_variable_types arg. * remove redundant return_variable_types attr from deep_feature_synthesis._run_dfs() * add datetime tests. minor formatting change. * remove bad line break * Update deep_feature_synthesis.py
| import s3fs | ||
| from botocore.exceptions import NoCredentialsError | ||
| except ImportError: | ||
| raise ImportError("The s3fs library is required to handle s3 files") |
There was a problem hiding this comment.
s3fs is actually only used as a fallback since smart_open cannot handle the anonymous clients which leads to the NoCredentialsError case. smart open can open s3 files in other cases.
see this issue: piskvorky/smart_open#250
perhaps move this try/except into the NoCredentialsError block which is the only place s3fs is used.
There was a problem hiding this comment.
I see. I was assuming that smart_open also uses s3fs under the hood. I've now placed the s3fs import as you suggested.
|
do we need to add the |
|
Both |
|
Looks good merging |
Based on the discussion in #340, this PR makes all S3-related dependencies optional, considering that S3-usage is presumably limited to a smaller range of users. However, the dependencies were moved to test-requirements.txt so that unit tests still run.
One corner case I left out is S3 buckets which are not supported right now. Adding such support would (IMHO) require adding some regular expressions which would unnecessarily clutter the code. If preferred, I can add those nonetheless.