-
Notifications
You must be signed in to change notification settings - Fork 242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use lazy_loader for pyspark #168
Conversation
Please update the description completely |
@@ -50,6 +50,7 @@ | |||
"urllib3>=1.22,<2.0.0", | |||
"wrapt>=1.0.0,<2.0.0", | |||
"papermill>=1.2.0", | |||
"ipykernel>=5.0.0", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See the description. Still not sure why it worked before but adding this does resolve the error I was seeing while testing: jupyter/notebook#1558
|
||
from flytekit.common.types.helpers import pack_python_std_map_to_literal_map as _packer | ||
from flytekit.contrib.notebook.supported_types import notebook_types_map as _notebook_types_map | ||
from flytekit.plugins import pyspark as _pyspark |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this will work because only when we access pyspark.x
will it load the real module?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, that's my understanding of lazy_loading here: https://github.com/lyft/flytekit/blob/master/flytekit/plugins/__init__.py#L5 . Have tested, it works.
* Remove dependence on pyspark for non-spark notebooks * Use lazy_loader for pyspark * lint * lint
TL;DR
Use lazy_loaded for pyspark so that non-spark notebooks can run without pyspark installed.
Also added a dependency on
ipykernel
. Executions in my local venv were failing with aNo module named ipykernel_launcher
.Type
Are all requirements met?
Complete description
Use lazy_loader for pyspark
Tracking Issue
flyteorg/flyte#488
Follow-up issue
NA
OR
https://github.com/lyft/flyte/issues/