Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Cloudpickle is unnecessarily a hard dependency of Spark backend #478

Closed
goodwanghan opened this issue Jun 6, 2023 · 0 comments · Fixed by #474
Closed

[BUG] Cloudpickle is unnecessarily a hard dependency of Spark backend #478

goodwanghan opened this issue Jun 6, 2023 · 0 comments · Fixed by #474
Labels
bug Something isn't working compatibility
Milestone

Comments

@goodwanghan
Copy link
Collaborator

This change: adf3884#diff-f6002ac0db7dcceed0d45e617d3c625008cff4d1807cf64a4702821cd2aa5d17R3 introduced hard dependency of cloudpickle to fugue spark backend. And from pyspark 3.4.0, cloudpickle is no longer a dependency of pyspark.

So when users install fugue[spark] in a spark 3.4 environment, fugue may complain that the spark objects are not recognized, it is because without cloudpickle, fugue_spark could not be correctly imported.

The solution is to change cloudpickle to pickle for the specific usage inside fugue_spark.

@goodwanghan goodwanghan added bug Something isn't working compatibility labels Jun 6, 2023
@goodwanghan goodwanghan added this to the 0.8.5 milestone Jun 6, 2023
@goodwanghan goodwanghan linked a pull request Jun 6, 2023 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working compatibility
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant