You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
So when users install fugue[spark] in a spark 3.4 environment, fugue may complain that the spark objects are not recognized, it is because without cloudpickle, fugue_spark could not be correctly imported.
The solution is to change cloudpickle to pickle for the specific usage inside fugue_spark.
The text was updated successfully, but these errors were encountered:
This change: adf3884#diff-f6002ac0db7dcceed0d45e617d3c625008cff4d1807cf64a4702821cd2aa5d17R3 introduced hard dependency of cloudpickle to fugue spark backend. And from pyspark 3.4.0, cloudpickle is no longer a dependency of pyspark.
So when users install
fugue[spark]
in a spark 3.4 environment, fugue may complain that the spark objects are not recognized, it is because without cloudpickle,fugue_spark
could not be correctly imported.The solution is to change cloudpickle to pickle for the specific usage inside
fugue_spark
.The text was updated successfully, but these errors were encountered: