Once #566 is implemented we could investigate making it easier for users to copy their Spark DataFrame into Dask and perform additional computation. This could benefit users that want to read from traditional hadoop data sources, such as parquet, in a distributed fashion.
Implementation concerns:
- When a spark worker goes away we need to figure out reconstitute the corresponding dataframe. We can do this easily with the long running one, but its an open question for how dask will handle this temporary loss of data.
Once #566 is implemented we could investigate making it easier for users to copy their Spark DataFrame into Dask and perform additional computation. This could benefit users that want to read from traditional hadoop data sources, such as parquet, in a distributed fashion.
Implementation concerns: