Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Use _collect_as_arrow for fugue_api.as_arrow(spark_df) #516

Closed
ion-elgreco opened this issue Sep 22, 2023 · 3 comments · Fixed by #511
Closed

[FEATURE] Use _collect_as_arrow for fugue_api.as_arrow(spark_df) #516

ion-elgreco opened this issue Sep 22, 2023 · 3 comments · Fixed by #511

Comments

@ion-elgreco
Copy link

Is your feature request related to a problem? Please describe.
Convert spark df to arrow can be done with private method inside pyspark: _collect_as_arrow https://github.com/apache/spark/blob/06ccb6d434476afacc08936cf473670102d41010/python/pyspark/sql/pandas/conversion.py#L244

@ion-elgreco ion-elgreco changed the title [FEATURE] Use _collect_as_arrow for sparkdf.as_arrow() [FEATURE] Use _collect_as_arrow for fugue_api.as_arrow(spark_df) Sep 22, 2023
@goodwanghan
Copy link
Collaborator

This is a great idea, will release a dev version

@goodwanghan
Copy link
Collaborator

@ion-elgreco please try 0.8.7.dev5, it uses _collect_as_arrow

@ion-elgreco
Copy link
Author

@ion-elgreco please try 0.8.7.dev5, it uses _collect_as_arrow

Nice, I'll try it out later this week!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants