Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consider to remove experimental PyarrowOnRay execution #6812

Closed
anmyachev opened this issue Dec 7, 2023 · 1 comment · Fixed by #6848
Closed

Consider to remove experimental PyarrowOnRay execution #6812

anmyachev opened this issue Dec 7, 2023 · 1 comment · Fixed by #6848
Labels
Code Quality 💯 Improvements or issues to improve quality of codebase P2 Minor bugs or low-priority feature requests

Comments

@anmyachev
Copy link
Collaborator

Some details:

  • Pandas supports pyarrow as a backend for its objects, starting with version 2.0 (optional). Modin can reuse it for different engines (Dask, Ray, Unidist).
  • Moreover, it is planned to make pyarrow dependency required starting with version 3.0.
  • Current testing of this execution is extremely sparse. At CI we ran only TestCsv tests (we're skipping even those tests right now).

@devin-petersohn your feedback here is especially valuable, as I recall it was implemented before many of us started developing Modin.

@modin-project/modin-core your feedback is also welcome
@anmyachev anmyachev added Code Quality 💯 Improvements or issues to improve quality of codebase P2 Minor bugs or low-priority feature requests labels Dec 7, 2023
@dchigarev
Copy link
Collaborator

+1 on removing it

anmyachev added a commit to anmyachev/modin that referenced this issue Jan 10, 2024
@anmyachev
REFACTOR-modin-project#6812: Remove 'PyarrowOnRay' execution in favou…
2edc7cf 
…r of pyarrow-backed pandas dataframes

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
@anmyachev anmyachev mentioned this issue Jan 10, 2024
Merged
7 tasks
YarShev pushed a commit that referenced this issue Jan 10, 2024
@anmyachev
REFACTOR-#6812: Remove 'PyarrowOnRay' execution in favour of pyarrow-…
31f8bd0 
…backed pandas dataframes (#6848)

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Code Quality 💯 Improvements or issues to improve quality of codebase P2 Minor bugs or low-priority feature requests
Projects
None yet
Milestone
No milestone
2 participants
@anmyachev @dchigarev