Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PERF] Use pyarrow table for pickling rather than ChunkedArray #1488

Merged
merged 6 commits into from
Oct 13, 2023
Merged

[PERF] Use pyarrow table for pickling rather than ChunkedArray #1488

merged 6 commits into from
Oct 13, 2023

Conversation

samster25
Copy link
Member

@samster25 samster25 commented Oct 12, 2023 •
edited
Loading

  • Use pyarrow table for pickling rather than ChunkedArray so we can exploit ray's special pickling for arrow tables which doesn't work on ChunkedArray

@samster25 samster25 changed the title try table for pickling [PERF] Use pyarrow table for pickling rather than ChunkedArray Oct 12, 2023
@codecov
Copy link

codecov bot commented Oct 13, 2023

Codecov Report

Merging #1488 (f0ddc2c) into main (8ad7fda) will increase coverage by 0.08%.
Report is 2 commits behind head on main.
The diff coverage is 100.00%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1488      +/-   ##
==========================================
+ Coverage   74.87%   74.96%   +0.08%     
==========================================
  Files          60       60              
  Lines        6102     6123      +21     
==========================================
+ Hits         4569     4590      +21     
  Misses       1533     1533
Files Coverage Δ
daft/series.py 92.65% <100.00%> (+0.41%) ⬆️

@samster25 samster25 merged commit e1e5eaf into main Oct 13, 2023
24 checks passed
@samster25 samster25 deleted the sammy/pickle-series-as-tables branch October 13, 2023 19:00
jaychia added a commit that referenced this pull request Nov 14, 2023
@jaychia
Revert "[PERF] Use pyarrow table for pickling rather than ChunkedArray (
07d367a 
#1488)"

This reverts commit e1e5eaf.
jaychia added a commit that referenced this pull request Nov 14, 2023
@jaychia
[PERF][REVERT] Reverts: use pyarrow table for pickling rather than Ch…
fd1b8be 
…unkedArray (#1488) (#1605)

This reverts commit e1e5eaf.

It appears that this was causing a performance regression
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@clarkzinzow clarkzinzow Awaiting requested review from clarkzinzow

@jaychia jaychia Awaiting requested review from jaychia

Assignees
No one assigned
Labels
performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@samster25