Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: expose timings in verbose state of OOC sort #14979

Merged
merged 1 commit into from
Mar 11, 2024
Merged

feat: expose timings in verbose state of OOC sort #14979

merged 1 commit into from
Mar 11, 2024

Conversation

ritchie46
Copy link
Member

OOC sort started
Temporary directory path in use: temp
run UdfExec
RUN STREAMING PIPELINE
parquet -> sort -> parquet_sink
RefCell { value: [] }
STREAMING CHUNK SIZE: 2941 rows
finished sinking into OOC sort in 3.982244806s
full file dump of OOC sort took 5.155624213s
processing 245 files
partitioning sort took: 5.063799475s
spilling all partitioned files took: 7.694338058s
started sort source phase
sort source phase took: 3.780204067s
full ooc sort took: 16.632243722s

Shows we are twiddling thumbs waiting for all files to be spilled. We probably can do something useful in that time.

@github-actions github-actions bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars labels Mar 11, 2024
@ritchie46 ritchie46 merged commit 78dc628 into main Mar 11, 2024
19 checks passed
@ritchie46 ritchie46 deleted the stats branch March 11, 2024 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant