Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Very basic s3 parquet microbenchmark #954

Merged
merged 2 commits into from May 24, 2023
Merged

Very basic s3 parquet microbenchmark #954

merged 2 commits into from May 24, 2023

Conversation

xcharleslin
Copy link
Contributor

  • 1 x 64mb parquet file
  • 32 x 2mb parquet files
  • Column pruning vs whole file

Data in s3://daft-public-data/test_fixtures so microbenchmark can be run from anywhere.

@xcharleslin
Copy link
Contributor Author

@clarkzinzow you can run these with something like:

pytest tests/benchmarks/test_file_read.py -k parquet --benchmark-only

@xcharleslin xcharleslin marked this pull request as ready for review May 24, 2023 00:42
@xcharleslin xcharleslin enabled auto-merge (squash) May 24, 2023 00:42
@codecov
Copy link

codecov bot commented May 24, 2023

Codecov Report

Merging #954 (2cf9e40) into main (67d1ece) will decrease coverage by 0.01%.
The diff coverage is n/a.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #954      +/-   ##
==========================================
- Coverage   86.06%   86.06%   -0.01%     
==========================================
  Files         178      178              
  Lines       15010    15010              
==========================================
- Hits        12919    12918       -1     
- Misses       2091     2092       +1     

see 3 files with indirect coverage changes

@xcharleslin
Copy link
Contributor Author

on 69f4f0a:

------------------------------------------------------
Name (time in s)                            Min       
------------------------------------------------------
test_s3_parquet_read_1x64mb[prune:1]     2.3513 (1.0) 
test_s3_parquet_read_1x64mb[prune:0]     3.7535 (1.60)
test_s3_parquet_read_32x2mb[prune:0]     5.5111 (2.34)
test_s3_parquet_read_32x2mb[prune:1]     6.1635 (2.62)
------------------------------------------------------

@xcharleslin xcharleslin merged commit baaacdd into main May 24, 2023
13 checks passed
@xcharleslin xcharleslin deleted the charles/iobench branch May 24, 2023 00:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant