Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmarks: lance file format for dependency table #425

Closed
wants to merge 1 commit into from

Conversation

hagenw
Copy link
Member

@hagenw hagenw commented May 30, 2024

Based on the comment in #424, I benchmarked lance as a file format for storing the dependency table.

Results for writing 1000000 lines.

method csv pickle parquet lance
pd.DataFrame[object] 2.017 0.683 0.491 0.470
pd.DataFrame[string] 2.031 0.641 0.524 0.560
pd.DataFrame[pyarrow] 2.050 0.275 0.272 0.308
pd.DataFrame[object] -> pd.DataFrame[pyarrow] 2.203 0.501
pd.DataFrame[object] -> pa.Table 0.552
pd.DataFrame[string] -> pa.Table 0.603
pd.DataFrame[pyarrow] -> pa.Table 0.272
pd.DataFrame[object] -> pd.DataFrame[pyarrow] -> pa.Table 0.477
pa.Table 0.237 0.290 0.260

Results for reading 1000000 lines.

method csv pickle parquet lance
----> pd.DataFrame[object] 1.126 0.266 0.349 0.671
----> pd.DataFrame[string] 1.206 0.244 0.396 0.678
----> pd.DataFrame[pyarrow] 2.508 0.124 0.385 0.638
----> pd.DataFrame[pyarrow] -> pd.DataFrame[object] 2.714 0.289
-c--> pd.DataFrame[object] 1.165
-c--> pd.DataFrame[string] 1.133
-c--> pd.DataFrame[pyarrow] 2.572
-c--> pd.DataFrame[pyarrow] -> pd.DataFrame[object] 2.664
-pa-> pd.DataFrame[object] 0.426
-pa-> pd.DataFrame[string] 0.415
-pa-> pd.DataFrame[pyarrow] 0.542
-pa-> pd.DataFrame[pyarrow] -> pd.DataFrame[object] 0.801
----> pa.Table -> pd.DataFrame[object] 0.287 0.217
----> pa.Table -> pd.DataFrame[string] 0.305 0.320
----> pa.Table -> pd.DataFrame[pyarrow] 0.131 0.092
----> pa.Table -> pd.DataFrame[pyarrow] -> pd.DataFrame[object] 0.370
----> pa.Table 0.054 0.048 0.332
----> lance.LanceDataset 0.014

As can be seen, reading speed is only better if we read to an lance.LanceDataset object. Which means we also need to check, how the dependency table methods would behave when directly working on a lance dataset.


NOTE: results are not yet added to the README

@hagenw hagenw marked this pull request as draft May 30, 2024 08:48
@hagenw hagenw mentioned this pull request May 30, 2024
@hagenw
Copy link
Member Author

hagenw commented Aug 14, 2024

At the moment, there is no need to integrate this into main.
I will close this merge request.

@hagenw hagenw closed this Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant