Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

using arrow for larger tables #49

Closed
pachadotdev opened this issue Jan 18, 2022 · 0 comments
Closed

using arrow for larger tables #49

pachadotdev opened this issue Jan 18, 2022 · 0 comments

Comments

@pachadotdev
Copy link
Collaborator

I made some changes in the API, which offer a 50% time reduction by reading parquet instead of JSON from the API

library(microbenchmark)

microbenchmark(
  ots_create_tidy_data(table = "yrpc-parquet", reporters = "can"),
  ots_create_tidy_data(table = "yrpc", reporters = "can"),
  times = 10L
)

Unit: seconds
                                                            expr      min       lq     mean   median       uq      max neval
 ots_create_tidy_data(table = "yrpc-parquet", reporters = "can") 1.390202 1.894805 2.006423 1.989092 2.309964 2.427866    10
         ots_create_tidy_data(table = "yrpc", reporters = "can") 3.548397 3.750299 3.891013 3.889045 3.950003 4.364702    10

I already sent the PR to allow Parquet serialization in the plumber package, api.tradestatistics.io is now using plumber development version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant