Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rewrite model pipelines for speed, simplicity #33

Closed
dfsnow opened this issue Nov 2, 2023 · 1 comment
Closed

Rewrite model pipelines for speed, simplicity #33

dfsnow opened this issue Nov 2, 2023 · 1 comment
Assignees
@dfsnow
Labels
pipeline Code that glues all the model bits together
Milestone
2024 pipeline upg...

Comments

@dfsnow
Copy link
Member

dfsnow commented Nov 2, 2023

Last year's model had some performance issues (slow) during the post-modeling stages (assess, evaluate). I believe that these issues are likely due to heavy use of complicated dplyr calls. We should refactor each pipeline stage to improve runtimes. The easiest option is to simply drop in dtplyr and use data.table to speed things up, but the more prudent option is to rewrite in polars or data.table.
@dfsnow dfsnow self-assigned this Nov 2, 2023
@dfsnow dfsnow mentioned this issue Dec 5, 2023
Merged
@dfsnow dfsnow added this to the 2024 pipeline upgrades milestone Dec 5, 2023
@dfsnow dfsnow added the pipeline Code that glues all the model bits together label Dec 5, 2023
@dfsnow
Copy link
Member Author

dfsnow commented Dec 13, 2023

Going to close this for now. See #71 for more details. #80 fixes bugs and issues imported from #71.

@dfsnow dfsnow closed this as completed Dec 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dfsnow dfsnow

Labels
pipeline Code that glues all the model bits together
Projects
None yet
Milestone
2024 pipeline upgrades
Development

No branches or pull requests
1 participant
@dfsnow