New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
drake is slow in a Windows use case #1086
Comments
|
Awesome! With the full Rprof file, I could generate some As I say below, Full workflow: Just Just pprof file: Some things I notice right away:
|
|
@adamkski, I made a couple changes that should decrease the runtime of For slowness due to |
|
Trying to remove the |
|
G. Grothendieck has a super promising candidate solution to the slowness in |
With pleasure! I'm using 7.8.0.9000 now.
I'll likely be rebuilding some targets this week so I'll try to run the profiler if I notice a slowdown again. |
|
|
That explains why |
|
FYI: #1089 and eddelbuettel/digest#138 are related. You may see speedups if you install development |
|
Deparsing should be about 5x faster in f19efcb. With development |
|
I've uploaded a code example that appears to have a similar slow down issue as I had before on some data I could get openly. I tried to replicate some patterns I use at work to setup data for SQL queries (though in my example I don't actually query a SQL db). Hope that gives you the flavor of what you were looking for. Code: https://github.com/adamkski/ships/blob/master/drake/plan.R P.S. I haven't tried "fst" format yet. |
|
Awesome, thanks for uploading the example! It could help with other use cases going forward. Did you mean to say library(proffer) # https://github.com:wlandau/proffer
download.file(
"https://raw.githubusercontent.com/adamkski/ships/master/your_samples.rds",
"your_samples.rds"
)
px <- serve_rprof("your_samples.rds")To get Another thing I notice is that One little opportunity for a speedup in |
It's at 4.86 seconds just now. That's still much better than before. I was just posting the code example for the record. And so I can at least try to have a reprex |
|
The speed improvement is good news! Thank you for posting the example. I have a suspicion it will come in handy later on. |
|
Unfortunately, the time for a I have two more Rprof files that could be useful (I just used an arbitrary name "project emerald"). It's using the code from my original post, plus additional targets to bring it to 57. What's remarkable is that the first run calls to P.S. I'll try to add a pseudo-reprex to my ships example when I get the chance. P.P.S.
This had a speed boost the first time I set the options, but once I ran it a second time it seemed to also slow down again. Haven't tried turning off logging build times because that's pretty useful! ;) |
|
Just so I know, what commit hash of packageDescription("drake")$GithubSHA1From the results you sent, it looks like the second run was the fast one. library(proffer)
serve_rprof("2nd-run.rds")The first one seems to spend the vast majority of time actually running the commands in the plan (especially serve_rprof("1st-run.rds")Did you mean to mean to send me a different set of files? Alternatively, now that Also, |
|
Thanks I'll check that package out! Those were the files I meant to send... and it looks like it's sped back up. Feels magical sometimes. Anyways maybe I'll wait for a longer term pattern to emerge, if it does. Thanks for the reply. |
Glad it took care of itself.
No worries, I would rather have false alarms than miss bottlenecks. These things can be elusive. |






Prework
drake's code of conduct.remotes::install_github("ropensci/drake")) and mention the SHA-1 hash of the Git commit you install.Description
As I'm building a drake plan out to include more and more steps, I'm finding that make(plan) and vis_drake_graph() are taking longer to start executing.
Link to Rstudio community post.
Reproducible example
Partial example that seems to at least illustrate the slow down problem:
Code: https://github.com/adamkski/ships/blob/master/drake/plan.R
Rprof: https://github.com/adamkski/ships/blob/master/your_samples.rds
Benchmarks
My drake plan on the other hand has 35 targets and it takes 17 seconds to run the same command. There is a similar delay for targets for make(plan) to start creating targets.
Rprof file:
your_samples.zip
The text was updated successfully, but these errors were encountered: