-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NULL model blows up memory usage with parallelised fits #230
Comments
A reprex for this would be great. |
Here's a reprex: library(tidyr)
library(dplyr)
library(tsibble)
library(feasts)
library(fable)
data(orangeJuice, package="bayesm")
start_date <- as.Date("1970-01-01")
oj_data <- orangeJuice$yx %>%
complete(store, brand, week) %>%
mutate(week=yearweek(start_date + week*7)) %>%
as_tsibble(index=week, key=c(store, brand))
subset_oj_data <- function(start, end)
{
start <- yearweek(start_date + start*7)
end <- yearweek(start_date + end*7)
filter(oj_data, week >= start, week <= end)
}
ncores <- max(2, parallel::detectCores(logical=FALSE) - 2)
cl <- parallel::makeCluster(ncores, type="PSOCK")
parallel::clusterEvalQ(cl,
{
library(feasts)
library(fable)
library(tsibble)
})
res_par <- parallel::parLapply(cl, list(subset_oj_data(40, 135)), function(df)
{
model(df,
ets=ETS(logmove ~ error("A") + trend("A") + season("N"))
)
}) On my laptop, this uses 10GB of memory before returning.
As an aside, it takes |
By memory usage, are you referring to the object size of the returned object? |
No, I mean the memory usage as shown in Task Manager |
Btw, I see the same problem in an Ubuntu VM. The object returned from the |
Is there a way to generate a null model from scratch? Would probably save a lot of time in trying to reproduce this problem. |
I'm surprised that it is the null models which cause this issue. |
Ok, I just tried
Here is an example failed model fit from ETS:
It's probably not the null model object that is the problem, but all the other bits that get returned from |
library(fabletools)
tsibble::pedestrian %>%
model(null_model(Count))
#> # A mable: 4 x 2
#> # Key: Sensor [4]
#> Sensor `null_model(Count)`
#> <chr> <model>
#> 1 Birrarung Marr <NULL model>
#> 2 Bourke Street Mall (North) <NULL model>
#> 3 QV Market-Elizabeth St (West) <NULL model>
#> 4 Southern Cross Station <NULL model> Created on 2020-02-20 by the reprex package (v0.3.0) |
My guess again is the environments held in the transformation. |
Yup, I just tried the following; mod_list <- parallel::parLapply(cl, oj_train, function(df) model(df, null=null_model(logmove)))
mod_list
str(mod_list[[1]]$null[[1]])
That's very truncated compared to the "real" null model above. |
No, cancel that, it appears to be the same structure. So there must be an environment that unclass(mod_list[[1]]$null[[1]])
|
Here's an even simpler reprex. library(fable)
library(feasts)
library(tsibble)
set.seed(12345)
df <- expand.grid(x=1:1000, t=1:100)
df$y <- runif(nrow(df))
miss <- rbinom(nrow(df), 1, 0.25)
df$y[as.logical(miss)] <- NA
df$t <- as.Date("1970-01-01") + df$t
df <- as_tsibble(df, key=x, index=t)
cl <- parallel::makeCluster(4)
parallel::clusterEvalQ(cl,
{
library(fable)
library(feasts)
library(tsibble)
})
bad <- parallel::parLapply(cl, list(df, df, df, df), function(df)
{
model(df, ets=ETS(y ~ error("A") + trend("A") + season("N")))
}) This chews up memory on both my Windows laptop and an Ubuntu VM in Azure. Interestingly, if I replace the ETS model with a good <- parallel::parLapply(cl, list(df, df, df, df), function(df)
{
model(df, null=null_model(y))
}) |
Looks like this is also an issue without parallel (to a lesser extent).
|
Added some performance improvements to the model parser.
|
Well, I would (maybe) expect ETS to take more time and resources, since it's actually trying to fit a model. null_model can return immediately since it doesn't have to do anything. This seems to be separate from the parallel issue, where returning a buggy object back to the master triggers a massive memory allocation. Note that if I replace the ETS could probably check for NAs right at the start, and return immediately if found. This should reduce the resource requirements to the minimum. I'm trying to find a similar reprex for ARIMA, but it seems to be behaving well so far. |
Closing as this issue is largely driven by tidyverts/fabletools#146 Parallel processing performance will be optimised in the next two months. |
model()
sometimes blows up and eats all the memory on my machine (32GB, Win 10 Pro). I'll try to make a reprex. It seems kind of random though.The text was updated successfully, but these errors were encountered: