Deep learning results are not reproducible

[hasithjp]

Motivation

H2O's Deep Learning uses a technique called HOGWILD! which greatly increases the speed of training, but is not reproducible by default.

Solution

In order to obtain reproducible results, you must set reproducible = TRUE and seed = 1 (for example, but you can use any seed as long as you use the same one each time). If you force reproducibility, it will slow down the training because this only works on a single thread. By default, H2O clusters are started with the same number of threads as number of cores (e.g. 8 is typical on a laptop).

The R example below demonstrates how to produce reproducible deep learning models:

library(h2o)
h2o.init(nthreads = -1)

# Import a sample binary outcome train/test set into R
train <- read.table("http://www.stat.berkeley.edu/~ledell/data/higgs_10k.csv", sep=",")
test <- read.table("http://www.stat.berkeley.edu/~ledell/data/higgs_test_5k.csv", sep=",")


# Convert R data.frames into H2O parsed data objects
training_frame <- as.h2o(train)
validation_frame <- as.h2o(test)
y <- "V1"
x <- setdiff(names(training_frame), y)
family <- "binomial"
training_frame[,c(y)] <- as.factor(training_frame[,c(y)])  #Force Binary classification
validation_frame[,c(y)] <- as.factor(validation_frame[,c(y)])

Now we will fit two models and show that the training AUC is the same both times (ie. reproducible).

fit <- h2o.deeplearning(x = x, y = y, 
                            training_frame = training_frame, 
                            reproducible = TRUE, 
                            seed = 1)
h2o.auc(fit)
#[1] 0.8715931

fit2 <- h2o.deeplearning(x = x, y = y, 
                            training_frame = training_frame, 
                            reproducible = TRUE, 
                            seed = 1)
h2o.auc(fit2)
#[1] 0.8715931

JIRA Issue Migration Info

Jira Issue: TN-3
Assignee: Erin Ledel
Reporter: Erin Ledel
State: Resolved