hdf5_matrix as model input

[Fubukimaru]

Hello,

I have modified the convolutional LSTM example (https://keras.rstudio.com/articles/examples/conv_lstm.html) for processing my own data. It works with a regular multidimensional array, however if I use an HDF5 matrix it doesn't work.

The code is the following:

dataset <- hdf5_matrix(datasetPath, datasetHDF5Path)

# Extract input shape
df_input_shape <- dataset$shape
# Time is NULL to enable variable length in sequences.
df_input_shape[2] <- list(NULL) 
df_input_shape[1] <- NULL # Remove number of samples

# Model definition --------------------------------------------------------
{
#Initialize model
model <- keras_model_sequential()
    # Input shape
    # With channels_last (default) -> (samples,time, rows, cols, channels)
    # In input_shape we don't include the samples
model %>%
  # Begin with 2D convolutional LSTM layer
  layer_conv_lstm_2d(
    input_shape = df_input_shape,  # Requiered when this layer is the first.
    data_format = "channels_last",
    filters = 5, kernel_size = c(3,3),
    padding = "same", 
    return_sequences = TRUE
  ) %>%
  # Normalize the activations of the previous layer
  layer_batch_normalization() %>%
  
  # Add 3x hidden 2D convolutions LSTM layers, with
  # batch normalization layers between
  layer_conv_lstm_2d(
    filters = 5, kernel_size = c(3,3),
    data_format = "channels_last",
    padding = "same", return_sequences = TRUE
  ) %>%
  layer_batch_normalization() %>%
  layer_conv_lstm_2d(
    filters = 5, kernel_size = c(3,3),
    data_format = "channels_last",
    padding = "same", return_sequences = TRUE
  ) %>%
  layer_batch_normalization() %>% 
  layer_conv_lstm_2d(
    filters = 5, kernel_size = c(3,3),
    data_format = "channels_last",
    padding = "same", return_sequences = TRUE
  ) %>%
  layer_batch_normalization() %>%
  
  # Add final 3D convolutional output layer 
  layer_conv_3d(
    filters = df_input_shape[[4]], kernel_size = c(3,3,3),
    activation = "sigmoid", 
    padding = "same", data_format ="channels_last"
  )
}

# Prepare model for training
model %>% compile(
  loss = "binary_crossentropy", 
  optimizer = "adadelta"
)

model


# Training ----------------------------------------------------------------



model %>% fit(
  dataset,
  dataset,
  batch_size = 1,
  epochs = 30
  #validation_split = 0.05
)

The model compiles properly using the extracted shape:

Model                                                                                                     
___________________________________________________________________________________________________________
Layer (type)                                    Output Shape                              Param #          
===========================================================================================================
conv_lst_m2d_2 (ConvLSTM2D)                     (None, None, 762, 1000, 5)                1460             
___________________________________________________________________________________________________________
batch_normalization_1 (BatchNormalization)      (None, None, 762, 1000, 5)                20               
___________________________________________________________________________________________________________
conv_lst_m2d_3 (ConvLSTM2D)                     (None, None, 762, 1000, 5)                1820             
___________________________________________________________________________________________________________
batch_normalization_2 (BatchNormalization)      (None, None, 762, 1000, 5)                20               
___________________________________________________________________________________________________________
conv_lst_m2d_4 (ConvLSTM2D)                     (None, None, 762, 1000, 5)                1820             
___________________________________________________________________________________________________________
batch_normalization_3 (BatchNormalization)      (None, None, 762, 1000, 5)                20               
___________________________________________________________________________________________________________
conv_lst_m2d_5 (ConvLSTM2D)                     (None, None, 762, 1000, 5)                1820             
___________________________________________________________________________________________________________
batch_normalization_4 (BatchNormalization)      (None, None, 762, 1000, 5)                20               
___________________________________________________________________________________________________________
conv3d_1 (Conv3D)                               (None, None, 762, 1000, 3)                408              
===========================================================================================================
Total params: 7,408
Trainable params: 7,368
Non-trainable params: 40
___________________________________________________________________________________________________________

However, at the moment I try to fit the model it gives me an error:

> model %>% fit(                                                                                          
+   dataset,                                                                                              
+   dataset,                                                                                              
+   batch_size = 1,                                                                                       
+   epochs = 30                                                                                           
+   #validation_split = 0.05                                                                              
+ )                                                                                                       
Error in dim(x) <- length(x) : invalid first argument     

Am I doing anything wrong? Are these functions already working with hdf5_matrix type? I cannot find the code for the fit() function in this repository, so it's a blind guess.

Thanks in advance!

[jjallaire]

Just search the repository for "fit <- function" to find the source code:

https://github.com/rstudio/keras/search?utf8=%E2%9C%93&q=%22fit+%3C-+function%22&type=

We'd certainly be open to a PR to get this working!

[Fubukimaru]

Thanks @jjallaire!

It's not implemented.

https://github.com/rstudio/keras/blob/a82b0ecb9853567ab31f24422edacbabdc99730b/R/model.R Line 420 is:

 args$x <- keras_array(x)

Which leads to this:

> keras_array                                                                                             
function (x, dtype = NULL)                                                                                
{
    if (is.null(x)) 
        return(x)
    if (is.list(x)) 
        return(lapply(x, keras_array))
    if (!inherits(x, "numpy.ndarray")) {
        if (is.null(dtype) && is.double(x)) 
            dtype <- backend()$floatx()
        if (!is.array(x)) 
            x <- as.array(x)
        x <- r_to_py(x)
    }
    if (is.null(dtype)) 
        dtype <- x$dtype
    x$astype(dtype = dtype, order = "C", copy = FALSE)
}

This type of data ends up in:

  if (!is.array(x)) 
            x <- as.array(x)

class(dataset)                                                                                          
>> [1] "keras.utils.io_utils.HDF5Matrix" "python.builtin.object"  
as.array(dataset)                                                                                       
>> Error in dim(x) <- length(x) : invalid first argument 

I'm not really sure about what would be the steps to implement this, but I understand that this is already solved in Python Keras, right?

[jjallaire]

What happens if you add a first clause like this:

if (inherits(x, "keras.utils.io_utils.HDF5Matrix"))
  return(x)

That would be my naive way of going about fixing it. If that works, I'll make the change. If that doesn't work if you could provide me with a short example a Gist (including an HDF5 file) then I'll sort out what we need to do from there.

[Fubukimaru]

This week and the following I'll be trying to do what I wanted to do in python so I know how everything works in order to be sure about modifications (Basically test the ConvLSTM example with a HDF5Matrix).

I'll come back as soon as possible.

[jjallaire]

Okay, if you could point me to even the simplest Python example that uses an HDF5Matrix I can take a closer look.

…

On Thu, May 24, 2018 at 5:07 AM, Alberto Gutiérrez Torre < ***@***.***> wrote: This week and the following I'll be trying to do what I wanted to do in python so I know how everything works in order to be sure about modifications (Basically test the ConvLSTM example with a HDF5Matrix). I'll come back as soon as possible. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#410 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGXx1NdB-IhuiBfQ3JTXLuTbb8NIQ6Pks5t1nhRgaJpZM4UKilj> .

[Fubukimaru]

I grabbed a snippet from keras-team/keras#6874 and tested it in python. It worked. It includes the HDF5 dataset creation and a simple model execution.

So I adapted it to illustrate the case:

from keras.models import Sequential
from keras.layers import Dense
from keras.utils.io_utils import HDF5Matrix
import numpy as np
import math

HDF5Path = "/tmp/test.h5"

def create_dataset(N,M):
    import h5py
    X = np.random.randn(N,M).astype('float32')
    y = np.random.randint(0, 2, size=(N,1))
    f = h5py.File(HDF5Path, 'w')
    # Creating dataset to store features
    X_dset = f.create_dataset('my_data', (N,M), dtype='f')
    X_dset[:] = X
    # Creating dataset to store labels
    y_dset = f.create_dataset('my_labels', (N,1), dtype='i')
    y_dset[:] = y
    f.close()

N = 10**7
M = 100

create_dataset(N,M)

# Instantiating HDF5Matrix for the training set, which is a slice of the first 150 elements
X_train = HDF5Matrix(HDF5Path, 'my_data', start=0, end=math.floor(2*N/3))
y_train = HDF5Matrix(HDF5Path, 'my_labels', start=0, end=math.floor(2*N/3))

# Likewise for the test set
X_test = HDF5Matrix(HDF5Path, 'my_data', start=math.floor(2*N/3), end=N)
y_test = HDF5Matrix(HDF5Path, 'my_labels', start=math.floor(2*N/3), end=N)


model = Sequential()
model.add(Dense(64, input_shape=(M,), activation='relu'))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='sgd')


batchSize = 1000

# Note: you have to use shuffle='batch' or False with HDF5Matrix
model.fit(X_train, y_train, batch_size=batchSize, shuffle='batch')

model.evaluate(X_test, y_test, batch_size=batchSize)

The expected behavior is that the RAM memory should be stable along all the process as it should read the data for each batch from disk.

You'll notice the following:

• If you have a big batch size you'll notice disk usage on specific points of time (like a burst).
• If you have a small batch size, you'll be more or less constantly reading from disk.

Use M and N to modify the dataset size.

[Fubukimaru]

I couldn't modify the previous comment, so I'm making a new one.

• N: number of rows.
• M: number of variables/features.

[jjallaire]

When I try this code:

from keras.utils.io_utils import HDF5Matrix
import numpy as np
import math

HDF5Path = "test.h5"

def create_dataset(N,M):
    import h5py
    X = np.random.randn(N,M).astype('float32')
    y = np.random.randint(0, 2, size=(N,1))
    f = h5py.File(HDF5Path, 'w')
    # Creating dataset to store features
    X_dset = f.create_dataset('my_data', (N,M), dtype='f')
    X_dset[:] = X
    # Creating dataset to store labels
    y_dset = f.create_dataset('my_labels', (N,1), dtype='i')
    y_dset[:] = y
    f.close()

N = 10**7
M = 100

create_dataset(N,M)

# Instantiating HDF5Matrix for the training set, which is a slice of the first 150 elements
X_train = HDF5Matrix(HDF5Path, 'my_data', start=0, end=int(math.floor(2*N/3)))
y_train = HDF5Matrix(HDF5Path, 'my_labels', start=0, end=int(math.floor(2*N/3)))

I get the following error:

KeyError: "Unable to open object (object 'my_data' doesn't exist)"

Do you see that as well?

[jjallaire]

This is a speculative fix for the issue (basically just leaving objects of type HDF5 alone): d3d65b4

Let me know if that addresses the issue on your end.

[Fubukimaru]

KeyError: "Unable to open object (object 'my_data' doesn't exist)"

This looks like the HDF5 couldn't be created properly (create_dataset).

On the other hand I tried your commit with the following code:

# from https://keras.rstudio.com/ example
library(keras)

HDF5Path <- "/tmp/test.h5"

X <- hdf5_matrix(HDF5Path, "/my_data")
Y <- hdf5_matrix(HDF5Path, "/my_labels")

inputshp <- X$data$shape
inputshp[1] <- NULL #list(NULL)

model <- keras_model_sequential()
{
model %>%
  layer_dense(units = 64, activation = 'relu', input_shape = inputshp) %>%
  layer_dense(units = 1, activation = 'sigmoid')
}

model %>% compile(
  loss = 'binary_crossentropy',
  optimizer = optimizer_rmsprop(),
  metrics = c('accuracy')
)

model

history <- model %>% fit(
  X, Y, 
  epochs = 30, batch_size = 10**6, 
  validation_split = 0.2
)

model

history <- model %>% fit(
  X, Y, 
  epochs = 30, batch_size = 10**6, 
  validation_split = 0.2
)

And it seems to be working with the dataset created with the other script :).

I'm not sure if the behaviour is 100% the same but I'm assuming so as HDF5_Matrix is already a Python object, right?

Thanks!

[jjallaire]

Yes, it is indeed already a Python object.

Seems like we've got this working and can close the issue. Correct?

[Fubukimaru]

I think we can close this issue. It appears to be working correctly.
If I find any issue with it I'll reopen this ticket.

Thanks!