Hyperparameter Optimization for Keras model with large dataset

[NTNguyen13]

Condition Check:

• [x ] I'm up-to-date with the latest release:

  pip install -U talos
  

• [x ] I've confirmed that my Keras model works outside of Talos.

If you still have an error, please submit complete trace and a code with:

• output of shape for x and y e.g. (212,12)
• Talos params dictionary
• The Keras model wired for Talos
• Description of extra variables in the model

You can provide the code in pastebin / gist or any other format you like.

---

I want to perform Hyperparameter Optimization on my Keras Model. The problem is the dataset is quite big, normally in training I usefit_generatorto load the data in batch from disk, but the Talos only support fit method.

I tried to load the whole data to memory, by using this:

train_generator = train_datagen.flow_from_directory(
    original_dir,
    target_size=(img_height, img_width),
    batch_size=train_nb,
    class_mode='categorical')
X_train,y_train = train_generator.next()

But the when performing talos.Scan(), the OS kills it because of large memory usage. I also tried to undersampling my dataset to only 10%, but it's still too big.

I saw that the issue #11 is being working on, but I wonder is there any workaround strategy to perform Hyperparameter Opimization for large dataset in this case?

[matthewcarbone]

@NTNguyen13 Indeed it is something being worked on. Just curious: how large is your dataset exactly?

[NTNguyen13]

Hi, @x94carbone my dataset has 11000 images, each with 30-40 KB

[matthewcarbone]

Hmm ok. Does your model work with Keras' fit on its own? Without using fit_generator?

[NTNguyen13]

I tried it with small random data and it works, it didn't work in the real case because of the large dataset

[matthewcarbone]

So this likely means that there's nothing wrong with Talos at the moment, we've just gotta implement a feature to get fit_generator to work. Feel free to let me know if I'm missing something here.

[mikkokotila]

Looks like all clear so closing here.

[johndavidmiller]

bump.

I'm fairly new to DL and Keras (but not new to other AI and ML) and I gotta say that it is amazing that most tools, examples, and academic papers are all built around these teeny tiny "toy" datasets like MNIST and CIFAR. Who uses 32x32 pixel images in a real application?

Please add my vote for adding Keras Sequence support in Talos so it can be used in real applications.