model.fit_generator to fit data into memory

[johnkorn]

Hi, all!
I'm trying to train deep net on a big dataset that doesn't fit into memory.
Is there any way to use generators to read batches into memory on every training step?
I'm looking for behaviour similar to fit_generator method in Keras.

I know that in pure tensorflow following snippet can be wrapped by for loop to train on several batches:

batch_gen = generator(data)
batch = batch_gen.next()

sess.run([optm, loss, ...], feed_dict = {X: batch[0], y: batch[1]})

[aymericdamien]

That is a good idea! While this get implemented, you can use image_preloader or hdf5 dataset to deal with that issue.

[gboeer]

I have to say that especially the case of using large datasets wit tflearn is very poorly implemented (or maybe just badly documented ?).
In tf we have queues and file readers of all kinds but there is no easy way to supply queues to tflearn, or at least I couldn't find a way.

Regarding the hdf5 I cant do
from tflearn.data_utils import build_hdf5_image_dataset
ImportError: cannot import name 'build_hdf5_image_dataset'

Yes, I am using the latest version via
pip install --upgrade git+https://github.com/tflearn/tflearn.git

[iuria21]

Hi, is it yet any fit_generator available? or we have to change to Keras?

[GEOFBOT]

@imanoluria for now you can use regular TensorFlow to train a TFLearn network (which is what I've been doing)