Implementing clstm's network architecture in Keras

[Zerithious]

I wish to implement clstm's network architecture in Keras, to facilitate extension and modifications to the network, and also to add gpu support.

I'm pretty much a newbie to NNs, and I'm trying to wrap my head around the code and the inner workings of clstm. It would greatly help my effort if someone can answer these questions:

1. What is the exact architecture/topology of the network implemented in clstm? (I mean, what layers, what do they consist of and in which order).

2. How does it encode the images to be used as input to the network? Is it just the raw pixels of the image with varying width/height?

3. What other things would I need to know in order to implement this in Keras or a similar framework like tensorflow?

[zuphilip]

@tmbdev Tom, maybe you can share some insights for this questions. Moreover, I think you are already doing something with tensorflow.

[amitdo]

How does it encode the images to be used as input to the network?
Is it just the raw pixels of the image with varying width/height?

Yes. it takes raw pixels as input. The width is always 1 and the height is fixed - by default to 48.

[amitdo]

You might want to look at ocropy's lstm.py code, because I think it will be easier to understand. It was written by the same author who wrote clstm.

[mittagessen]

Keras already implements/utilizes LSTM nets (the network architecture) and Connectionist Temporal Classification (the loss function used for training) from both theano and tensorflow. See keras-team/keras#3436 for more details.

The only thing that's not part of keras is the preprocessing of line images, chiefly dewarping (see lineest.py) whose utility is somewhat questionable as I trained perfectly working models without it.

[tmbdev]

I already have an LSTM-based OCR for TensorFlow that works like CLSTM, but TensorFlow (and by extension, Keras) support for LSTMs is still less than ideal.

[Zerithious]

@tmbdev Can you link to the repo containing the tensorflow code? Did you mean this?

[tmbdev]

No, that's a multidimensional add-on to TensorFlow. The LSTM code is experimental and in a bunch of iPython notebooks. I'm working on a Torch version right now, which I think will work better.

[wanghaisheng]

@tmbdev any progress ?

[rayush7]

@tmbdev I am also interested in torch or tensorflow/keras based implementation of clstm. Is there any update from your end?

[chinakook]

A tensorflow implementation of popular library CLSTM and OCROPY line reader
https://github.com/ferjad/tlstm
Does anyone want to modify the repository to the cudnnLSTM or cudnnPersistentRNN implementation, by which the training could speed up about 6x to 10x times ?