Could this work with tf.nn.dynamic_rnn?

[carlthome]

No description provided.

[iwyoo]

It is related with issue#1. There is no guarantee that ConvLSTM can work with the functions provided by TensorFlow, because they are using different operators. I'm looking on it, but I'm not sure when I completely fix that issue.
You can dynamically link ConvLSTM blocks like my example code. If you are in hurry, I recommend to implement in that way.

[carlthome]

What do you think of doing something like this instead? First an initial tf.conv3d to get channels to match RNNCell._num_units and then some reshaping in RNNCell.__call__ to fit 5D into a 3D tensors as far as the RNN API is concerned? Does the math hold up or will this cause problems?

[iwyoo]

The difference of ConvLSTM and an original LSTM is the 'operator' inside. Therefore, your 'convolve_input' can be used for the original LSTM and it can work well, but it will not work same with ConvLSTM. However, if you are not in a (badly) special case, I recommend use the original LSTM as your suggestion. Many literature uses 1d code from CNN for LSTM codes rather than using 2d data like images.
Therefore, your code is the more common and useful way to handle 2d sequential data.
In the reference paper, the authors are comparing the original LSTM and their ConvLSTM. It only means that it is harmful to feed raw image data to LSTM, but it doesn't mean that ConvLSTM is the best way to handle 2d image sequences.