Unable to successfully convert/run keras LSTM that uses batchsize>1

[lauriebyrum]

It appears that batchsize>1 isn't currently supported in coreml with LSTMs. I haven't been able to find proof of this. If it is true, it means i need to break a single model into several as i have several LSTMs that all use batch. Please confirm if this is the case. If it isn't the case, then there is a bug somewhere. Here is the code that shows what I'm saying:
import keras
from keras.models import Sequential
import coremltools
import numpy as np

model = Sequential()
model.add(keras.layers.InputLayer(input_shape=(1, 1)))
bidir_layer = keras.layers.Bidirectional(keras.layers.LSTM(3, batch_size=2))
model.add(bidir_layer)
inputarray = np.array([[[1]],[[.5]]], dtype=np.float32)
keras_out = model.predict(inputarray)
coreml_model = coremltools.converters.keras.convert(model, input_names='input')
coreml_model.save('from_keras.mlmodel')

model = coremltools.models.MLModel('from_keras.mlmodel')

print("Both right answers from keras:" + str(keras_out))

predictions = model.predict({'input': inputarray[0][0]})
print("First right answer from coreml:" + str(predictions))
predictions = model.predict({'input': inputarray[1][0]})
print("Second right answer from coreml:" + str(predictions))

predictions = model.predict({'input':inputarray})
print("Wrong answer from coreml:" + str(predictions))
print(predictions)

Based on code in #147 (comment), the transpose below looks necessary, but it gives an error when predict is called:
Traceback (most recent call last):
File "keras-converter.py", line 32, in
predictions = model.predict({'input':inputarray})
File "lib/python3.6/site-packages/coremltools/models/model.py", line 326, in predict
return self.proxy.predict(data,useCPUOnly)
RuntimeError: {
NSLocalizedDescription = "Different batch numbers for input features.";
}
inputarray = np.transpose(inputarray, (1,0,2))
predictions = model.predict({'input':inputarray})

[aseemw]

Yes CoreML does have a limitation currently that Batch and Sequence size cannot be more than 1 concurrently.

For LSTMs related networks, CoreML input should be either rank 3 (Seq,B,C) or rank 5 (Seq,B,C,H,W)

You can either write a for loop to iterate over the batches, or the suggested way is to use the CoreML Batch API.

[lauriebyrum]

i guess the batch api is this? https://developer.apple.com/documentation/coreml/mlmodel/2962866-predictions

Ok. This means we will have to break our model into multiple models since we have stuff before and after the lstm and that all needs the batching. Can you consider this a feature request?

[lauriebyrum]

(thanks for all your help Aseem!. I'm going to close this)

[aseemw]

Actually, to make it official it would be great if you can file a "feature request" bug at https://developer.apple.com/bug-reporting/
Since this requires a change in the framework, that would be the correct place to make an official feature request. Thanks!

[lauriebyrum]

Sure. Meanwhile, I don't see an equivalent batch api in the python api. Is that right? It would be really (really) nice if there were a way to run it in python so we can have a batch job that compares the results of our converted models with the original. Is there something there that i'm missing?

[aseemw]

Yes, batch API hasn't been exposed to Python. Since the batch API is a performance feature for the app, it has not been exposed to Python.

The results from the batch API, however, should be identical to what you would get by looping over the elements in the batch.

[lauriebyrum]

I'm sorry if i'm being dense, and i really appreciate all this help...doesn't this take us back to why we need at least c to not be 0 from #263 ?

[aseemw]

Just a few comments to clarify:

In Keras the input to a Bi-dir LSTM is a rank 3 tensor where the 3 dimensions corresponds to
(Batch, Sequence, feature_vector) in that order.

Whereas in CoreML, the order is different, it should be (Sequence, Batch, feature_vector).
(actually, underneath, all CoreML tensors are mapped to rank 5, with a fixed order: (Seq,Batch,C,H,W), so rank 3 is equivalent to (Seq,B,C,1,1))
Hence the transpose.

If an input of rank 1 is provided, like you did in the code snippet with inputarray[0][0], it is automatically pre-pended with 1s (corresponding to Seq and Batch).

I am not sure why the transpose did not work for you. Which OS version did you try it on?
For me, it works on 10.14.

print(inputArray.shape) #lets say (1,10,120) : (B,S,C) interpretation in Keras, so it means batch size of 1, seq length of 10, feature vector of size 120
inputarray = np.transpose(inputarray, (1,0,2)) #now the shape is (10,1,120): (Seq,B,C)
predictions = model.predict({'input':inputarray}) #this works, although the values are not correct, whenever batch size is more than 1

Now with LSTMs, only Batch dimension 1 is supported, so the input that you would feed need to have shape (Seq_length, 1, C)

Where do you need the hidden memory state of the Bi-dir LSTM (which is generally denoted by c)?
Its not like that its being passed around between batches. Every element in the batch is independent and the values of c are not shared among them.
And for sequence length more than 1, the hidden states will be passed within the LSTM, automatically by CoreML. So you don't need to worry about that.

To summarize: In the example snippet, that you presented initially, since you were using a Batch size =2 in Keras; to get the CoreML prediction out using coremltools, you would need to do a for loop over the batch size. Example:

import keras
from keras.models import Sequential
import coremltools
import numpy as np

np.random.seed(80)

Batch = 4
Seq_length = 10
C = 40
C_out = 3

model = Sequential()
model.add(keras.layers.InputLayer(input_shape=(Seq_length, C)))
bidir_layer = keras.layers.Bidirectional(keras.layers.LSTM(C_out, batch_size=Batch))
model.add(bidir_layer)

print(model.summary())

inputarray = np.random.rand(Batch, Seq_length, C)
print('keras input shape: (Batch, Seq, C) ', inputarray.shape)
keras_out = model.predict(inputarray)
print('keras out shape: ', keras_out.shape)

coreml_model = coremltools.converters.keras.convert(model, input_names='input')
coreml_model.save('from_keras.mlmodel')

model = coremltools.models.MLModel('from_keras.mlmodel')

coreml_out_final = np.zeros((Batch, 2*C_out))

for batch_id in range(Batch):
    x = inputarray[batch_id, :, :] # shape: (Seq, C)
    x = np.expand_dims(x, axis= 1) # shape: (Seq, 1, C)
    predictions = model.predict({'input':x})['output1']
    coreml_out_final[batch_id, :] = predictions

print(np.sum(np.abs(keras_out[:] - coreml_out_final[:])))