Unable to convert caffe ResNet model

[EpochalEngineer]

Moved this from #8 since it doesn't really fit under that topic:

I was able to convert ResNet50 with the current commit and adding the data layer to the model text, but wasn't able to get inference working.

Assuming it was accidentally getting recompiled , I removed the explicit compile step and got this error (same as if replacing mode=0 with mode=2 in the model files):

ValueError: GpuReshape: cannot reshape input of shape (2048) to shape (1, 1, 1, 7).
Apply node that caused the error: GpuReshape{4}(scale5a_branch1_gamma, TensorConstant{[1 1 1 7]})
Toposort index: 447
Inputs types: [CudaNdarrayType(float32, vector), TensorType(int64, vector)]
Inputs shapes: [(2048,), (4,)]
Inputs strides: [(1,), (8,)]
Inputs values: ['not shown', array([1, 1, 1, 7])]
Outputs clients: [[GpuElemwise{Composite{(((i0 * (i1 + i2)) + i3) + ((i4 * i5) + i6))},no_inplace}(GpuReshape{4}.0, GpuElemwise{Composite{(i0 * (i1 / i2))},no_inplace}.0, GpuReshape{4}.0, GpuReshape{4}.0, GpuReshape{4}.0, GpuElemwise{Composite{((i0 * (i1 / i2)) + i3)},no_inplace}.0, GpuReshape{4}.0)]]

only change to test_converted.py was replacing the 3 losses with one when compiling.

model.summary() looks reasonable around that area.

Using tensorflow instead as a backend crashes when trying to load the weights.

ValueError: Shapes (112,) and (64,) are not compatible

python2.7/site-packages/keras/engine/topology.pyc in load_weights(self, filepath, seq_to_functional)
2381 ' elements.')
2382 weight_value_tuples += zip(symbolic_weights, weight_values)
-> 2383 K.batch_set_value(weight_value_tuples)
2384 f.close()
2385

I modified batch_set_value to pass a sanitized version of the name along, and the problem occurs on the first caffe batchnorm scaling layer:

site-packages/tensorflow/python/ops/gen_state_ops.py(45)assign()
43 result = _op_def_lib.apply_op("Assign", ref=ref, value=value,
44 validate_shape=validate_shape,
---> 45 use_locking=use_locking, name=name)
46 return result
47

ipdb> name
u'scale_conv1_scale_conv1_gamma'

So it seems related to the new Scale layer which attempts to replicate caffe's batchnorm+scale division

[MarcBS]

I have replicated the first error you mention, but it looks like it is not in the first Scale layer, but as this line of Theano's error suggests:

Apply node that caused the error: GpuReshape{4}(scale5a_branch1_gamma, TensorConstant{[1 1 1 7]})

it is in layer 'scale5a_branch1', which is close to the end of the network.

If we look at the result of model.summary():

bn5a_branch1 (BatchNormalization)(None, 2048, 7, 7) 4096 res5a_branch1[0][0]

---

bn5a_branch2c (BatchNormalization(None, 2048, 7, 7) 4096 res5a_branch2c[0][0]

---

scale5a_branch1 (Scale) (None, 2048, 7, 7) 4096 bn5a_branch1[0][0]

---

scale5a_branch2c (Scale) (None, 2048, 7, 7) 4096 bn5a_branch2c[0][0]

---

res5a (Merge) (None, 2048, 7, 7) 0 scale5a_branch1[0][0]
scale5a_branch2c[0][0]

it seems that for some reason it is trying to reshape the 'gamma' parameter, which has a shape of (2048) to (1,1,1,7), when in fact it should reshape it to (1,2048,1,1) for applying the corresponding operations on the input data.

About your tests with Tensorflow I must say that I am not providing compatibility for it, so several unknown errors might arise.

I will continue taking a look at this, if you have any other idea or clue let me know.

[sfrodrigues]

Any update on this front? I'm getting the same error..

[MarcBS]

I am sorry, I didn't have time to give direct support to this topic and enable the caffe2keras conversion for residual networks.

Although, an implementation of ResNet50 together with its trained weights was recently added to Keras by @fchollet. You can find the info to use it in this link.

[sfrodrigues]

Hey Marc!

I understand. However Im not trying to convert a resnet pretrained in Imagenet but in places2 (which we then could add to keras). Any idea on when you might have a look at this?

[MarcBS]

I am pretty busy right now, so I'm afraid can't promise anything. Although, PRs are welcome in case anybody is willing to help with this or any other possible functionality.