Is there way to import caffe model into tensorlayer

[fangde]

Dear all:

I find the tensor layer is resemble the caffe structure, both are based model block.
I just wonder if there is a way to load caffe model and initialize tensor layer model?

comments and suggestions are welcome

best regards

[JindongJiang]

Yes. By using caffe-tensorflow .

First, follow the instructions in caffe-tensorflow e.g. :

convert.py VGG_ILSVRC_19_layers_deploy.prototxt --caffemodel VGG_ILSVRC_19_layers.caffemodel --data-output-path=vgg19.npy

You can get prototxt and caffemodel here: Model Zoo.

Then you get vgg19.npy that stores pre-trained weights. Since npy here share different structure with npz which was commonly used in TensorLayer, you still need another step to extract the weighted (python3):

npy = np.load(vgg_weights_dir, encoding='latin1')
params = []
for val in sorted(npy.item().items()):
    if val[0] == 'conv5_2':
        break
    params.append(val[1]['weights'])
    params.append(val[1]['biases'])
tl.files.assign_params(sess, params, network)

The if val[0] == 'conv5_2': break means to skip the weights after conv5_2, which is just for my own project.

[ghost]

@JindongJiang what network graph for VGG19 do you use with that loading code?

I guess one could also need to transpose the caffe weights as often in tensorflow/tensorlayer the network uses (width, height, channels) format instead of (channels, width, height) as in caffe for example?

[JindongJiang]

@joekr552 The graph I use comes from one of the TensorLayer examples.

Well, without going deeper into caffe-tensorflow's code, I found by experimenting on jupyter notebook that caffe-tensorflow already did that for us.

[ghost]

Aha, ok! thanks!

Did you check the GBR and RGB channel ordering as well?

[JindongJiang]

@joekr552 I just check caffe-tensorflow code for you. The good news is that they did transpose the c, w, h.

def transform_data(self):
        if self.params is None:
            transformers = [

                # Reshape the parameters to TensorFlow's ordering
                DataReshaper({
                    # (c_o, c_i, h, w) -> (h, w, c_i, c_o)
                    NodeKind.Convolution: (2, 3, 1, 0),

                    # (c_o, c_i) -> (c_i, c_o)
                    NodeKind.InnerProduct: (1, 0)
                }),

                # Pre-process batch normalization data
                BatchNormPreprocessor(),

                # Convert parameters to dictionaries
                ParameterNamer(),
            ]
            self.graph = self.graph.transformed(transformers)
            self.params = {node.name: node.data for node in self.graph.nodes if node.data}
        return self.params

But I can't find the code where they transpose the BGR into RGB, so I think they did not. And that also explain why we should do the transpose by hand at the TL vgg19 example

if tf.__version__ <= '0.11':
        red, green, blue = tf.split(3, 3, rgb_scaled)
    else: # TF 1.0
        print(rgb_scaled)
        red, green, blue = tf.split(rgb_scaled, 3, 3)
    assert red.get_shape().as_list()[1:] == [224, 224, 1]
    assert green.get_shape().as_list()[1:] == [224, 224, 1]
    assert blue.get_shape().as_list()[1:] == [224, 224, 1]
    if tf.__version__ <= '0.11':
        bgr = tf.concat(3, [
            blue - VGG_MEAN[0],
            green - VGG_MEAN[1],
            red - VGG_MEAN[2],
        ])
    else:
        bgr = tf.concat([
            blue - VGG_MEAN[0],
            green - VGG_MEAN[1],
            red - VGG_MEAN[2],
        ], axis=3)

[ghost]

thanks!

[ghost]

Actually I am a bit confused on whatever the approach used to get the weights used in the vgg16 example in TL is a good approach: I don't think it is a valid approach, the approach you mentioned above does seem correct however.

See my discussion post here> https://www.cs.toronto.edu/~frossard/post/vgg16/

[JindongJiang]

You're welcome. But the post in your link seems more like a tutorial than an issue, where are your confusion.

[ghost]

I am referring to my comment at the bottom of the page :

"Joel Kronander to Davi Frossard • a day ago
To me it seems weird/wrong to just reorganize the filter channels in the first layers?
Later layers assume that they were assigned in that order to specific color channels?!
Ie if a higher order filter wants to get high activity for say a red edge, by re-ordering first layer filter weights this is effectively turned into a blue edge"

My issue was that in the blog post the author said he post-processed the weights after running the caffe-tensorflow code, which included a step explained as: "..reorder the filters in the first layer so that they are in RGB order (instead of the default BGR. "

To me that is a strange thing to do as consecutive higher filters will assume that the first layer filters where in BGR order. An extreme example is that a "red cat" detector would become all confused.

The approach you mentioned above, where the input is first transformed to BGR from RGB should be correct though.

[JindongJiang]

@joekr552 That's a tricky yet reasonable confusion, thank you so much for mention it here. But after a second thought, I figure that maybe the author's method is rational. Let me put it in this way:
Just looking at the input layer I_rgb with RGB format and the weight W_rgb (conv kernel) along with the first hidden layer H.

H = I_rgb * W_rgb

That is, all neurons in the H receive some particular area of I convoluted by W. While it seems that to get the same value of neuron when the order of the channel of I change, all we need to do is modify the order of W.
After we make sure that

 I_bgr * W_bgr ==  I_rgb * W_rgb

in the first hidden layer, we know the later� hidden layer wouldn't face the problem you mentioned.

[ghost]

Hmm, yes u are right. Thanks for the clarification.

[prabhjot-singh-gogana]

hi @JindongJiang , Could you please tell how to convert .npz model into .caffemodel. Thanks in advance

[JindongJiang]

Hi @prabhjot-singh-gogana , sorry for my late response, the graduate application takes much of my time.

I might not be so familiar with TensorLayer now since I have switched to Tensorflow and PyTorch. But I think onnx could help converting the model, you may wanna try it.

[passionguan]

Hi @prabhjot-singh-gogana, Have you solved your conversion problem yet? I faced the similar issue about converting numpy .npz format model to .caffemodel. Can I discuss this problem with you? Many thanks for that!