Cannot load pretrained weights into a modified network

[Scienceseb]

Following the issue #5 (now the problem is with the loading of the pretrained weights)

I'm able to assign a tensor...and load the value contain in the checkpoints, but... But it's seem I still have some problems...

Code 1 (saving the pretrained weight in numpy, line variables_to_restore = [vv for vv in variables_to_restore if 'conv1' not in vv.name] not active):

 for var, val in zip(vars, vars_vals): #to save in numpy the pretrained weights
          if 'rgb/resnet_v1_50/conv1/weights' in var.name:
                np.save('weights_conv1', val)

Code 2 (assigning the pretrained weights) :

 vars = tf.trainable_variables()
 vars_vals = sess.run(vars)

 for var, val in zip(vars, vars_vals):
            if 'rgb/resnet_v1_50/conv1/weights' in var.name:

                numpy_tensor=np.load('/media/seb/SSD2/master/comparaison/admd/NYUD/weights_conv1.npy')

                t1=tf.convert_to_tensor(numpy_tensor,dtype=tf.dtypes.float32)

                t2 = tf.random.uniform([7, 7, 1, 64], minval=0, maxval=None,
                                       dtype=tf.dtypes.float32)  # the 4th channel here
                res = tf.concat(axis=2, values=[t1,
                                                t2])  # the concatenation on the channel axis to obtain a [7,7,4,64] tensor
                var.assign(res)

So I made those verifications:

 numpy_tensor=np.load('/media/seb/SSD2/master/comparaison/admd/NYUD/weights_conv1.npy')
 t1=tf.convert_to_tensor(numpy_tensor,dtype=tf.dtypes.float32)
       with tf.Session() as sess:
              print(sess.run(t1))

This print exactly the value expected, it contain the pretrained weights. So that's ok.
But after executing the block Code 2 to assign te pretrained weights to the convolution initialized randomly I do:

vars = tf.trainable_variables()
for var, val in zip(vars, vars_vals):
            if 'rgb/resnet_v1_50/conv1/weights' in var.name:
                print(val)

I get exactly the randomly initialized weights from before that block of code and not the pretrained ones...the assign dont have any effect on the trainable_variables

The problem, because it's seem I dont have any effect on the trainable_variables, must be with :
vars = tf.trainable_variables()
vars_vals = sess.run(vars)

According to some person on Stack_Overflow: ass=sess.run(var) and sess.run(ass) at the end of block Code 2, that resolve that, the transfer is ok for the weights of conv1.

Or do you think that variables_to_restore = [vv for vv in variables_to_restore if 'conv1' not in vv.name] also affect variable like rgb/resnet_v1_50/block3/unit_3/bottleneck_v1/conv1/weights:0 (float32_ref 1x1x1024x256) [262144, bytes: 1048576] because of the conv1...? I changed it for variables_to_restore = [vv for vv in variables_to_restore if 'rgb/resnet_v1_50/conv1/weights' not in vv.name] so it's suppose to be exactly the same weights and it's working. But I still have weird thing happening...

[pmorerio]

I did not check, but I guess var.assign only creates the assignment operation.
That is, you have to run that operation after you create it, like this

assign_op = var.assign(res)
sess.run(assign_op)  # also assign_op.op.run()

[Scienceseb]

That's strange because in PyTorch with the same network, optimizer, parameters and loss, 4 channels (RGB+Depth) gives better results than just 3 (RGB only) and that's what's expected, but with your code in tensorflow accuracy is lower with 4 channels...I think I will have no choice but to compare in PyTorch and extrapolate from there...

[Scienceseb]

This issue should not be closed because your code still don’t work with RGB+D concatenation...It should work like in pytorch (improve the performance and not decreasing them).

[pmorerio]

Hi. This code refers to our paper. We never ever perform rgb+d concatenation there. I have been happy to help you with your code, but I cannot be in charge of solving issues with your modified version of my code.

[Scienceseb]

Ok but do you have any idea how its possible that your code gives lower performances with RGB+Depth concatenate than just RGB and Depth while in PyTorch its better ?

[pmorerio]

There can be many reasons I think

1. are you sure you are assigning the rgb weights correctly?
2. random uniform initialization may not be a good choice, you can leave the default init which is automatically run in the solver (here)
3. make sure you are training the new 4D filters and that they are in the list of trainable variable
   3a) make sure the loss is optimized wrt such variables (here)
4. what kind of input are you using? depth in meters? here I am using HHA encoding, which is 3 channels.
   4a) if you normalize depth in meters in the range of RGB this is probably not going to work, you are squeezeing too much. Try to visualized the nromalized data.

Hope this is useful.

[Scienceseb]

Thanks a lot I’m gonna check all those possible issues, when the problem will be solved I will write de solution here!