Input to InstanceNorm does not take Tensor

[gnsmrky]

Trying to convert an Instance Normalization layer but never successfully done so. Did a test script as the following:

package version:
pytorch2keras: 0.1.10
tensorflow: 1.12
pytorch: 0.4.1
numpy: 1.15.1

# pytorch2tf_instancenorm_test.py

import numpy as np
import torch
from torch.autograd import Variable
from pytorch2keras import converter

class Pytorch2KerasTestNet(torch.nn.Module):
    def __init__(self):
        super(Pytorch2KerasTestNet, self).__init__()
        self.conv1 = ConvLayer(3, 32, kernel_size=9, stride=1)
        self.in1 = torch.nn.InstanceNorm2d(32, affine=True)
        self.relu = torch.nn.ReLU()

    def forward(self, x):
        y = self.relu(self.in1(self.conv1(x)))
        return y


class ConvLayer(torch.nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, stride):
        super(ConvLayer, self).__init__()
        reflection_padding = kernel_size // 2
        self.reflection_pad = torch.nn.ReflectionPad2d(reflection_padding)
        self.conv2d = torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride)

    def forward(self, x):
        out = self.reflection_pad(x)
        
        print("conv2d")
        out = self.conv2d(out)
        return out
        
model   = Pytorch2KerasTestNet()

input_np = np.random.uniform(0, 1, (1, 3, 224, 224))
input_var = Variable(torch.FloatTensor(input_np))
k_model = converter.pytorch_to_keras(model, input_var, [(3, 224, 224,)], verbose=True)  

When running it using python pytorch2tf_instancenorm_test.py, it results in the following error:

2018-12-05 20:57:56.505141: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Using TensorFlow backend.
conv2d
graph(%0 : Float(1, 3, 224, 224)
      %1 : Float(32, 3, 9, 9)
      %2 : Float(32)
      %3 : Float(32)
      %4 : Float(32)) {
  %5 : Float(1, 3, 232, 232) = onnx::Pad[mode=reflect, pads=[0, 0, 4, 4, 0, 0, 4, 4]](%0), scope: Pytorch2KerasTestNet/ConvLayer[conv1]/ReflectionPad2d[reflection_pad]
  %6 : Float(1, 32, 224, 224) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[9, 9], pads=[0, 0, 0, 0], strides=[1, 1]](%5, %1, %2), scope: Pytorch2KerasTestNet/ConvLayer[conv1]/Conv2d[conv2d]
  %7 : Dynamic = onnx::Constant[value=<Tensor>](), scope: Pytorch2KerasTestNet/InstanceNorm2d[in1]
  %8 : Dynamic = onnx::Constant[value=<Tensor>](), scope: Pytorch2KerasTestNet/InstanceNorm2d[in1]
  %9 : Float(1, 32, 224, 224) = onnx::InstanceNormalization[epsilon=1e-05](%6, %7, %8), scope: Pytorch2KerasTestNet/InstanceNorm2d[in1]
  %10 : Float(1, 32, 224, 224) = onnx::Relu(%9), scope: Pytorch2KerasTestNet/ReLU[relu]
  return (%10);
}

[10 defined in (%10 : Float(1, 32, 224, 224) = onnx::Relu(%9), scope: Pytorch2KerasTestNet/ReLU[relu]
)]
[%5 : Float(1, 3, 232, 232) = onnx::Pad[mode=reflect, pads=[0, 0, 4, 4, 0, 0, 4, 4]](%0), scope: Pytorch2KerasTestNet/ConvLayer[conv1]/ReflectionPad2d[reflection_pad]
, %6 : Float(1, 32, 224, 224) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[9, 9], pads=[0, 0, 0, 0], strides=[1, 1]](%5, %1, %2), scope: Pytorch2KerasTestNet/ConvLayer[conv1]/Conv2d[conv2d]
, %7 : Dynamic = onnx::Constant[value=<Tensor>](), scope: Pytorch2KerasTestNet/InstanceNorm2d[in1]
, %8 : Dynamic = onnx::Constant[value=<Tensor>](), scope: Pytorch2KerasTestNet/InstanceNorm2d[in1]
, %9 : Float(1, 32, 224, 224) = onnx::InstanceNormalization[epsilon=1e-05](%6, %7, %8), scope: Pytorch2KerasTestNet/InstanceNorm2d[in1]
, %10 : Float(1, 32, 224, 224) = onnx::Relu(%9), scope: Pytorch2KerasTestNet/ReLU[relu]
]
Graph outputs: ['10']
State dict: ['conv1.conv2d.weight', 'conv1.conv2d.bias', 'in1.weight', 'in1.bias']
 ____ 
graph node: Pytorch2KerasTestNet/ConvLayer[conv1]/ReflectionPad2d[reflection_pad]
id: 5
type: onnx::Pad
inputs: ['input0']
outputs: ['Pytorch2KerasTestNet/ConvLayer[conv1]/ReflectionPad2d[reflection_pad]']
name in state_dict: conv1.reflection_pad
attrs: {'mode': 'reflect', 'pads': [0, 0, 4, 4, 0, 0, 4, 4]}
is_terminal: False
Converting padding...
 ____ 
graph node: Pytorch2KerasTestNet/ConvLayer[conv1]/Conv2d[conv2d]
id: 6
type: onnx::Conv
inputs: ['5']
outputs: ['Pytorch2KerasTestNet/ConvLayer[conv1]/Conv2d[conv2d]']
name in state_dict: conv1.conv2d
attrs: {'dilations': [1, 1], 'group': 1, 'kernel_shape': [9, 9], 'pads': [0, 0, 0, 0], 'strides': [1, 1]}
is_terminal: False
Converting convolution ...
 ____ 
graph node: Pytorch2KerasTestNet/InstanceNorm2d[in1]
id: 7
type: onnx::Constant
inputs: []
outputs: ['Pytorch2KerasTestNet/InstanceNorm2d[in1]']
name in state_dict: in1
attrs: {'value': tensor([0.8979, 0.3156, 0.1349, 0.2346, 0.8874, 0.7174, 0.3563, 0.0794, 0.9949,
        0.6409, 0.7893, 0.0516, 0.6400, 0.9812, 0.6296, 0.0555, 0.8294, 0.1040,
        0.7664, 0.8627, 0.5198, 0.5813, 0.6676, 0.4292, 0.9838, 0.8877, 0.0068,
        0.0730, 0.9636, 0.5794, 0.4843, 0.7476])}
is_terminal: False
Converting constant ...
 ____ 
graph node: Pytorch2KerasTestNet/InstanceNorm2d[in1]
id: 8
type: onnx::Constant
inputs: []
outputs: ['Pytorch2KerasTestNet/InstanceNorm2d[in1]']
name in state_dict: in1
attrs: {'value': tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])}
is_terminal: False
Converting constant ...
 ____ 
graph node: Pytorch2KerasTestNet/InstanceNorm2d[in1]
id: 9
type: onnx::InstanceNormalization
inputs: ['6', '7', '8']
outputs: ['Pytorch2KerasTestNet/InstanceNorm2d[in1]']
name in state_dict: in1
attrs: {'epsilon': 1e-05}
is_terminal: False
Converting instancenorm ...
Traceback (most recent call last):
  File "pytorch2tf_instancenorm_test.py", line 45, in <module>
    k_model = converter.pytorch_to_keras(model, input_var, [(3, 224, 224,)], verbose=True)  
  File "/home/myang/py3env/local/lib/python3.6/site-packages/pytorch2keras/converter.py", line 316, in pytorch_to_keras
    names
  File "/home/myang/py3env/local/lib/python3.6/site-packages/pytorch2keras/layers.py", line 653, in convert_instancenorm
    layers[scope_name] = lambda_layer(layers[inputs[0]])
  File "/home/myang/py3env/local/lib/python3.6/site-packages/keras/engine/base_layer.py", line 457, in __call__
    output = self.call(inputs, **kwargs)
  File "/home/myang/py3env/local/lib/python3.6/site-packages/keras/layers/core.py", line 687, in call
    return self.function(inputs, **arguments)
  File "/home/myang/py3env/local/lib/python3.6/site-packages/pytorch2keras/layers.py", line 646, in target_layer
    param_initializers={'beta': tf.constant_initializer(beta), 'gamma': tf.constant_initializer(gamma)},
  File "/home/myang/py3env/local/lib/python3.6/site-packages/tensorflow/python/ops/init_ops.py", line 208, in __init__
    "tuple of values, or numpy.ndarray)." % type(value))
TypeError: Invalid type for initial value: <class 'tensorflow.python.framework.ops.Tensor'> (expected Python scalar, list or tuple of values, or numpy.ndarray).

The code is an excerpt from PyTorch Fast Style Transfer. Is this related to #37? Really trying to get it to work...

[gmalivenko]

Hello @gnsmrky!
Yes, there was a problem related to instancenorm2d. Can you check the latest version (pip or git master) of the converter?

[gnsmrky]

Hi @nerox8664,

Updated using 'pip install pytorch2keras --upgrade' to update the package from v0.1.10 to the just-released v0.1.11. It now works correctly! Thanks a lot!

Posting the good result here for reference:

conv2d
graph(%0 : Float(1, 3, 224, 224)
      %1 : Float(32, 3, 9, 9)
      %2 : Float(32)
      %3 : Float(32)
      %4 : Float(32)) {
  %5 : Float(1, 3, 232, 232) = onnx::Pad[mode=reflect, pads=[0, 0, 4, 4, 0, 0, 4, 4]](%0), scope: Pytorch2KerasTestNet/ConvLayer[conv1]/ReflectionPad2d[reflection_pad]
  %6 : Float(1, 32, 224, 224) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[9, 9], pads=[0, 0, 0, 0], strides=[1, 1]](%5, %1, %2), scope: Pytorch2KerasTestNet/ConvLayer[conv1]/Conv2d[conv2d]
  %7 : Dynamic = onnx::Constant[value=<Tensor>](), scope: Pytorch2KerasTestNet/InstanceNorm2d[in1]
  %8 : Dynamic = onnx::Constant[value=<Tensor>](), scope: Pytorch2KerasTestNet/InstanceNorm2d[in1]
  %9 : Float(1, 32, 224, 224) = onnx::InstanceNormalization[epsilon=1e-05](%6, %7, %8), scope: Pytorch2KerasTestNet/InstanceNorm2d[in1]
  %10 : Float(1, 32, 224, 224) = onnx::Relu(%9), scope: Pytorch2KerasTestNet/ReLU[relu]
  return (%10);
}

[10 defined in (%10 : Float(1, 32, 224, 224) = onnx::Relu(%9), scope: Pytorch2KerasTestNet/ReLU[relu]
)]
[%5 : Float(1, 3, 232, 232) = onnx::Pad[mode=reflect, pads=[0, 0, 4, 4, 0, 0, 4, 4]](%0), scope: Pytorch2KerasTestNet/ConvLayer[conv1]/ReflectionPad2d[reflection_pad]
, %6 : Float(1, 32, 224, 224) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[9, 9], pads=[0, 0, 0, 0], strides=[1, 1]](%5, %1, %2), scope: Pytorch2KerasTestNet/ConvLayer[conv1]/Conv2d[conv2d]
, %7 : Dynamic = onnx::Constant[value=<Tensor>](), scope: Pytorch2KerasTestNet/InstanceNorm2d[in1]
, %8 : Dynamic = onnx::Constant[value=<Tensor>](), scope: Pytorch2KerasTestNet/InstanceNorm2d[in1]
, %9 : Float(1, 32, 224, 224) = onnx::InstanceNormalization[epsilon=1e-05](%6, %7, %8), scope: Pytorch2KerasTestNet/InstanceNorm2d[in1]
, %10 : Float(1, 32, 224, 224) = onnx::Relu(%9), scope: Pytorch2KerasTestNet/ReLU[relu]
]
Graph outputs: ['10']
State dict: ['conv1.conv2d.weight', 'conv1.conv2d.bias', 'in1.weight', 'in1.bias']
 ____ 
graph node: Pytorch2KerasTestNet/ConvLayer[conv1]/ReflectionPad2d[reflection_pad]
id: 5
type: onnx::Pad
inputs: ['input0']
outputs: ['Pytorch2KerasTestNet/ConvLayer[conv1]/ReflectionPad2d[reflection_pad]']
name in state_dict: conv1.reflection_pad
attrs: {'mode': 'reflect', 'pads': [0, 0, 4, 4, 0, 0, 4, 4]}
is_terminal: False
Converting padding...
 ____ 
graph node: Pytorch2KerasTestNet/ConvLayer[conv1]/Conv2d[conv2d]
id: 6
type: onnx::Conv
inputs: ['5']
outputs: ['Pytorch2KerasTestNet/ConvLayer[conv1]/Conv2d[conv2d]']
name in state_dict: conv1.conv2d
attrs: {'dilations': [1, 1], 'group': 1, 'kernel_shape': [9, 9], 'pads': [0, 0, 0, 0], 'strides': [1, 1]}
is_terminal: False
Converting convolution ...
 ____ 
graph node: Pytorch2KerasTestNet/InstanceNorm2d[in1]
id: 7
type: onnx::Constant
inputs: []
outputs: ['Pytorch2KerasTestNet/InstanceNorm2d[in1]']
name in state_dict: in1
attrs: {'value': tensor([0.4023, 0.7445, 0.4929, 0.0314, 0.6642, 0.4988, 0.6340, 0.6122, 0.1814,
        0.4681, 0.0292, 0.4813, 0.8027, 0.0029, 0.9860, 0.3159, 0.6080, 0.1868,
        0.6314, 0.1304, 0.5272, 0.4891, 0.1304, 0.3229, 0.0008, 0.3625, 0.3277,
        0.4975, 0.8130, 0.7173, 0.9650, 0.1704])}
is_terminal: False
Converting constant ...
 ____ 
graph node: Pytorch2KerasTestNet/InstanceNorm2d[in1]
id: 8
type: onnx::Constant
inputs: []
outputs: ['Pytorch2KerasTestNet/InstanceNorm2d[in1]']
name in state_dict: in1
attrs: {'value': tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])}
is_terminal: False
Converting constant ...
 ____ 
graph node: Pytorch2KerasTestNet/InstanceNorm2d[in1]
id: 9
type: onnx::InstanceNormalization
inputs: ['6', '7', '8']
outputs: ['Pytorch2KerasTestNet/InstanceNorm2d[in1]']
name in state_dict: in1
attrs: {'epsilon': 1e-05}
is_terminal: False
Converting instancenorm ...
 ____ 
graph node: Pytorch2KerasTestNet/ReLU[relu]
id: 10
type: onnx::Relu
inputs: ['9']
outputs: ['Pytorch2KerasTestNet/ReLU[relu]']
name in state_dict: relu
attrs: {}
is_terminal: True
Converting relu ...
Your model was (probably) successfully converted! Please, follow the repository https://github.com/nerox8664/pytorch2keras and give a star :)
<keras.engine.training.Model object at 0x7f7d174e3ac8>

Going to do some more works to see if converted keras model works correctly.