KeyError: 'module.weight'

[I-C-Karakozis]

Describe the bug
I am trying to convert a simple Pytorch Gender Classifier to Keras (model is provided below). Whenever I run model = pytorch_to_keras(model, orig_img, [np.shape(orig_img.cpu().numpy())], names='short'), I get

Traceback (most recent call last):
File "main.py", line 154, in
main(args)
File "main.py", line 92, in main
model = pytorch_to_keras(model, orig_img, [np.shape(orig_img.cpu().numpy())], names='short')
File "/mnt/c/Users/Yannis/CIFAR10S/real_world_experiments/CEM/venv/lib/python3.6/site-packages/pytorch2keras/converter.py", line 325, in pytorch_to_keras
names
File "/mnt/c/Users/Yannis/CIFAR10S/real_world_experiments/CEM/venv/lib/python3.6/site-packages/pytorch2keras/convolution_layers.py", line 35, in convert_conv
if len(weights[weights_name].numpy().shape) == 5: # 3D conv
KeyError: 'module.weight'

To Reproduce
Here is my pytorch model:

class resnet_modified_small(nn.Module):
    def base_size(self): return 512
    def rep_size(self): return 1024

    def __init__(self, n_classes):
        super(resnet_modified_small, self).__init__()
        self.resnet = tv.models.resnet34(pretrained=True)

        # define layers
        self.n_classes = n_classes
        self.linear = nn.Linear(7 * 7 * self.base_size(), self.rep_size())
        self.cls = nn.Linear(self.rep_size(), self.n_classes)
        self.dropout2d = nn.Dropout2d(.5)
        self.dropout = nn.Dropout(.5)
        self.relu = nn.LeakyReLU()
        initLinear(self.linear)

    def forward(self, out0):
        x = self.resnet.conv1(out0)
        x = self.resnet.bn1(x)
        x = self.resnet.relu(x)
        out1 = self.resnet.maxpool(x)

        out2 = self.resnet.layer1(out1)
        out3 = self.resnet.layer2(out2)
        out4 = self.resnet.layer3(out3)
        out5 = self.resnet.layer4(out4)
     
        x = self.dropout2d(out5)

        features = self.dropout(self.relu(self.linear(x.view(-1, 7*7*self.base_size()))))
        cls_scores = self.cls(features)
        return [out0, out1, out2, out3, out4, out5, features, cls_scores]

Environment:

• OS: Bash for Windows
• Python 3.6
• Pytorch 0.4.1

[gmalivenko]

Hi, @I-C-Karakozis.
Here are many problems in the converter related to your model. All nested modules (ModuleList, big submodules, etc) are very complex to trace. Actually I plan to refactor the tracing part, but a bit later.
So, If you want to convert your model right now:

1. Set resnet layers as a submodules
2. Don't return an input.

class resnet_modified_small(nn.Module):
    def base_size(self): return 512
    def rep_size(self): return 1024

    def __init__(self, n_classes):
        super(resnet_modified_small, self).__init__()
        self.resnet = tv.models.resnet34(pretrained=True)
        self.conv1 = self.resnet.conv1
        self.bn1 = self.resnet.bn1
        self.layer1 = self.resnet.layer1
        self.layer2 = self.resnet.layer2
        self.layer3 = self.resnet.layer3
        self.layer4 = self.resnet.layer4

        # define layers
        self.n_classes = n_classes
        self.linear = nn.Linear(7 * 7 * self.base_size(), self.rep_size())
        self.cls = nn.Linear(self.rep_size(), self.n_classes)
        self.dropout2d = nn.Dropout2d(.5)
        self.dropout = nn.Dropout(.5)
        self.relu = nn.LeakyReLU()

    def forward(self, out0):
        x = self.conv1(out0)
        x = self.bn1(x)
        x = self.resnet.relu(x)
        out1 = self.resnet.maxpool(x)

        out2 = self.layer1(out1)
        out3 = self.layer2(out2)
        out4 = self.layer3(out3)
        out5 = self.layer4(out4)
     
        x = self.dropout2d(out5)

        features_ = self.relu(self.linear(x.view(-1, 7*7*self.base_size())))
        features = self.dropout(features_.clone())
        cls_scores = self.cls(features.clone())
        return [out1, out2, out3, out4, out5, features, cls_scores]

[I-C-Karakozis]

@nerox8664 Thanks a lot for the incredibly quick response. The fix you suggested seemed to improve things, but I now get the following error:

Converting batchnorm ...
Converting relu ...
Converting pooling ...
Converting convolution ...
Converting batchnorm ...
Converting relu ...
Converting convolution ...
Converting batchnorm ...
Converting elementwise_add ...
Converting relu ...
Traceback (most recent call last):
File "main.py", line 154, in
main(args)
File "main.py", line 92, in main
model = pytorch_to_keras(model, orig_img, [np.shape(orig_img.cpu().numpy())[1:]], names='short')
File "/mnt/c/Users/Yannis/CIFAR10S/real_world_experiments/CEM/venv/lib/python3.6/site-packages/pytorch2keras/converter.py", line 325, in pytorch_to_keras
names
File "/mnt/c/Users/Yannis/CIFAR10S/real_world_experiments/CEM/venv/lib/python3.6/site-packages/pytorch2keras/activation_layers.py", line 32, in convert_relu
layers[scope_name] = relu(layers[inputs[0]])
IndexError: list index out of range

Any tips for how to go about this?

[gmalivenko]

The problem may be caused by wrongly called converter function. Could you please try to convert your model with something like:

input_np = np.random.uniform(0, 1, (1, 3, 224, 224))
input_var = Variable(torch.FloatTensor(input_np))
output = model(input_var)
k_model = pytorch_to_keras(model, input_var, (3, 224, 224,), verbose=True)

[I-C-Karakozis]

Same error:

Converting batchnorm ...
Converting relu ...
Converting pooling ...
Converting convolution ...
Converting batchnorm ...
Converting relu ...
Converting convolution ...
Converting batchnorm ...
Converting elementwise_add ...
Converting relu ...
Traceback (most recent call last):
File "main.py", line 160, in
main(args)
File "main.py", line 95, in main
model = pytorch_to_keras(model, input_var, (3, 224, 224), names='short')
File "/mnt/c/Users/Yannis/CIFAR10S/real_world_experiments/CEM/venv/lib/python3.6/site-packages/pytorch2keras/converter.py", line 325, in pytorch_to_keras
names
File "/mnt/c/Users/Yannis/CIFAR10S/real_world_experiments/CEM/venv/lib/python3.6/site-packages/pytorch2keras/activation_layers.py", line 32, in convert_relu
layers[scope_name] = relu(layers[inputs[0]])
IndexError: list index out of range

[gmalivenko]

@I-C-Karakozis could you please send a whole python script?

[I-C-Karakozis]

Of course here is my main.py:

def main(args):
    # set DEBUG parametrization
    if args.debug:
        args.batch_size = 1
        args.maxiter = 10
        args.binary_steps = 1
    assert(args.batch_size == 1)

    # seed setup
    random.seed(121)
    np.random.seed(1211)

    # argument parsing
    arg_max_iter = args.maxiter
    arg_b = args.binary_steps
    arg_init_const = args.init_const
    arg_mode = args.mode
    arg_kappa = args.kappa
    arg_beta = args.beta
    arg_gamma =args.gamma

    # load data and prediction model
    trainloader, devloader, testloader, encoder = COCO_dataloaders(args.prefix, args, gender_cls=True, path='../COCO/')
    n_classes = encoder.n_genders()
    model, device = load_net(n_classes, gender_cls=True)
    model.load_state_dict(torch.load(os.path.join("../COCO/models", "obj_mlc_gender_cls_debug"), map_location=device))
    model.eval()

    # get original image, groudntruth label, predicted label, and predicted label softmax score 
    batch = next(iter(testloader))
    idx, orig_img, targets = batch[0].data.cpu().numpy()[0], batch[1], batch[2]
    orig_prob, orig_class, orig_prob_str = torch_model_prediction(model, orig_img, device)
    target = np.zeros(n_classes)
    target[orig_class] = 1
    print("Image:{}, inferred label:{}".format(idx, orig_class))

    # convert model to keras-tensorflow
    batch_shape = np.shape(orig_img.cpu().numpy())
    input_np = np.random.uniform(0, 1, (1, 3, 224, 224))
    input_var = torch.autograd.Variable(torch.FloatTensor(input_np))
    output = model(input_var)
    model = pytorch_to_keras(model, input_var, (3, 224, 224), names='short')
    orig_img = pytorch_tensor_to_tf(orig_img)

Here is my load_net():

def load_net(n_classes, gender_cls=False, small=False, medium=False, 
             conditional_small=False, conditional_medium=False):
    # load network architecture
    assert(sum([gender_cls, small, medium, conditional_small, conditional_medium])==1)
    if gender_cls:  net = network.resnet_gender_cls(n_classes)
    if small:       net = network.resnet_modified_small(n_classes)
    if medium:      net = network.resnet_modified_medium(n_classes)
    if conditional_small: net = network.conditional_resnet_modified_small(n_classes)
    if conditional_medium: net = network.conditional_resnet_modified_medium(n_classes)

    # determine processing device
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    net = net.to(device)
    if device == 'cuda':
        if not gender_cls: net = torch.nn.DataParallel(net)
        cudnn.benchmark = True

    return net, device

network.resnet_gender_cls(n_classes) calls the model class you gave me. The code breaks at the very last line. You can ignore the dataloading components of my code. Let me know if anything is unclear. The model I am loading was built in GPU, but I am currently testing on a CPU (maybe that causes the problem?) to ensure the components work before submitting my code to my lab's GPU queue.

EDIT: Here is also my torch_model_prediction():

def torch_model_prediction(model, inputs, device):
    with torch.no_grad():
        prob = model(inputs.to(device))
        _, labels = prob.max(1)
        predicted_class = labels.cpu().numpy()[0]
        prob = prob.cpu().numpy()
        prob_str = np.array2string(prob).replace('\n','')

    return prob, predicted_class, prob_str

[I-C-Karakozis]

@nerox8664 I also tried replacing load_net() with the following so that I use the torch built-in resnet and I got the same issue:

def load_net(n_classes, gender_cls=False, small=False, medium=False, 
             conditional_small=False, conditional_medium=False):
    # load network architecture
    assert(sum([gender_cls, small, medium, conditional_small, conditional_medium])==1)
    # if gender_cls:  net = network.resnet_gender_cls(n_classes)
    if gender_cls:  
        net = torchvision.models.resnet34(pretrained=True)
        num_ftrs = net.fc.in_features
        net.fc = nn.Linear(num_ftrs, n_classes)
    if small:       net = network.resnet_modified_small(n_classes)
    if medium:      net = network.resnet_modified_medium(n_classes)
    if conditional_small: net = network.conditional_resnet_modified_small(n_classes)
    if conditional_medium: net = network.conditional_resnet_modified_medium(n_classes)

    # determine processing device
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    net = net.to(device)
    if device == 'cuda':
        if not gender_cls: net = torch.nn.DataParallel(net)
        cudnn.benchmark = True

    return net, device

[I-C-Karakozis]

@nerox8664 When I checked the verbose error output, I noticed the following:

---

graph node:
node id: 280
type: onnx::Add
inputs: ['278', '243']
outputs: []
name in state_dict:
attrs: {}
is_terminal: False
Converting elementwise_add ...

---

graph node: ResNet/Sequential[layer1]/BasicBlock[0]/ReLU[relu]
node id: 283
type: onnx::Relu
inputs: []
outputs: ['ResNet/Sequential[layer1]/BasicBlock[0]/ReLU[relu]']
name in state_dict: layer1.0.relu
attrs: {}
is_terminal: False

The top node seems to not have an output and the bottom node seems to not have an input, which seems to be causing the error.

[gmalivenko]

@I-C-Karakozis have you tried to modify the model? Please, show me the latest version.

[I-C-Karakozis]

@nerox8664 here: (the only thing I changed was the naming of the intermediate variables and I am only returning the final output)

class resnet_gender_cls(nn.Module):
    def base_size(self): return 512
    def rep_size(self): return 1024

    def __init__(self, n_classes):
        super(resnet_gender_cls, self).__init__()
        self.resnet = tv.models.resnet34(pretrained=True)
        self.conv1 = self.resnet.conv1
        self.bn1 = self.resnet.bn1
        self.layer1 = self.resnet.layer1
        self.layer2 = self.resnet.layer2
        self.layer3 = self.resnet.layer3
        self.layer4 = self.resnet.layer4

        # define layers
        self.n_classes = n_classes
        self.linear = nn.Linear(7 * 7 * self.base_size(), self.rep_size())
        self.cls = nn.Linear(self.rep_size(), self.n_classes)
        self.dropout2d = nn.Dropout2d(.5)
        self.dropout = nn.Dropout(.5)
        self.relu = nn.LeakyReLU()

    def forward(self, out0):
        x = self.conv1(out0)
        x = self.bn1(x)
        x = self.resnet.relu(x)
        x = self.resnet.maxpool(x)

        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)
     
        x = self.dropout2d(x)

        x = self.relu(self.linear(x.view(-1, 7*7*self.base_size())))
        x = self.dropout(x.clone())
        cls_scores = self.cls(x.clone())
        return cls_scores

PS: Thanks a lot for all the help. I really appreciate it. This is a major blocker in my project.

[gmalivenko]

I've just converted the model without a problem. So, have you tried the latest converter version? Please, upgrade the package with pip install --upgrade pytorch2keras, I published the release 5 minutes ago. I hope it will help.

[I-C-Karakozis]

It worked. You are the best @nerox8664 . Have a wonderful day!