[Ask for advice] Modify models by adding skip connections

[crossknight]

Hi,

I need to modify models by adding skip connections between encoder layers and decoder layers like this

x = input  
x = self.layer0[-1](x)  
x = x + input  
x1 = self.layer1(x)  
x1 = x1 + x  
x2 = self.layer2(x1)  
x2 = x2 + x1  
x3 = self.layer3(x2)  
x3 = x3 + x2  
x4 = self.layer4(x3)  
x4 = x4 + x3  

I've tried implementing this and encountered the problem which dimensions after and before passing layers are unequal and the values and can't be added.

Is it possible to implement this? and How to implement it?

Thank you.

[qubvel]

Hi @crossknight
Add a conv2d layer with 1x1 kernel to make them equal (look at resnet architecture).

[crossknight]

Hi @qubvel,

Thanks for your reply. I tried your solution and It seems to be working fine but I'm sure that I implemented it correctly. Could you please review my codes?

Encoder

def forward(self, x):
        x0 = self.conv1(x)
        x0 = self.bn1(x0)
        x0 = self.relu(x0)

        x1 = self.maxpool(x0)

       out_tensor = x1

        x1 = self.layer1(x1)

        in_tensor = x1
        x1 = x1 + out_tensor.to('cuda')

        x2 = self.layer2(x1)

        conv2 = nn.Conv2d(64, 128, 1).to('cuda')
        maxpool2 = nn.MaxPool2d(2).to('cuda')
        out_tensor = conv2(in_tensor)
        out_tensor = maxpool2(out_tensor)
        in_tensor = x2
        x2 = x2 + out_tensor.to('cuda')

        x3 = self.layer3(x2)

        conv3 = nn.Conv2d(128, 256, 1).to('cuda')
        maxpool3 = nn.MaxPool2d(2).to('cuda')
        out_tensor = conv3(in_tensor)
        out_tensor = maxpool3(out_tensor)
        in_tensor = x3
        x3 = x3 + out_tensor.to('cuda')

        x4 = self.layer4(x3)

        conv4 = nn.Conv2d(256, 512, 1).to('cuda')
        maxpool4 = nn.MaxPool2d(2).to('cuda')
        out_tensor = conv4(in_tensor)
        out_tensor = maxpool4(out_tensor)
        x4 = x4 + out_tensor.to('cuda’)

        return [x4, x3, x2, x1, x0]

Decoder

def forward(self, x):
        encoder_head = x[0]
        skips = x[1:]

        in_tensor = encoder_head
        m = nn.ConstantPad2d((8, 8, 8, 8), 0)
        out_tensor = self.conv1(in_tensor)
        out_tensor = m(out_tensor)

        x = self.layer1([encoder_head, skips[0]])

        in_tensor = x
        x = x + out_tensor.to('cuda')
        m = nn.ConstantPad2d((16, 16, 16, 16), 0)
        out_tensor = self.conv2(in_tensor)
        out_tensor = m(out_tensor)

        x = self.layer2([x, skips[1]])

        in_tensor = x
        x = x + out_tensor.to('cuda')
        m = nn.ConstantPad2d((32, 32, 32, 32), 0)
        out_tensor = self.conv3(in_tensor)
        out_tensor = m(out_tensor)

        x = self.layer3([x, skips[2]])

        in_tensor = x
        x = x + out_tensor.to('cuda')
        m = nn.ConstantPad2d((64, 64, 64, 64), 0)
        out_tensor = self.conv4(in_tensor)
        out_tensor = m(out_tensor)

        x = self.layer4([x, skips[3]])

        in_tensor = x
        x = x + out_tensor.to('cuda')
        m = nn.ConstantPad2d((128, 128, 128, 128), 0)
        out_tensor = self.conv5(in_tensor)
        out_tensor = m(out_tensor)

        x = self.layer5([x, None])

        x = x + out_tensor.to('cuda')

        x = self.final_conv(x)

        return x

[qubvel]

Hi @crossknight
It is hard to read your code, look at some style guidlines/examples how to use pytorch to create a model in pytorch.
Why did you pad tensors?
Why did you move them to cuda in forward?

Make it step by step, print tensor shapes, use some tool for network architecture visualization (e.g. hiddenlayers)

[crossknight]

Hi @qubvel

Why did you pad tensors?

• (Decoder) because the 3rd,4th dimension of tensor are not the same size after passed the layer

Why did you move them to cuda in forward?

• because it will shows error about data are not on the same gpu/cpu