Feature map with 0 channels? mit_b3 SegFormer

[omarequalmars]

I have an issue where I'm running this part:

`import segmentation_models_pytorch as smp
import torch

model = smp.from_pretrained(r'modelsaves\Teacher\mit_b3_Segformer')

out = model.encoder(torch.rand(1,2,512,512))
print([s.size() for s in out])`

In which I isolated the error, and I get this:
[torch.Size([1, 2, 512, 512]), torch.Size([1, 0, 256, 256]), torch.Size([1, 64, 128, 128]), torch.Size([1, 128, 64, 64]), torch.Size([1, 320, 32, 32]), torch.Size([1, 512, 16, 16])]

For some reason the 2nd Feature Map has 0 channels? How could this be? I have tried this with other encoders and model archs and I don't get this. Is there something I am missing?

=> Note, the model.safetensors is saved with an earlier version of SMP, I tried with 0.3.5 version, then updated to the latest and the issue persists.

I tried this:

model = smp.from_pretrained(r'modelsaves\Teacher\mit_b3_Segformer')

out = model.encoder(torch.rand(1,2,512,512))
print([s.size() for s in out])

model = smp.Segformer(encoder = 'mit_b3', in_channels = 2, classes = 3, encoder_weights = None)
# model = model.from_pretrained(r'modelsaves\Teacher\mit_b3_Segformer')
out = model.encoder(torch.rand(1,2,512,512))
print([s.size() for s in out])

Got this

[torch.Size([1, 2, 512, 512]), torch.Size([1, 0, 256, 256]), torch.Size([1, 64, 128, 128]), torch.Size([1, 128, 64, 64]), torch.Size([1, 320, 32, 32]), torch.Size([1, 512, 16, 16])]
[torch.Size([1, 2, 512, 512]), torch.Size([1, 64, 256, 256]), torch.Size([1, 64, 128, 128]), torch.Size([1, 128, 64, 64]), torch.Size([1, 256, 32, 32]), torch.Size([1, 512, 16, 16])]

So the issue is for sure in the saved model.safetensors .
I would like to understand how I can troubleshoot something like this

[qubvel]

Hi, @omarequalmars! This is intended to be this way, the reason is that Mix Vision Transformer (which is backbone of segformer) does not have features for H//2 W//2 resolution, thus a dummy feature map is created with 0 channels for compatibility of the backbone (encoder) with other model architectures. However, for SegFormer this feature is not used somehow.

[qubvel]

This dummy feature with 0 channels was there from the beginning, and does not change with newer versions. But let me know if you found a difference

[omarequalmars]

Alright, did not know that, thanks for the heads-up @qubvel. In that case, would the "fresh" model's version of this feature map be false? Specifically this:

[torch.Size([1, 2, 512, 512]), torch.Size([1, 0, 256, 256]), torch.Size([1, 64, 128, 128]), torch.Size([1, 128, 64, 64]), torch.Size([1, 320, 32, 32]), torch.Size([1, 512, 16, 16])] [torch.Size([1, 2, 512, 512]), torch.Size([1, 64, 256, 256]), torch.Size([1, 64, 128, 128]), torch.Size([1, 128, 64, 64]), torch.Size([1, 256, 32, 32]), torch.Size([1, 512, 16, 16])]

[qubvel]

I'm getting 0 channels for fresh model as well:

import torch
import segmentation_models_pytorch as smp

model = smp.Segformer(encoder_name="mit_b3", in_channels=2, classes=3, encoder_weights=None)
out = model.encoder(torch.rand(1,2,512,512))

print([s.size() for s in out])

[torch.Size([1, 2, 512, 512]), torch.Size([1, 0, 256, 256]), torch.Size([1, 64, 128, 128]), torch.Size([1, 128, 64, 64]), torch.Size([1, 320, 32, 32]), torch.Size([1, 512, 16, 16])]

[omarequalmars]

Just checked again, I got the 0 channel this time. Really sketchy. It must have been a mistake on my behalf. Sorry!

Thanks for the help, @qubvel! Hoping to make it up someday :)

[qubvel]

No worries, thanks for opening an issue :)