What happened when using pretrained weight but with mismatched bands

[vinson2233]

Hi, right now I want to adjust the Prithvi on RGB aerial images.

I understand that Prithvi expects 6 different bands. But when I specify backbone_pretrained=True and RGB Bands as input, the model works fine. It simply just give me this when I load the image :

model = model_factory.build_model(task="regression",
        backbone="prithvi_vit_100",
        decoder="FCNDecoder",
        backbone_bands=[
            HLSBands.RED,
            HLSBands.GREEN,
            HLSBands.BLUE,
        ],
        # necks=[
        #        {"name": "ReshapeTokensToImage"}],
        backbone_pretrained=True,
        backbone_num_frames=1,
        decoder_channels=128,
        head_dropout=0.2
    )

INFO:timm.models._builder:Loading pretrained weights from Hugging Face hub (('ibm-nasa-geospatial/Prithvi-100M', 'Prithvi_100M.pt'))

I can input images with 3 channels, get the result, and do some backpropagation with RGB images (with freezing the encoder).
My question is, why this is working ? Should I be worry ? Should I also retrain the encoder part for my downstream task ?

Thank you

• The version I'm using is the installation from the repo directly

[Joao-L-S-Almeida]

Hi, @vinson2233 Sorry by the delay.
If you look terratorch/models/encoder_decoder_factory.py:73, you will that this key will enter as kwargs. As it is filtered as argument for the backbone, it's saved in backbone_kwargs as bands. The backbone_kwargs dictionary is used in some moments to collect variables, but it seems this variable is never explicitly used. So I suspect the backone is using the default config.