Convnext and SwinT backbones support loading pre-trained ImageNet weights from torchvision.models repository. To initialize the model with these weights, we use the model_config.pretrained_weights parameter. However, since this pretrained_weights parameter is only supported for convnext and swint backbones, it would be more structures to move it under model_config.convnext and model_config.swint sections.