thoughts on training with other in-domain datasets, before fine-tuning on actual dataset

[roybenhayun]

Hi,
there is an interesting statement in section 4.3 "Dataset benchmarks" of the TorchGeo paper:

The in-domain pre-training method in Neumann et al. [42] trains models starting from ImageNet weights, then further trains on remote sensing (in domain) datasets, before fine-tuning on the actual target dataset, which performs better than simply starting from ImageNet weights.

we are currently training a semantic segmentation model to which we will have a limited number of labels. so we have to think about how to compensate for lack of a significant number of ground truth data.

at the time of writing this post we have two steps:

1. start from imagenet and the below setup:
   task = SemanticSegmentationTask(
   model="unet",
   backbone="resnext50_32x4d",
   weights="imagenet",
   ...
2. train the model on the limited number of labels we have

we hope to get more labels in the future.

for now, would be happy to understand if we could do another "intermediate" training on "similar in-domain dataset".

the above article statement could mean that it is possible, and even recommended. then the steps would be:

1. start from imagenet and the above setup
2. train on other remote sensing in-domain dataset
3. fine-tuning on the actual limited size target dataset

could it be any other remote-sensing semantic segmentation e.g., buildings or else, could some datasets actually deteriorate the training e.g., a very different geographic region, what is similar enough or would any remote-sensing stuff be good, etc etc. still generally thinking about the statement and how to implement it methodically.

any suggestion, advice, tips and questions would be welcomed!

thanks in advance.

[calebrob6]

Hey @roybenhayun, that could definitely help -- I always recommend setting up the train/test pipeline with the limited labels you currently have so you can understand the baseline performance first before trying to do more complicated things like domain transfer. Depending on your task, the imagery, and what you expect the model to do, you might find that few labels are actually fine :)

[roybenhayun]

thanks @calebrob6 , good point on setting a baseline performance. working on establishing it now.

as for the intermediate transfer learning:

looking again at the the semantic segmentation example you did, it might be that the amount of labels there, are more or less what we would get also. and the segments to identify will be challenging to identify due to very subtle data so an effective pre-training could be critical for high success rates.
let's say that if that example was our end goal and actual dataset, how and with what should we train the model before?
any specific example dataset and labels for intermediate training that would improve specifically this example results? and how would it be done?

thanks!

[calebrob6]

I'm not sure without more details about the task you are actually working on and examples of the imagery. Generally, the closer the pre-training task is to the domain task the better (provided you have enough pre-training data)