How Do I fine-tune a custom DETR model as we fine-tune Hugging face transformer

[tanulsingh]

I am doing something like

class CustomDETR(nn.Module):

    def __init__(self):
        super(CustomDETR,self).__init__()
        self.model = torch.hub.load('facebookresearch/detr', 'detr_resnet50', pretrained=True)
        self.model.class_embed = nn.Linear(in_features=256,out_features=2)
        self.model.num_queries = 100
    
    def forward(self,):
        output = self.model()

My question is the output of the model is class and bboxes and no loss , how do I calculate the loss and backpropagate it so that I can fine tune the model

[fmassa]

Hi,

There is a lot of useful information about finetuning in #9, so let's keep the discussion there.

For your particular question, you should also add a SetCriterion, which takes the predictions and the ground-truths and outputs a loss, see

detr/models/detr.py

Lines 83 to 88 in 10a2c75

	class SetCriterion(nn.Module):
	""" This class computes the loss for DETR.
	The process happens in two steps:
	1) we compute hungarian assignment between ground truth boxes and the outputs of the model
	2) we supervise each pair of matched ground-truth / prediction (supervise class and box)
	"""

Our training loop looks like the following

detr/engine.py

Lines 32 to 33 in 10a2c75

	outputs = model(samples)
	loss_dict = criterion(outputs, targets)

IMO, the easiest way to get started with training is to clone the repo and modify the functions you need, see https://github.com/facebookresearch/detr#training for information.

I hope this helps.

I believe I've answered your question, and as such I'm closing this issue, but let us know if you have further questions

[tanulsingh]

@fmassa I was able to import HungarianMatcher from detr.models.matcher and it was working fine , but now it says module util not found after the last update I guess , I wrote a whole pipeline and somehow now it doesn't work , I would really appreciate some help

[fmassa]

@tanulsingh there should be no breaking change in the repo since its release. One thing to keep in mind is that you need to run the code from the repo root, as DETR is not packaged as a library

[tanulsingh]

@fmassa Thanks It helped I was able to write a complete pipeline that can be used to train DETR on any custom dataset just by changing the dataloader
https://www.kaggle.com/tanulsingh077/end-to-end-object-detection-with-transformers-detr
I would be glad if you share your views/suggestions on it , that I can improve
Thanks a lot