NEW Feature: DeTR Model to torchvision

[ambujpawar]

🚀 The feature

Adding the first Transformer-based detection model to Torchvision. A draft PR was submitted regarding the model: #5922
but a Github Issue regarding it did not exist. Therefore creating it now.

Paper: here
Official implementation: here

Pinging @xiaohu2015 and @deepwilson if you guys would like to contribute

Motivation, pitch

First Transformer-based detection model to Torchvision, which has been missing until now in torchvision.

Alternatives

No response

Additional context

No response

[oke-aditya]

One problem I do see is that DETR at one step uses scipy.linear_assigment.

How are we going to do that in torchvision? Scipy is not a hard dependency for us

cc @datumbox

[ambujpawar]

Thanks, I did not know that.
Perhaps, linear_sum_assignment from scipy.optimize has to be ported to torch?

[oke-aditya]

Officially I don't think so. But we can try to port the scipy code

https://github.com/scipy/scipy/blob/v0.18.1/scipy/optimize/_hungarian.py

I can help you with DETR I have some experience on using it.

[oke-aditya]

My other concern is the background class choice

In current torchvision models 0 label is considered as background. But I guess for DETR label 0 is not background.
I don't remember exactly. I need to see a bit.

[datumbox]

I think given that the original repo uses scipy.optimize.linear_sum_assignment only for training, it's OK to keep it as optional dependency. This means that we can still make the model work without it during inference and thus be jit-scriptable. Currently there are no plans to adding it on core, and give that Scipy's implementation is in C++, it would be quite slow to do this in pure python.

Looping in @fmassa who is one of the authors of the paper in case he can provide a better recommendation.

[oke-aditya]

Although it's possible to write custom ops. Like we have NMS etc in cuda and C++ in torchvision we would need to package this too. But let's hear from @fmassa who knows this best.

[deepwilson]

Are we still waiting for @fmassa ? We could start building the other code blocks.