Vit-hybrid is deprecated, however still shown in the official documentation (with broken links)

[gmanlan]

System Info

Looks like a PR (#30781) was merged a few months ago, deprecating "low use models" like vit-hybrid. I'm not sure what the rationale is for deprecating models that fast (vit-hybrid is Y2022, isn't?) and just based on "downloads", any gain to be had? less maintenance maybe? has the whole industry already settled that vit-hybrid is completely useless?

In any case, the official documentation/website (https://huggingface.co/docs/transformers/en/model_doc/vit_hybrid) still mentions VIT-HYBRID as a valid/supported model, without any indication that it's been deprecated. What is more, links to Github to see the source code of the ViTHybridImageProcessor (https://huggingface.co/docs/transformers/en/model_doc/vit_hybrid#transformers.ViTHybridImageProcessor) or ViTHybridForImageClassification (https://huggingface.co/docs/transformers/en/model_doc/vit_hybrid#transformers.ViTHybridForImageClassification) lead to broken 404 pages.

@amyeroberts, @stevhliu

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. Check the official website/documentation - see VIT-HYBRID listed as officially supported, without any deprecation notice.
2. Check the VIT-HYBRID links to Github, are all broken, because the model was moved to "deprecated".
3. General question: why are we deprecating "low use" models just based on "downloads". Sounds like a weak (not very scientific) criteria. Have we done some usefulness/state-of-the-art/competitive benchmark/academic, etc. analysis to back this up? Is there any significant advantage/gain for removing these?

Expected behavior

At the minimum, consistency: either remove it from official documentation/website, or add a deprecation notice, and have working links to Github regardless.
Ideally: a better rationale, explanation or criteria for why a Y2022 model, which seems to be useful, has been deprecated just based on "downloads".

[amyeroberts]

Hi @gmanlan, thanks for opening an issue!

Regarding the docs, the model was deprecated in between the release of v4.41.0 and the new release today v4.42.0. The docs should show the latest release, v4.42.0 but seems to be still showing for the previous one. If you go onto the docs for the development branch, you will see a message highlighting that the model has been deprecated, and the links to source code point to the new, deprecated path. Thanks for highlighting that the docs are doing this! We'll look into trying to fix this as soon as possible as incorrect docs are both frustrating and confusing.

In terms of why models are deprecated, this is a maintenance decision. The transformers library is huge and currently has over 200 models. Every one of those models has an active maintenance cost in terms of things like compute for running tests, fixing bugs, updates to align with the rest of the library, reviewing new PRs etc. The reality is the transformers team is pretty small and in order for us to have time to develop new features, add new models, and maintain the library it's necessary to prioritise our time and resources. One way we reduce our maintenance burden is through periodic deprecations of lower use models.

We deprecate models which are at least a year old and have had a low number of checkpoint downloads from the hub over the past 30 days. This serves a proxy for lower community usage and hence reduced effort to impact ratio for on-going maintenance. This isn't a judgement on whether the model is useful, which is why we still keep it in the library: importable and runnable.

[gmanlan]

Thanks @amyeroberts for the clarification. Glad to be of help WRT documentation, and I now have a better understanding on the burden HF has WRT maintenance here. I'm glad to hear we are not claiming these models are useless.

One general thought is that since HF has become a central hub for most AI practitioners, it would be great to keep model (model architecture specifically) diversity in mind as we move forward. Most people will simply rank the models by downloads/popularity and run with that, which creates a less competitive environment (if 99% of people are using the same Swin Transformer or Llama model...) and gives new/lesser known options/models a hard time proving they could be better given the opportunity.

Thanks again! Keep up the good work.

[amyeroberts]

@gmanlan The docs should be up-to-date now!

Yes, having many different models available is both good and something we want. There's a wider question about how we can make lesser-known models more discoverable, but this is quite a difficult question and will probably require many different approaches.

The good news is that it's easier than ever for people to add their own models directly onto the hub e.g. like florence2. This removes the entry barrier of the PR process into the transformers library and relieves maintenance responsibilities for the transformers team meaning more models can be added quickly. Hopefully this helps in increasing diversity of models available for everyone in the community.

[gmanlan]

That's wonderful - very excited for the future of huggingface, thank you @amyeroberts!