Community contribution: enabling `device_map="auto"` support for more vision and multimodal models

### Feature request

# Feature Request 

`transformers` models can be easily loaded across multiple devices using `device_map="auto"`. This will automatically allocate weights across available devices e.g. GPUs and offload any weights onto CPU, then disk as necessary. This is useful when doing inference with large models. 

To enable this, `_no_split_modules` has to be defined in the model's pretrained model class e.g. [like here for LLaMa](https://github.com/huggingface/transformers/blob/fadb053379b3ef24c4ec8e6d7d58555af21f58db/src/transformers/models/llama/modeling_llama.py#L793). This defines layers which should not be split across devices, and should contain as few layers as possible.

### Steps to add
* Pick a model to work on and open a PR - comment on this issue to say which model you're working on
* Define `_no_split_modules` in the PreTrainedModel subclass. Try with `_no_split_modules = []` first
* Enable testing
    * Ensure the following tests are not skipped for the model: `test_disk_offload_bin`, `test_disk_offload_safetensors`, `test_cpu_offload`, `test_model_parallelism`, `test_model_parallel_beam_search`
    * Run the tests in a multi-gpu environment `pytest tests/models/{MODEL_NAME}/test_modeling_{MODEL_NAME}.py -vv -k "offload or parallelism"`

## Models
- [ ] [Align](https://github.com/huggingface/transformers/blob/main/src/transformers/models/align/modeling_align.py)
- [ ] [Altclip](https://github.com/huggingface/transformers/blob/main/src/transformers/models/altclip/modeling_altclip.py)
- [x] [Beit](https://github.com/huggingface/transformers/blob/main/src/transformers/models/beit/modeling_beit.py) #30379
- [ ] [Bit](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bit/modeling_bit.py)
- [ ] [Blip](https://github.com/huggingface/transformers/blob/main/src/transformers/models/blip/modeling_blip.py)
- [ ] [Chinese_clip](https://github.com/huggingface/transformers/blob/main/src/transformers/models/chinese_clip/modeling_chinese_clip.py)
- [x] [Convnext](https://github.com/huggingface/transformers/blob/main/src/transformers/models/convnext/modeling_convnext.py) #30207
- [x] [Convnextv2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/convnextv2/modeling_convnextv2.py) #30207
- [x] [Cvt](https://github.com/huggingface/transformers/blob/main/src/transformers/models/cvt/modeling_cvt.py) #30207
- [ ] [Data2vec](https://github.com/huggingface/transformers/blob/main/src/transformers/models/data2vec/modeling_data2vec_vision.py)
- [ ] [Depth_anything](https://github.com/huggingface/transformers/blob/main/src/transformers/models/depth_anything/modeling_depth_anything.py)
- [ ] [Dinat](https://github.com/huggingface/transformers/blob/main/src/transformers/models/dinat/modeling_dinat.py)
- [ ] [Dinov2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/dinov2/modeling_dinov2.py)
- [ ] [Donut](https://github.com/huggingface/transformers/blob/main/src/transformers/models/donut/modeling_donut_swin.py)
- [ ] [Dpt](https://github.com/huggingface/transformers/blob/main/src/transformers/models/dpt/modeling_dpt.py)
- [ ] [Efficientformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/efficientformer/modeling_efficientformer.py)
- [x] [Efficientnet](https://github.com/huggingface/transformers/blob/main/src/transformers/models/efficientnet/modeling_efficientnet.py) #29989
- [ ] [Flava](https://github.com/huggingface/transformers/blob/main/src/transformers/models/flava/modeling_flava.py)
- [ ] [Focalnet](https://github.com/huggingface/transformers/blob/main/src/transformers/models/focalnet/modeling_focalnet.py) #30207
- [ ] [Git](https://github.com/huggingface/transformers/blob/main/src/transformers/models/git/modeling_git.py)
- [x] [Glpn](https://github.com/huggingface/transformers/blob/main/src/transformers/models/glpn/modeling_glpn.py) #30207
- [ ] [Groupvit](https://github.com/huggingface/transformers/blob/main/src/transformers/models/groupvit/modeling_groupvit.py)
- [x] [Imagegpt](https://github.com/huggingface/transformers/blob/main/src/transformers/models/imagegpt/modeling_imagegpt.py) #30207
- [ ] [Layoutlmv3](https://github.com/huggingface/transformers/blob/main/src/transformers/models/layoutlmv3/modeling_layoutlmv3.py)
- [x] [Levit](https://github.com/huggingface/transformers/blob/main/src/transformers/models/levit/modeling_levit.py) #30207
- [ ] [Mask2former](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mask2former/modeling_mask2former.py)
- [ ] [Maskformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/maskformer/modeling_maskformer.py)
- [ ] [Maskformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/maskformer/modeling_maskformer_swin.py)
- [x] [Mgp_str](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mgp_str/modeling_mgp_str.py) #30207
- [x] [Mobilenet_v1](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mobilenet_v1/modeling_mobilenet_v1.py) #30207
- [x] [Mobilenet_v2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mobilenet_v2/modeling_mobilenet_v2.py) #30207
- [x] [Mobilevit](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mobilevit/modeling_mobilevit.py) #30207
- [ ] [Mobilevitv2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mobilevitv2/modeling_mobilevitv2.py)
- [ ] [Nat](https://github.com/huggingface/transformers/blob/main/src/transformers/models/nat/modeling_nat.py)
- [ ] [Oneformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/oneformer/modeling_oneformer.py)
- [ ] [Perceiver](https://github.com/huggingface/transformers/blob/main/src/transformers/models/perceiver/modeling_perceiver.py)
- [x] [Poolformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/poolformer/modeling_poolformer.py) #30207
- [ ] [Pvt](https://github.com/huggingface/transformers/blob/main/src/transformers/models/pvt/modeling_pvt.py)
- [x] [Regnet](https://github.com/huggingface/transformers/blob/main/src/transformers/models/regnet/modeling_regnet.py) #30207
- [ ] [Resnet](https://github.com/huggingface/transformers/blob/main/src/transformers/models/resnet/modeling_resnet.py) #30207
- [x] [Sam](https://github.com/huggingface/transformers/blob/main/src/transformers/models/sam/modeling_sam.py) #30207
- [ ] [Segformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/segformer/modeling_segformer.py)
- [x] [Swiftformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/swiftformer/modeling_swiftformer.py) #30207
- [x] [Swin](https://github.com/huggingface/transformers/blob/main/src/transformers/models/swin/modeling_swin.py) #30207
- [ ] [Swin2sr](https://github.com/huggingface/transformers/blob/main/src/transformers/models/swin2sr/modeling_swin2sr.py)
- [x] [Swinv2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/swinv2/modeling_swinv2.py) #30207
- [ ] [Timesformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/timesformer/modeling_timesformer.py)
- [ ] [Timm_backbone](https://github.com/huggingface/transformers/blob/main/src/transformers/models/timm_backbone/modeling_timm_backbone.py)
- [x] [Trocr](https://github.com/huggingface/transformers/blob/main/src/transformers/models/trocr/modeling_trocr.py) #30207
- [ ] [Tvlt](https://github.com/huggingface/transformers/blob/main/src/transformers/models/tvlt/modeling_tvlt.py)
- [ ] [Tvp](https://github.com/huggingface/transformers/blob/main/src/transformers/models/tvp/modeling_tvp.py)
- [x] [Upernet](https://github.com/huggingface/transformers/blob/main/src/transformers/models/upernet/modeling_upernet.py) #30207
- [ ] [Videomae](https://github.com/huggingface/transformers/blob/main/src/transformers/models/videomae/modeling_videomae.py)
- [ ] [Vit_mae](https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit_mae/modeling_vit_mae.py)
- [ ] [Vit_msn](https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit_msn/modeling_vit_msn.py)
- [x] [Vitmatte](https://github.com/huggingface/transformers/blob/main/src/transformers/models/vitmatte/modeling_vitmatte.py) #30379
- [x] [Vivit](https://github.com/huggingface/transformers/blob/main/src/transformers/models/vivit/modeling_vivit.py) #30379
- [ ] [X_clip](https://github.com/huggingface/transformers/blob/main/src/transformers/models/x_clip/modeling_x_clip.py)
- [x] [Yolos](https://github.com/huggingface/transformers/blob/main/src/transformers/models/yolos/modeling_yolos.py) #30207

### Motivation

Enable a powerful HF feature for all of our vision models

### Your contribution

Ping me for review 🤗 

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Community contribution: enabling `device_map="auto"` support for more vision and multimodal models #29786

Feature request

Feature Request

Steps to add

Models

Motivation

Your contribution

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Community contribution: enabling device_map="auto" support for more vision and multimodal models #29786

Description

Feature request

Feature Request

Steps to add

Models

Motivation

Your contribution

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Community contribution: enabling `device_map="auto"` support for more vision and multimodal models #29786