Encoder model support for the Transformers backend

[hmellor]

Adds support for encoder models to the Transformers backend.

Depends on

I have conditioned this feature on the Transformers version, so as soon as the following dependency is merged we can merge this PR and users installing vLLM and Transformers from main can use this feature:

• 🔴[Attention] Bert-based Models Attention Refactor huggingface/transformers#38301 - so that encoder models use the ALL_ATTENTION_FUNCTIONS interface necessary for the Transformers backend

Changes

• Use EncoderOnlyAttention if an encoder model is detected
• Skip position_ids buffers if they're found in the checkpoint because vLLM always passes position_ids anyway

Testing

python examples/offline_inference/basic/embed.py --model BAAI/bge-base-en-v1.5 --model-impl transformers

pytest tests/models/test_transformers.py -k test_embed_correctness

[gemini-code-assist]

Code Review

This pull request adds support for encoder models to the Transformers backend. The main changes include detecting encoder models by checking for is_causal=False and using EncoderOnlyAttention accordingly. It also adds logic to skip loading position_ids from checkpoints for encoder models, as vLLM handles this. The changes look good, but I have a critical suggestion to improve the type hint for create_attention_instances to reflect that it can return EncoderOnlyAttention instances, which will improve code clarity and maintainability.

[Isotr0py]

Just a nit. Otherwise LGTM!