Motivation.
Huggingface recently updated the argument from torch_dtype to a simple dtype to be coherent with PyTorch (see PR here).
Although their implementation is backward compatible, we could consider also making the change torch_dtype -> dtype to not trigger the warning:
`torch_dtype` is deprecated! Use `dtype` instead!
The warning is triggered for every call that involves huggingface (even if it is just for loading the model).
For example:
vllm bench latency --model "any_model_from_hf_hub" . . .
triggers the above warning.
Proposed Change.
Following changes are suggested:
- Replace
torch_dtype with dtype to silence above warning and be prepared to when torch_dtype is fully removed in hf.
- Ensure backwards compatibility from vLLM user perspective.
- (Optional) add a test for backwards compatibility.
Feedback Period.
No response
CC List.
No response
Any Other Things.
No response
Before submitting a new issue...
Motivation.
Huggingface recently updated the argument from
torch_dtypeto a simpledtypeto be coherent with PyTorch (see PR here).Although their implementation is backward compatible, we could consider also making the change
torch_dtype->dtypeto not trigger the warning:The warning is triggered for every call that involves huggingface (even if it is just for loading the model).
For example:
triggers the above warning.
Proposed Change.
Following changes are suggested:
torch_dtypewithdtypeto silence above warning and be prepared to whentorch_dtypeis fully removed in hf.Feedback Period.
No response
CC List.
No response
Any Other Things.
No response
Before submitting a new issue...