Command to run Mosaic 7B on multi GPU RTX-4090 #760
Unanswered
karrtikiyer-tw
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have a RTX 4090 desktop with 2 GPU's, 24 GB each, would the below be the correct way to run the Mosaic 7B inference using VLLM?
CUDA_VISIBLE_DEVICES=0,1 python -m vllm.entrypoints.api_server --model mosaicml/mpt-7b --tensor-parallel-size 2 --trust-remote-code --port 4040 --host 0.0.0.0Beta Was this translation helpful? Give feedback.
All reactions