How can I run VLLM serving without an internet connection?

[ZPerling]

I cloned the model repository on Hugging Face to my local machine and used the --download-dir parameter to specify the directory. However, when running VLLM, it still tries to connect to Hugging Face, which doesn't work without an internet connection. Even after setting export HF_HUB_OFFLINE=1, offline mode doesn't seem to be working. Is it possible to run VLLM offline and if so, how can I achieve this?

[MarkoMFilip]

We're facing the same issue, any feedback here would be appreciated. Running vLLM online is currently not possible for us...

[Martyprod]

same issue, i'm offline 99% of the time and VLLM is not working offline :(.

[Moran232]

can you show the command you run vlllm?

[Martyprod]

I'm running it in Stability matrix, so can it run totally offline ? I can install it locally if the issue is related to stability matrix itself !
?

[HYZ17]

Using absolute path to load the model, rather than the name on huggingface, works for me. For example,

import os 
os.environ["CUDA_VISIBLE_DEVICES"] = "5"
from vllm import LLM

prompts = ["Hello, my name is", "The capital of France is"]  # Sample prompts.
#llm = LLM(model="/home/.cache/huggingface/hub/models--huggyllama--llama-13b/snapshots/bf57045473f207bb1de1ed035ace226f4d9f9bba",)
outputs = llm.generate(prompts)  # Generate texts from the prompts.
print(outputs)

[llmwesee]

It works but the generated responses are wrong most of the time.

[smahdavi4]

Setting the hugging face transformers module to offline mode (via a global environment variable) worked for me: TRANSFORMERS_OFFLINE=1

[juud79]

same...