Running vLLM on Intel B60 Cards with Arrow Lake CPU

[azhuvath]

I have been trying to get vLLM running on Intel B60 Cards with Arrow Lake CPU. The procedure I followed are as follows. It is taken from this site. I have 2 B60 Cards in the machine. Using Ubuntu 24.04.

Linux ARLS-ASRK-02 6.14.0-33-generic #33~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 19 17:02:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

https://docs.vllm.ai/en/stable/getting_started/installation/gpu.html#build-wheel-from-source

git clone https://github.com/vllm-project/vllm.git
cd vllm

docker build -f docker/Dockerfile.xpu -t vllm-xpu-env --shm-size=4g .
docker run -it
--rm
--network=host
--device /dev/dri
-v /dev/dri/by-path:/dev/dri/by-path
vllm-xpu-env

python -m vllm.entrypoints.openai.api_server
--model=facebook/opt-13b
--dtype=bfloat16
--max_model_len=1024
--distributed-executor-backend=mp
--pipeline-parallel-size=2
-tp=8

Should I be using llm-scaler for 2 Intel B60 Cards with Arrow Lake CPU?

[liu-shaojun]

Hi,

We recommend directly using our latest image intel/llm-scaler-vllm:0.10.0-b3 instead of building from source.
Please follow these two sections in the documentation for detailed steps:

• Pulling and running the vLLM Docker container
• Launching the serving service

Also, the parameters in your command need a small adjustment — set

--pipeline-parallel-size=1
--tp=2

instead of --pipeline-parallel-size=2 and --tp=8.

Thanks!

[azhuvath]

Hi,

We recommend directly using our latest image intel/llm-scaler-vllm:0.10.0-b3 instead of building from source. Please follow these two sections in the documentation for detailed steps:

• Pulling and running the vLLM Docker container
• Launching the serving service

Also, the parameters in your command need a small adjustment — set

--pipeline-parallel-size=1
--tp=2

instead of --pipeline-parallel-size=2 and --tp=8.

Thanks!

If I want to infer a 8B model using BF16 using two B60 Cards with 24 GB of Memory per card, what is the best configuration so that the model gives best end to end latency, best throughput with least TTFT? I am not sure what is the significance of --pipeline-parallel-size=1 since that should be the default.

[liu-shaojun]

For your 8B model with BF16 on two B60 cards (24 GB each), we recommend adjusting --tp (tensor parallelism) rather than --pipeline-parallel-size.

• If the 8B BF16 model can fit on a single card, you can simply set --tp=1.
• Otherwise, set --tp=2 to split the model across the two cards.

You don’t need to explicitly set --pipeline-parallel-size; its default value is 1.

[azhuvath]

For your 8B model with BF16 on two B60 cards (24 GB each), we recommend adjusting --tp (tensor parallelism) rather than --pipeline-parallel-size.

• If the 8B BF16 model can fit on a single card, you can simply set --tp=1.
• Otherwise, set --tp=2 to split the model across the two cards.

You don’t need to explicitly set --pipeline-parallel-size; its default value is 1.

How to load balance two instances through vLLM one each running on single card. That means one model per one card but load balanced across two cards so that more concurrent request can be handled.

[liu-shaojun]

vLLM supports Data Parallel deployment, where model weights are replicated across multiple GPUs to handle independent batches of requests. You can enable it with, for example:

--data-parallel-size=2 --tensor-parallel-size=1

Alternatively, you can set up an external load balancer (e.g., Nginx or HAProxy) to distribute requests across multiple vLLM API servers running on different ports.

[azhuvath]

vLLM supports Data Parallel deployment, where model weights are replicated across multiple GPUs to handle independent batches of requests. You can enable it with, for example:

--data-parallel-size=2 --tensor-parallel-size=1
Alternatively, you can set up an external load balancer (e.g., Nginx or HAProxy) to distribute requests across multiple vLLM API servers running on different ports.

Thanks for the response.