GPU usage and performance with StandAlone Vs Distributed Milvus deployment

[gnreddy06]

During our performance evaluation of Standalone Vs Distributed Milvus deployment, we noticed two GPUs are equally utilized by the standalone deployment (about 84% of 2x NVIDIA L40s GPUs on the worker node) when running the vectordb-bench search perf test.
While in case of Distributed deployment, only one GPU is being utilized ( by query node only).

Might be related, we are seeing a better QPS with stand alone deployment compared to distributed deployment.

Setup :

Both the deployments uses 2x GPUs for one for query node and other for index node.
External Object storage is used instead of Minio.

Here is the custom values.yaml files used for each of the deployment.

Standalone deployment

helm install milvus-deployment milvus/milvus --set cluster.enabled=false --set etcd.replicaCount=1 --set minio.mode=standalone --set pulsar.enabled=false -f custom_values_standalone.yaml -n milvus

cat custom_values_standalone.yaml
image:
all:
repository: milvusdb/milvus
tag: v2.5.2-gpu
pullPolicy: IfNotPresent

indexNode:
resources:
requests:
nvidia.com/gpu: "1"
limits:
nvidia.com/gpu: "1"
queryNode:
resources:
requests:
nvidia.com/gpu: "1"
limits:
nvidia.com/gpu: "1"
minio:
enabled: false

externalS3:
enabled: true
host: “X.X.X.X
port: "80"

Distributed deployment

helm install milvus-deployment milvus/milvus -f custom_values_distributed.yaml -n milvus

cat custom_values_distributed.yaml
image:
all:
repository: milvusdb/milvus
tag: v2.5.2-gpu
pullPolicy: IfNotPresent

indexNode:
resources:
requests:
nvidia.com/gpu: "1"
limits:
nvidia.com/gpu: "1"
queryNode:
resources:
requests:
nvidia.com/gpu: "1"
limits:
nvidia.com/gpu: "1"
dataNode:
replicas: 2
minio:
enabled: false

externalS3:
enabled: true
host: “X.X.X.X
port: "80"

Questions:

Is this an issue ? or Should i need to change anything on the distributed deployment to make use of two GPUs ?

[yhmo]

I didn't get a chance to test milvus-gpu on k8s. But seems you didn't explicitly assign different GPUs to the indexNode and the queryNode.
https://milvus.io/docs/install_cluster-helm-gpu.md

indexNode:
  resources:
    requests:
      nvidia.com/gpu: "1"
    limits:
      nvidia.com/gpu: "1"
  extraEnv:
    - name: CUDA_VISIBLE_DEVICES
      value: "0"
queryNode:
  resources:
    requests:
      nvidia.com/gpu: "1"
    limits:
      nvidia.com/gpu: "1"
  extraEnv:
    - name: CUDA_VISIBLE_DEVICES
      value: "1"

[gnreddy06]

Thank you. It worked.