Why aren't my proxy requests evenly distributed across each of the proxy nodes' pods?

[midonKing]

We have a Milvus cluster deployed using Kubernetes, with the version being 2.3.21. There are 3 proxy node pods inside it. Additionally, we have deployed a gateway named Caesar to enable the business side to connect to the backend address of Milvus through Caesar. Recently, we noticed that in the requests sent to Milvus, all search requests were only sent to 2 of the proxy pods, while the other one received absolutely no search requests, only a small number of delete, update, and show requests. What could be the reason for this? I understand that proxies should not distribute requests evenly across all pods, right?

[midonKing]

The configuration information of the network service is as follows,The IP information has been de-identified.

kind: Service
apiVersion: v1
metadata:
name: aiui-knowsecond-product-milvus
namespace: milvus-aiui-knowledge-product02
uid: d08df21e-3c85-4ac5-b033-f30a75c4ae93
resourceVersion: '824993312'
creationTimestamp: '2024-12-02T12:24:10Z'
labels:
app: aiui-knowsecond-product-milvus
app.kubernetes.io/instance: aiui-knowsecond-product
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: milvus
app.kubernetes.io/version: 2.3.16
component: proxy
helm.sh/chart: milvus-4.1.36
sae.cloud.iflytek.com/clusterID: '6'
sae.cloud.iflytek.com/creator: hhrong
sae.cloud.iflytek.com/projectID: '10'
sae.cloud.iflytek.com/repo: stc-sae-helm-repo
sae.cloud.iflytek.com/updater: wxhe5
annotations:
meta.helm.sh/release-name: aiui-knowsecond-product
meta.helm.sh/release-namespace: milvus-aiui-knowledge-product02
sae.cloud.iflytek.com/lastUpdatedTime: '2025-11-20T23:00:20+08:00'
sae.cloud.iflytek.com/updater: wxhe5
spec:
ports:
- name: milvus
protocol: TCP
port: 19530
targetPort: milvus
- name: metrics
protocol: TCP
port: 9091
targetPort: metrics
selector:
app.kubernetes.io/instance: aiui-knowsecond-product
app.kubernetes.io/name: milvus
component: proxy
clusterIP: x.x.x.x
clusterIPs:
- x.x.x.x
type: ClusterIP
sessionAffinity: None
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
internalTrafficPolicy: Cluster
status:
loadBalancer: {}

[yhmo]

Milvus can't distribute requests to different proxies. Each proxy is a grpc listener, receives the requests sent to its ip. I believe it is a network configuration issue, not a milvus issue.

[xiaofan-luan]

1. once client is connect to server, the request will only send to that proxy. when you have a lot proxies, the traffic will automatically distributed evenly.
2. my suggestion is to put an alb or nginx before proxy to trafiic can be evely distributed