[Bug]: Upsert failed: quota exceeded[reason=rate type: DMLUpsert] although the memory and tt delay are normal

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:master-20240516-875ad88d
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):  pulsar  
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

1. deploy milvus with config:

    dataNode:
      paused: false
      replicas: 1
      resources:
        limits:
          cpu: "4" 
          memory: 16Gi
        requests:
          cpu: "2" 
          memory: 8Gi 
    queryNode:
      paused: false
      replicas: 1
      resources:
        limits:
          cpu: "4" 
          memory: 4Gi 
        requests:
          cpu: "2" 
          memory: 2Gi
  config:
    log:
      level: debug
    quotaAndLimits:
      flushRate:
        enabled: true
        max: 0.1
    trace:
      exporter: jaeger
      jaeger:
        url: http://tempo-distributor.tempo:14268/api/traces
      sampleFraction: 1

2. test: create a collection with schema

{'auto_id': False, 'description': '', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}, {'name': 'int64_1', 'description': '', 'type': <DataType.INT64: 5>, 'is_partition_key': True}], 'enable_dynamic_field': False}

3. index -> concurrent: upsert + flush. and upsert failed with error:

[2024-05-16 09:56:40,626 - ERROR - fouram]: RPC error: [upsert], <MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])>, <Time:{'RPC start': '2024-05-16 09:56:38.832061', 'RPC error': '2024-05-16 09:56:40.626541'}> (decorators.py:139)
[2024-05-16 09:56:40,627 - ERROR - fouram]: (api_response) : [Collection.upsert] <MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])>, [requestId: 96bc5296-136a-11ef-8c2c-f25810ef22f9] (api_request.py:57)
[2024-05-16 09:56:40,627 - ERROR - fouram]: [CheckFunc] upsert request check failed, response:<MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])> (func_check.py:48)
[2024-05-16 09:56:40,657 - ERROR - fouram]: RPC error: [upsert], <MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])>, <Time:{'RPC start': '2024-05-16 09:56:38.861677', 'RPC error': '2024-05-16 09:56:40.657705'}> (decorators.py:139)
[2024-05-16 09:56:40,658 - ERROR - fouram]: (api_response) : [Collection.upsert] <MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])>, [requestId: 96c07592-136a-11ef-8c2c-f25810ef22f9] (api_request.py:57)
[2024-05-16 09:56:40,658 - ERROR - fouram]: [CheckFunc] upsert request check failed, response:<MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])> (func_check.py:48)
[2024-05-16 09:56:40,684 - ERROR - fouram]: RPC error: [upsert], <MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])>, <Time:{'RPC start': '2024-05-16 09:56:38.889969', 'RPC error': '2024-05-16 09:56:40.684850'}> (decorators.py:139)
[2024-05-16 09:56:40,685 - ERROR - fouram]: (api_response) : [Collection.upsert] <MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])>, [requestId: 96c51c00-136a-11ef-8c2c-f25810ef22f9] (api_request.py:57)
[2024-05-16 09:56:40,685 - ERROR - fouram]: [CheckFunc] upsert request check failed, response:<MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])> (func_check.py:48)
[2024-05-16 09:56:40,714 - ERROR - fouram]: RPC error: [upsert], <MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])>, <Time:{'RPC start': '2024-05-16 09:56:38.922541', 'RPC error': '2024-05-16 09:56:40.714667'}> (decorators.py:139)
[2024-05-16 09:56:40,715 - ERROR - fouram]: (api_response) : [Collection.upsert] <MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])>, [requestId: 96c97e08-136a-11ef-8c2c-f25810ef22f9] (api_request.py:57)
[2024-05-16 09:56:40,715 - ERROR - fouram]: [CheckFunc] upsert request check failed, response:<MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])> (func_check.py:48)
[2024-05-16 09:56:40,745 - ERROR - fouram]: RPC error: [upsert], <MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])>, <Time:{'RPC start': '2024-05-16 09:56:38.950864', 'RPC error': '2024-05-16 09:56:40.745349'}> (decorators.py:139)
[2024-05-16 09:56:40,746 - ERROR - fouram]: (api_response) : [Collection.upsert] <MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])>, [requestId: 96ce4f1e-136a-11ef-8c2c-f25810ef22f9] (api_request.py:57)
[2024-05-16 09:56:40,746 - ERROR - fouram]: [CheckFunc] upsert request check failed, response:<MilvusException: (code=9, message=quota exceeded[reason=rate type: DMLUpsert])> (func_check.py:48)

4. metrics
   
   

Expected Behavior

No response

Steps To Reproduce

- argo: https://argo-workflows.zilliz.cc/archived-workflows/qa/96c8de33-3c44-4cc6-a0df-06dfbfccf9b8?nodeId=compact-opt-partitionkey-4b
- grafana: [metrics of compact-opt-op-47-9841](https://grafana-4am.zilliz.cc/d/uLf5cJ3Ga/milvus2-0?orgId=1&var-datasource=P1809F7CD0C75ACF3&var-cluster=&var-namespace=qa-milvus&var-instance=compact-opt-op-47-9841&var-collection=All&var-app_name=milvus&from=1715852428000&to=1715854322000)
- pod:

compact-opt-op-47-9841-milvus-datanode-79b984cd49-6ck9c           1/1     Running       1 (3h29m ago)   3h37m   10.104.13.86    4am-node16   <none>           <none>
compact-opt-op-47-9841-milvus-indexnode-865c4768b8-7ngrn          1/1     Running       0               3h37m   10.104.21.253   4am-node24   <none>           <none>
compact-opt-op-47-9841-milvus-indexnode-865c4768b8-9wb64          1/1     Running       0               3h37m   10.104.5.3      4am-node12   <none>           <none>
compact-opt-op-47-9841-milvus-indexnode-865c4768b8-nhlqr          1/1     Running       0               3h37m   10.104.6.143    4am-node13   <none>           <none>
compact-opt-op-47-9841-milvus-mixcoord-5ffd8bc6c7-qxrj9           1/1     Running       0               3h29m   10.104.4.182    4am-node11   <none>           <none>
compact-opt-op-47-9841-milvus-proxy-5858cb9b8b-vrq9w              1/1     Running       1 (3h29m ago)   3h37m   10.104.13.87    4am-node16   <none>           <none>
compact-opt-op-47-9841-milvus-querynode-0-67554f64-6nkmb          1/1     Running       5 (3h7m ago)    3h37m   10.104.30.19    4am-node38   <none>           <none>

### Milvus Log

_No response_

### Anything else?

_No response_

[ThreadDao]

• https://argo-workflows.zilliz.cc/archived-workflows/qa/17005b73-39aa-44be-8993-3ce24c3e72eb?nodeId=compact-opt-partitionkey-4d

[yanliang567]

/assign @aoiasd
I don't think we have enlable limit and quota in this case. but the quota exceeded error should not report
/unassign

[Reidddddd]

using v2.4.1, we met the same issue

we found that, in the yaml of rootcoord in live env (as pic shows): quotaAndLimits.enabled: true

But in fact, we didn't configure this parameter, which by default should be false.

So we explicitly set

quotaAndLimits:
    enabled: false

and issue was gone

The issue leads to 2 tasks, IMO:

1. quotaAndLimits.enabled not in any docs, I found it in code. So related documents needs to be updated
2. Find out why quotaAndLimits.enabled becomes true without configured..

[Reidddddd]

BTW, I don't think the attach PR is an actual fix, it doesn't solve the task 2 I mentioned

[Reidddddd]

using v2.4.1, we met the same issue

we found that, in the yaml of rootcoord in live env (as pic shows): quotaAndLimits.enabled: true But in fact, we didn't configure this parameter, which by default should be false.

So we explicitly set

quotaAndLimits:
    enabled: false

and issue was gone

The issue leads to 2 tasks, IMO:

1. quotaAndLimits.enabled not in any docs, I found it in code. So related documents needs to be updated
2. Find out why quotaAndLimits.enabled becomes true without configured..

related code line:

milvus/internal/proxy/simple_rate_limiter.go

Line 55 in 15b7874

if !Params.QuotaConfig.QuotaAndLimitsEnabled.GetAsBool() {

[ThreadDao]

waiting for verifying...