[Bug]: ExternalETCD - user name is empty error (authentication)

[aptmess]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.3.12-gpu
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): Kafka
- SDK version(e.g. pymilvus v2.0.0rc2): 2.3.10
- OS(Ubuntu or CentOS): Ubuntu
- CPU/Memory: 40 / 1024
- GPU: 3
- Others:

Current Behavior

Good day!

I am trying to deploy milvus-on-cluster with external components and the latest problem is ETCD. I got the error (the same as in this issue) while milvus is trying to connect:

{"level":"warn",
"ts":"2024-03-26T16:31:03.204Z",
"logger":"etcd-client",
"caller":"v3@v3.5.5/retry_interceptor.go:62",
"msg":"retrying of unary invoker failed",
"target":"etcd-endpoints://<host>:2379",
"attempt":0,
"error":"rpc error: code = InvalidArgument desc = etcdserver: user name is empty"}

I used all configurations in user.yaml - change authorizationEnabled true/false, rbac.enabled true/false:

    common:
      security:
        authorizationEnabled: true
    etcd:
      endpoints:
        - http://<host>:2379
      auth:
        rbac:
          enabled: true
        enable: true
        userName: user
        password: pass

How can I connect to external etcd and solve this problem? Thank you

Expected Behavior

Milvus can connect to External ETCD with authorization.

Steps To Reproduce

Deployment with milvus-on-cluster with `externalEtcd.enabled=true`

Milvus Log

{"level":"warn","ts":"2024-03-26T16:31:03.204Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://<host>:2379","attempt":0,"error":"rpc error: code = InvalidArgument desc = etcdserver: user name is empty"}

Anything else?

No response

[xiaofan-luan]

check the code and seems that the config is correct.

the etcd endpoints is also weird right?
etcd-endpoints://:2379 should it be localhost:2379 or something?

[xiaofan-luan]

/assign @PowderLi
could you check on this?

I think it might be here

[xiaofan-luan]

I will try to do a quick fix

[xiaofan-luan]

/assign @xiaofan-luan

[aptmess]

Thanks for quick response! I am observing pull request and waiting for merge and new release to test fixes

[yanliang567]

/unassign

[aptmess]

Good day! How is quick fix is going ? In which release it would be available (asking cause really need this fix to deploy milvus)? Thanks!

[xiaofan-luan]

this can be fixed on milvus 2.4

[aptmess]

And is there any possible datetime period of 2.4 release in the nearest future?

[xiaofan-luan]

@aptmess
Actaully I don't think 2.3 support using external username/password

[xiaofan-luan]

I checked the 2.3 code, there is no that support.
If you'd like maybe you can help on cherry pick this feature back

[aptmess]

To clarify - this fix would be able in 2.4?

[aptmess]

@xiaofan-luan is there any test docker image which I can try to use to understand that this issue has been solved?

[xiaofan-luan]

we will release 2.4.0 this week

[aptmess]

What about release ?(

[xiaofan-luan]

@aptmess
what is the current error log you get?

@yanliang567 can we reproduce this in house?

[aptmess]

@xiaofan-luan absolutely the same as in the previous one:

{"level":"warn","ts":"2024-03-26T16:31:03.204Z","logger":"etcd-client","caller":"v3@v3.5.5/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://<host>:2379","attempt":0,"error":"rpc error: code = InvalidArgument desc = etcdserver: user name is empty"}

[yanliang567]

lets track this issue in #32365