Permission deny when the user has the necessary role and privilege

[jubingc]

The logs of the proxy show the following error

{
  "level": "INFO",
  "time": "2025/02/19 09:11:55.775 +00:00",
  "caller": "proxy/privilege_interceptor.go:211",
  "message": "permission deny",
  "username": "jubing_owner",
  "role_names": [
    "public"
  ],
  "object_type": "Global",
  "object_privilege": "PrivilegeSelectOwnership",
  "db_name": "jubing",
  "object_index": -1,
  "object_name": "*",
  "object_indexs": 0,
  "object_names": [],
  "roles": [
    "public"
  ]
}

However, the user jubing_owner has the following role returned from MilvusClient.describe_user method

{
  "user_name": "jubing_owner",
  "roles": [
    "jubing_owner"
  ]
}

The role jubing_owner has the following privilege returned from MilvusClient.describe_role method

    {
      "object_type": "Global",
      "object_name": "*",
      "db_name": "jubing",
      "role_name": "jubing_owner",
      "privilege": "SelectOwnership",
      "grantor_name": "root"
    },

I assume the user and role already have the required privilege SelectOwnership to pass the authentication. Why does the "permission deny" log show that user jubing_owner has only the public role and does not have enough privilege as a result? How should I set the privileges correctly?

[yhmo]

I have no idea, let me ask somebody to help.

[SimFG]

Because milvus is distributed, during the entire authorization process, the rbac information will be persisted by rootcoord and then returned immediately, but this information is synchronized to other proxies. You can try it now, it should work properly

[jubingc]

Thanks for confirmation it is ok to see "GetPrivilegeExtObj failed" during proxy startup.

Any suggestion on how to solve the privilege missing issue when the proxy pod restarts? We have just one cluster having this issue. I don't see "GetPrivilegeExtObj failed" error in logs of other healthy cluster proxy pods.

[SimFG]

In k8s, there is a function to check whether the pod status is normal. Generally, I will configure an address for it to check whether the milvus service is normal.
Reference: https://github.com/zilliztech/milvus-helm/blob/d118ba680c352eb547a107e80076bc3055c953eb/charts/milvus/templates/proxy-deployment.yaml#L106
The corresponding milvus address is: xxx:9091/healthz

[jubingc]

curl http://localhost:9091/healthz printed OK

Proxy pod is always healthy and running even when "GetPrivilegeExtObj failed" shows up in logs.

[SimFG]

Is the permission still not working? What version are you using? Can you upload the log?

[jubingc]

Still not working. As mentioned above, "permission deny" lasts forever after the proxy pod restarts unless privileges are set again through APIs.

Summarize the issue again:

1. This issue only happens to this milvus cluster while I have more than 10 others.
2. This cluster has a role mvdb_plygd_francis_dev_role_devs cannot be deleted.
3. GetPrivilegeExtObj failed in logs of proxy pods
4. Privileges can be listed successfully with MilvusClient.describe_role using pymilvus
5. Privileges do not work after the proxy pod restarts unless privileges are reset by API

Proxy log is attached.
playground-dev-proxy.log

[SimFG]

I checked the logs and found that there seems to be no authorization information in the cluster.

[jubingc]

Can I have your suggestions for the next step? The privileges can be listed successfully from MilvusClient.describe_role. I need to solve this issue.

[SimFG]

From the current log, I think milvus is working properly. You can take the following actions:

1. Confirm whether the configuration of this proxy is correct, especially the configuration related to etcd. I suspect that it may be connected to the etcd of other milvus services
2. Restart the proxy to see if it recovers

[jubingc]

The configuration at the path of config/default.yaml in the proxy pod looks good to me.

etcd:
  endpoints:
  - milvus-playground-dev-etcd-0.milvus-playground-dev-etcd-headless.ns-db-milvus-playground-dev.svc.cluster.local:2379
  - milvus-playground-dev-etcd-1.milvus-playground-dev-etcd-headless.ns-db-milvus-playground-dev.svc.cluster.local:2379
  - milvus-playground-dev-etcd-2.milvus-playground-dev-etcd-headless.ns-db-milvus-playground-dev.svc.cluster.local:2379

etcd cluster is healyh

etcdctl endpoint health --cluster
http://milvus-playground-dev-etcd-0.milvus-playground-dev-etcd-headless.ns-db-milvus-playground-dev.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 2.423353ms
http://milvus-playground-dev-etcd-1.milvus-playground-dev-etcd-headless.ns-db-milvus-playground-dev.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 7.209328ms
http://milvus-playground-dev-etcd-2.milvus-playground-dev-etcd-headless.ns-db-milvus-playground-dev.svc.cluster.local:2379 is healthy: successfully committed proposal: took = 7.363482ms

etcd pod is reachable from proxy pod

nc -zv milvus-playground-dev-etcd-0.milvus-playground-dev-etcd-headless.ns-db-milvus-playground-dev.svc.cluster.local 2379
Connection to milvus-playground-dev-etcd-0.milvus-playground-dev-etcd-headless.ns-db-milvus-playground-dev.svc.cluster.local (100.65.92.116) 2379 port [tcp/*] succeeded!

There is only one etcd cluster deployed in this namespace. We also have network policies which deny traffic between other milvus namespaces.

If this proxy connects to other milvus's etcd, the privileges should be persistent once they are set again. However, privileges are always gone whenever the proxy pod restarts.

[SimFG]

I'm not sure what the problem is. But from the proxy log, milvus does not have any abnormalities. Can you upload the configuration files of proxy and rootcoord?

[jubingc]

Thanks for helping again. Logs and pod descriptions are attached. milvus-log-playground-dev.tar.gz

The python code used for resetting privileges:

db_name = "default"
role_name = "owner"

client = MilvusClient(
    uri="http://localhost",
    port=19530,
    user="root",
    password="",
    db_name=db_name,
)

client.grant_role(user_name=role_name, role_name=role_name)

if role_name.endswith("owner"):
    for _, object_privileges in owner_privileges.items():
        for p in object_privileges:
            client.grant_privilege_v2(
                role_name=role_name,
                privilege=p,
                collection_name="*",
                db_name=db_name,
            )

owner_privileges = {
    "Collection": [
        "CreateIndex",
        "DropIndex",
        "IndexDetail",
        "Load",
        "GetLoadingProgress",
        "GetLoadState",
        "Release",
        "Insert",
        "Delete",
        "Upsert",
        "Search",
        "Flush",
        "GetFlushState",
        "Query",
        "GetStatistics",
        "Compaction",
        "Import",
        "LoadBalance",
        "CreatePartition",
        "DropPartition",
        "ShowPartitions",
        "HasPartition",
    ],
    "Global": [
        "CreateCollection",
        "DropCollection",
        "DescribeCollection",
        "ShowCollections",
        "RenameCollection",
        "FlushAll",
        "SelectOwnership",
        "CreateResourceGroup",
        "DropResourceGroup",
        "DescribeResourceGroup",
        "ListResourceGroups",
        "TransferNode",
        "TransferReplica",
        "CreateDatabase",
        "DropDatabase",
        "ListDatabases",
        "DescribeDatabase",
        "AlterDatabase",
        "CreateAlias",
        "DropAlias",
        "DescribeAlias",
        "ListAliases",
    ],
    "User": [],
}

[SimFG]

The zip file is empty. And what is this owner_privileges value??

[jubingc]

If I used jubing_owner user log in on attu, then I got the following error in proxy logs:

{"level":"INFO","time":"2025/02/28 01:44:58.024 +00:00","caller":"proxy/privilege_interceptor.go:211","message":"permission deny","username":"jubing_owner","role_names":["public"],"object_type":"Global","object_privilege":"PrivilegeDescribeDatabase","db_name":"jubing","object_index":-1,"object_name":"*","object_indexs":0,"object_names":[],"roles":["public"]}

The output of MilvusClient.describe_user is

{"user_name": "jubing_owner", "roles": ["jubing_owner"]}

[jubingc]

Used milvus_cli to delete the role mvdb_plygd_francis_dev_role_devs and got the same error

milvus_cli > list grants -r mvdb_plygd_francis_dev_role_devs
+--------+-------------+---------+-----------+--------------+-----------+
| Object | Object Name | DB Name | Role Name | Grantor Name | Privilege |
+--------+-------------+---------+-----------+--------------+-----------+
+--------+-------------+---------+-----------+--------------+-----------+

As you can see, there are no privileges associated with the role. But it is still unable to delete that role.

milvus_cli >  delete role -r mvdb_plygd_francis_dev_role_devs
2025-03-03 18:19:09,385 [ERROR][handler]: RPC error: [drop_role], <MilvusException: (code=65535, message=fail to drop the role that it has privileges. Use REVOKE API to revoke privileges)>, <Time:{'RPC start': '2025-03-03 18:19:09.366785', 'RPC error': '2025-03-03 18:19:09.385795'}> (decorators.py:140)
Drop role error!<MilvusException: (code=65535, message=fail to drop the role that it has privileges. Use REVOKE API to revoke privileges)>

Is there another way to delete this role?

[xiaofan-luan]

and from the document,

client.describe_role(role_name="role_a")

is used for check a roles privlidge, can you check using describe_role("mvdb_plygd_francis_dev_role_devs") and check actually privilege they actaully have?

[jubingc]

Thanks for helping.

print(json.dumps(client.describe_role("mvdb_plygd_francis_dev_role_devs")))

The output is exactly the same as milvus_cli's.

{"role": "mvdb_plygd_francis_dev_role_devs", "privileges": []}

When dropping

client.drop_role("mvdb_plygd_francis_dev_role_devs")

pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=fail to drop the role that it has privileges. Use REVOKE API to revoke privileges)>

Milvus_cli uses pymilvus, right? So the same result is expected.

[jubingc]

One possible reason is that this role's privileges are tied to another database that does not exist anymore. As far as I know, when connecting to milvus using different db_name, the privileges from the describe_role will change. That's why I always got an empty list of privileges because I connect to milvus using default db_name.

And unfortunately, I don't know the name of the non-existing database. I am thinking if I can re-crease the database and list those privileges?

[xiaofan-luan]

@jubingc

from the log
jubing is under rolename jubing_owner

but here what we see is public

{"level":"INFO","time":"2025/02/28 01:44:58.024 +00:00","caller":"proxy/privilege_interceptor.go:211","message":"permission deny","username":"jubing_owner","role_names":["public"],"object_type":"Global","object_privilege":"PrivilegeDescribeDatabase","db_name":"jubing","object_index":-1,"object_name":"*","object_indexs":0,"object_names":[],"roles":["public"]}

[xiaofan-luan]

{"user_name": "jubing_owner", "roles": ["jubing_owner"]}

and here you get a third role.

Can you explain how you run into that states?

what is the relationship between jubing_owner and public

[jubingc]

That's where I am confused. From milvus_cli output:

milvus_cli > list roles
+----------------------------------+--------------------------+
|             roleName             |          users           |
+----------------------------------+--------------------------+
|              admin               |            ()            |
|         aidot_developer          |   ('aidot_developer',)   |
|           aidot_owner            |     ('aidot_owner',)     |
|           aidot_viewer           |    ('aidot_viewer',)     |
|            developer             |      ('developer',)      |
|       insertbot_developer        | ('insertbot_developer',) |
|         insertbot_owner          |   ('insertbot_owner',)   |
|         insertbot_viewer         |  ('insertbot_viewer',)   |
|         jubing_developer         |  ('jubing_developer',)   |
|           jubing_owner           |    ('jubing_owner',)     |
|          jubing_viewer           |    ('jubing_viewer',)    |
| mvdb_plygd_francis_dev_role_devs |            ()            |
|              owner               |        ('owner',)        |
|              public              |            ()            |
|        sdet_bat_developer        | ('sdet_bat_developer',)  |
|          sdet_bat_owner          |   ('sdet_bat_owner',)    |
|         sdet_bat_viewer          |   ('sdet_bat_viewer',)   |
|              viewer              |       ('viewer',)        |
+----------------------------------+--------------------------+

"public" role should be an extra role to any granted roles code reference.

There is no relationship between the "public" role and other roles as far as I can tell.

I am also wondering why this happens to this cluster. The issue happens every time the proxy pods restart.

[jubingc]

Here is another example reproducing the privilege issue. Even the user has role, the role has the privilege, the permission deny still occur.

client = MilvusClient(
    uri='https://milvus-playground-dev',
    port=443,
    user='root',
    password="****",
    db_name='jubing',
)
print(client.describe_user("jubing_owner"))
for p in client.describe_role("jubing_owner")['privileges']:
    if p['privilege'] == 'DescribeDatabase' and p['role_name'] == 'jubing_owner':
        print(f"Found the privilege: {p}")
        break

client = MilvusClient(
    uri='https://milvus-playground-dev',
    port=443,
    user='jubing_owner',
    password="****",
    db_name='jubing',
)
try:
    client.describe_database(db_name="jubing")
except Exception as e:
    print(str(e))

Python output

{'user_name': 'jubing_owner', 'roles': ('jubing_owner',)}
Found the privilege: {'object_type': 'Global', 'object_name': '*', 'db_name': 'jubing', 'role_name': 'jubing_owner', 'privilege': 'DescribeDatabase', 'grantor_name': 'root'}
2025-03-03 16:53:44,644 [ERROR][handler]: grpc RpcError: [describe_database], <_InactiveRpcError: StatusCode.PERMISSION_DENIED, PrivilegeDescribeDatabase: permission deny to jubing_owner in the `jubing` database>, <Time:{'RPC start': '2025-03-03 16:53:44.560920', 'gRPC error': '2025-03-03 16:53:44.644713'}> (decorators.py:151)
<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.PERMISSION_DENIED
	details = "PrivilegeDescribeDatabase: permission deny to jubing_owner in the `jubing` database"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"PrivilegeDescribeDatabase: permission deny to jubing_owner in the `jubing` database", grpc_status:7, created_time:"2025-03-03T16:53:44.641399-08:00"}"
>

Proxy logs

{"level":"INFO","time":"2025/03/04 00:53:44.093 +00:00","caller":"proxy/impl.go:6273","message":"connect received","traceID":"3f3ed8bcd44ebe377bee1009ab7b045b","sdk_type":
"Python","sdk_version":"2.5.4","local_time":"2025-03-03 16:53:44.054802","user":"root","host":"jubingc-mlt","db":"jubing"}
{"level":"INFO","time":"2025/03/04 00:53:44.095 +00:00","caller":"connection/manager.go:106","message":"client register","traceID":"3f3ed8bcd44ebe377bee1009ab7b045b","sdk_
type":"Python","sdk_version":"2.5.4","local_time":"2025-03-03 16:53:44.054802","user":"root","host":"jubingc-mlt","identifier":456405712648273923,"last_active_time":"2025/
03/04 00:53:44.095 +00:00"}
{"level":"INFO","time":"2025/03/04 00:53:44.554 +00:00","caller":"proxy/impl.go:6273","message":"connect received","traceID":"61a9d07096f2427432be47789d257711","sdk_type":
"Python","sdk_version":"2.5.4","local_time":"2025-03-03 16:53:44.480028","user":"jubing_owner","host":"jubingc-mlt","db":"jubing"}
{"level":"INFO","time":"2025/03/04 00:53:44.555 +00:00","caller":"connection/manager.go:106","message":"client register","traceID":"61a9d07096f2427432be47789d257711","sdk_
type":"Python","sdk_version":"2.5.4","local_time":"2025-03-03 16:53:44.480028","user":"jubing_owner","host":"jubingc-mlt","identifier":456405712766238723,"last_active_time
":"2025/03/04 00:53:44.555 +00:00"}
{"level":"INFO","time":"2025/03/04 00:53:44.636 +00:00","caller":"proxy/privilege_interceptor.go:211","message":"permission deny","username":"jubing_owner","role_names":["
ublic"[],"object_type":"Global","object_privilege":"PrivilegeDescribeDatabase","db_name":"jubing","object_index":-1,"object_name":"*","object_indexs":0,"object_names":[],"
roles":["public"]}

[shaoting-huang]

hello @jubingc, I use the same sdk and Milvus version and the below script works.

from pymilvus import connections, db
from pymilvus import CollectionSchema, FieldSchema
from pymilvus import Collection
from pymilvus import connections
from pymilvus import DataType
from pymilvus import Partition
from pymilvus import utility
from pymilvus import MilvusClient
from pymilvus import Role

connections.connect(host="localhost", user="root", password="Milvus")
client = MilvusClient(uri="http://localhost:19530", user="root", password="Milvus")
client.create_collection("col1", dimension=128)
db.create_database("jubing")
role = Role("jubing_owner")
role.create()
utility.list_roles(True)
utility.create_user(user="jubing_owner", password="Milvus")
role.add_user("jubing_owner")
utility.list_roles(True)
role.list_grants()

print(client.describe_user("jubing_owner"))
for p in client.describe_role("jubing_owner")['privileges']:
	if p['privilege'] == 'DescribeDatabase' and p['role_name'] == 'jubing_owner':
		print("Found the privilege")
		break

client = MilvusClient(uri="http://localhost:19530", user="jubing_owner", password="Milvus", db_name="jubing")

# permission denied
client.describe_database(db_name="jubing")

client = MilvusClient(uri="http://localhost:19530", user="root", password="Milvus")

role = Role("jubing_owner")
role.grant_v2("DescribeDatabase", "*", db_name="jubing")

role.list_grants()

client = MilvusClient(uri="http://localhost:19530", user="jubing_owner", password="Milvus", db_name="jubing")

# it works
client.describe_database(db_name="jubing")

Also I noticed that from the python output you paste, it shows this line
Found the privilege: {'object_type': 'Global', 'object_name': '*', 'db_name': 'jubing', 'role_name': 'jubing_owner', 'privilege': 'DescribeDatabase', 'grantor_name': 'root'}
but the script only print print("Found the privilege")

[jubingc]

@shaoting-huang Thanks for helping. Your script works. But when the proxy pod restarts, I still get the same permission deny exception.

  File "/Users/jubingc/.pyenv/versions/milvus-test/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.PERMISSION_DENIED
	details = "PrivilegeDescribeDatabase: permission deny to jubing_owner in the `jubing` database"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"PrivilegeDescribeDatabase: permission deny to jubing_owner in the `jubing` database", grpc_status:7, created_time:"2025-03-05T12:57:57.124431-08:00"}"

I think the issue is that when the proxy pod restarts, all privileges will have no effect even though those privileges exist. Can we have a call? I can walk you through

Right, I append the privilege object to the line "Found the privilege" to show the content of the privilege. I updated my original code. Thanks for pointing that out.

[SimFG]

@jubingc Can you start a new milvus and try to reproduce this problem, and then back up etcd to us?

[jubingc]

@SimFG This issue only happens to one of my clusters. I have around 10 milvus clusters running on 3 AWS EKS clusters. I am not sure how to reproduce it. I am not able to reproduce this issue by provisioning a new cluster on the same EKS cluster.

It seems to happen after the upgrade from 2.4.1x to 2.5.4. But I also did the same milvus upgrade to the rest of the milvus clusters, none of them has this issue.

I can back up etcd to you in a secure way. Please let me know how. Thanks!

[jubingc]

@SimFG Another milvus cluster managed by another team has this issue after upgrading from 2.4.1 to 2.4.20, e.g. privileges have no effects after proxy pods restart.

[xiaofan-luan]

could this happen becasue of role and priviledge is created on 2.4.1 and use rbac v1 instead of V2?

[jubingc]

@SimFG I have a snapshot of etcd. Please let me know how I can share the file with you. The snapshot size is 10 MB.

@xiaofan-luan I tried to grant privileges through the privilege group via MilvusClient.grant_privilege_v2. The issue still exists. Privileges have no effect after the proxy pod restart even though those privilege groups exist and are granted to roles.

[SimFG]

@jubingc
I think I found the cause of the problem. The solution is: upgrade to version 2.5.5
The commit that fixed this problem is: bc91b2a

[SimFG]

@thangTang it's a bug, and we will fix it in this week. issue: #40547

[thangTang]

@thangTang it's a bug, and we will fix it in this week. issue: #40547

thanks for this info, it will be released in v2.5.7?

[SimFG]

@thangTang yes

[jubingc]

@SimFG thanks for the confirmation. BTW, will there be a fix for 2.4.x branch as well?

[SimFG]

@jubingc Yes, but I am not sure whether there will be a new version in the future 2.4