Add a reverse mapping from role to namespaces

[PierreZ]

Hi!

Is your feature request related to a problem? Please describe.
I'm working on KoP, and I'm currently implementing security on it. I'm currently struggling to list all topics according to a Pulsar role.

Part of the Kafka handshake is called the Metadata request:

[...] Kafka brokers can answer a metadata request that describes the current state of the cluster: what topics there are, which partitions those topics have, which broker is the leader for those partitions, and the host and port information for these brokers.

Most of the client implementation are expecting to retrieve all available topics by sending a MetadataRequest with an empty list of topics. In the context of authorization, I can't list all topics related to a Pulsar Role because I can't reverse from a role the associated tenants and namespaces.

Describe the solution you'd like
I would like to add a new internal function that, given a role, is answering the full list of associated namespaces.

I think this type of code should be exposed within the admin API but maybe there is another(and better!) location like the NamespaceService?

[jiazhai]

Thanks @PierreZ . This would also be useful for admin and other protocol implementations.

[PierreZ]

Thanks @jiazhai, where do you think I should put the implementation? I am hesitating between:

• PulsarAuthorizationProvider
• NamespacesBase
• NamespaceService

[merlimat]

I think that most AuthZ implementation won't be able to fulfill that info though

[PierreZ]

I digged through the code, roles are stored within ZK as a serialized version of the policies class. Which means that it is PulsarAuthorizationProvider which is responsible for this. I'm wondering if we should add a new method to the interface to be sure that all AuthorizationProvider can reverse a role to their associated namespaces. What do you think?

[merlimat]

roles are stored within ZK as a serialized version of the policies class.

That's true for the default implementation, though the AuthZ backend can be (and is in real-life) also pluggable with different implementations.

'm wondering if we should add a new method to the interface to be sure that all AuthorizationProvider can reverse a role to their associated namespaces. What do you think?

That would be requiring to do a scan of the metadata to search for namespaces in which a role has access to. Or maintaining a different mapping, with the trouble of keeping that in sink with the source of truth.

[PierreZ]

That would be requiring to do a scan of the metadata to search for namespaces in which a role has access to. Or maintaining a different mapping, with the trouble of keeping that in sink with the source of truth.

True, I was thinking about implementing your first proposal. Something that is looping through znodes, and check if auth_policies.namespace_auth.containsKey(role) for the default implementation.

Or maintaining a different mapping, with the trouble of keeping that in sink with the source of truth.

I agree with you, not sure it's worth it for now and for a first implementation.

That's true for the default implementation, though the AuthZ backend can be (and is in real-life) also pluggable with different implementations.

That means that I don't have many options regarding the location of where that piece of code should be: it must be hold by the AuthorizationProvider.

I think that most AuthZ implementation won't be able to fulfill that info though

I must admit that I saw only one implementation within the codebase, so I might underestimate this particular point.

[merlimat]

True, I was thinking about implementing your first proposal. Something that is looping through znodes, and check if auth_policies.namespace_auth.containsKey(role) for the default implementation.

That might be expensive if called frequently from clients in any case with a non-trivial number of namespaces.

I must admit that I saw only one implementation within the codebase, so I might underestimate this particular point.

Yes, there's just one implementation in the codebase, the default where ACLs are stored with the namespace policies on ZK, though there are others using different authorization options, including :

• Use AuthZ information stored in the AuthN credentials (eg: JWT token, or TLS certificate, also containing which resources this token has access to.
• Third party ACLs provider (eg: Okta, Auth0, Vault, etc.. )

[PierreZ]

True, I was thinking about implementing your first proposal. Something that is looping through znodes, and check if auth_policies.namespace_auth.containsKey(role) for the default implementation.

That might be expensive if called frequently from clients in any case with a non-trivial number of namespaces.

I must admit that I saw only one implementation within the codebase, so I might underestimate this particular point.

Yes, there's just one implementation in the codebase, the default where ACLs are stored with the namespace policies on ZK, though there are others using different authorization options, including :

* Use AuthZ information stored in the AuthN credentials (eg: JWT token, or TLS certificate, also containing which resources this token has access to.

* Third party ACLs provider (eg: Okta, Auth0, Vault, etc.. )

Thanks a lot @merlimat for all your insights 😄 How this issue should be adressed in your opinion?

[yjshen]

I'm facing the same situation while developing pulsar-flink-connector.

My scene is as follows:
I was developing a Pulsar catalog in Flink and mapping tenant/namespace to database,topic to table. Therefore, for each role, I need to list all topics it has access to, and users could fire queries based on his/her tables.

[PierreZ]

I cheated for my part, I'm injecting the tenant/namespace in the user field of the SASL_PLAIN to find topics. Leaving the issue opened as the main problem is still here, like for @yjshen.