From 3bb460b1a3504d0bf99c6824ce78630011b98f48 Mon Sep 17 00:00:00 2001 From: thetumbled <843221020@qq.com> Date: Tue, 23 Apr 2024 16:15:10 +0800 Subject: [PATCH 1/7] add pip-347. --- pip/pip-347.md | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 pip/pip-347.md diff --git a/pip/pip-347.md b/pip/pip-347.md new file mode 100644 index 0000000000000..a37678b30106e --- /dev/null +++ b/pip/pip-347.md @@ -0,0 +1,68 @@ + +# PIP-347: add role field in consumer's stat + +# Background knowledge + +- In a typical Pulsar deployment, we set `authenticationEnabled=true` and `authorizationEnabled=true` to enable authentication and authorization. +- When users want to create a subscription in the management platform, we will grant the `consume` permission of topic and the permission of +that subscription to the user's token. so that user can create a subscription successfully. +- We also set config `allowAutoSubscriptionCreation` to be true to allow user creating subscriptions automatically if user's token has the `consume` permission. +But, **auto-creation of subscriptions will not grant the permission of the subscription to the user's token**. +- The permission control flow of subscription: + - we can only grant and revoke the permission of a subscription by the `grant-permission` and `revoke-permission` REST API. + - The permission of subscription is in the namespace level. When a user creates a subscription `sub` for topic `persistent://public/default/test`, + broker will check if user's token has the `consume` permission of `persistent://public/default/test`, if yes, broker will retrieve the permission of all subscriptions under the namespace `public/default` and get the role list of subscription `sub`. + - **If the role list is empty or null, broker allow to create the subscription.** However, the permission of the subscription is still not granted to the user's token. + - If the role list is not empty, broker will check if the role corresponding to the user's token is in the role list. If yes, broker allow to create the subscription. + - If the role corresponding to the user's token is not in the role list, broker will reject the request. + +There is a permission hole in the current design. If a user obtain the `consume` permission of a topic in management platform, he can create infinite subscriptions without notifying the administrator. +For example, user `jack` can obtain the `consume` permission of topic `persistent://public/default/test` and the permission of subscription `sub` in management platform. +At this time, the role list under the namespace `public/default` is +``` +'sub' -> ['jack'] +``` +Then, user `jack` can create other subscriptions `sub1`, `sub2`,... for topic `persistent://public/default/test` without notifying the administrator. The role list under the namespace `public/default` remain unchanged as +``` +'sub' -> ['jack'] +``` +The administrator can't know who is the owner of the subscriptions `sub1`, `sub2`,... + +What is worse, if another user `tom` obtain the permission of subscription `sub1` under the same namespace `public/default` in management platform, that result into the role list under the namespace `public/default` become to +``` +'sub' -> ['jack'] +'sub1' -> ['tom'] +``` +`jack` lose the permission of subscription `sub1` immediately. Because jack do not have the permission of subscription `sub1` really. + +He just uses an unregistered subscription `sub1` owned by nobody to consume the messages. Once there are any people obtain the permission of subscription `sub1`, his task consuming `sub1` will be shutdown immediately. + +# Motivation + +Such kind of permission lost problem cause us lots of problem. We need a tool to find out the owner of these unregistered subscriptions first. + +# Goals +Add a field `role` in the consumer's stat to show the owner of this consumer. + +## In Scope + +- help administrator to find out the owner of the consumer + +## Out of Scope + +Maybe we improve the permission control of subscriptions in the future to fix the permission lost problem described above. + + +# Backward & Forward Compatibility + +Fully compatible. + +# General Notes + +# Links + + +* Mailing List discussion thread: +* Mailing List voting thread: From ebccecd167ce1983a344701c2d43dd15fce1a455 Mon Sep 17 00:00:00 2001 From: thetumbled <843221020@qq.com> Date: Tue, 23 Apr 2024 16:25:25 +0800 Subject: [PATCH 2/7] add discussion link. --- pip/pip-347.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pip/pip-347.md b/pip/pip-347.md index a37678b30106e..99146d41bc31f 100644 --- a/pip/pip-347.md +++ b/pip/pip-347.md @@ -64,5 +64,5 @@ Fully compatible. -* Mailing List discussion thread: +* Mailing List discussion thread: https://lists.apache.org/thread/p9y9r8pb7ygk8f0jd121c1121phvzd09 * Mailing List voting thread: From 18aa29e045169626531b7b3f3d8d45bd48a26044 Mon Sep 17 00:00:00 2001 From: thetumbled <843221020@qq.com> Date: Sun, 5 May 2024 11:19:06 +0800 Subject: [PATCH 3/7] add conf. --- pip/pip-347.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/pip/pip-347.md b/pip/pip-347.md index 99146d41bc31f..369a418990276 100644 --- a/pip/pip-347.md +++ b/pip/pip-347.md @@ -39,10 +39,9 @@ He just uses an unregistered subscription `sub1` owned by nobody to consume the # Motivation -Such kind of permission lost problem cause us lots of problem. We need a tool to find out the owner of these unregistered subscriptions first. +Such kind of permission lost problem cause us lots of problem. We need a tool to find out the owner of these unregistered subscriptions first, and then we can grant the permission of these subscriptions to the real owner, in case of the permission lost problem. # Goals -Add a field `role` in the consumer's stat to show the owner of this consumer. ## In Scope @@ -53,6 +52,19 @@ Add a field `role` in the consumer's stat to show the owner of this consumer. Maybe we improve the permission control of subscriptions in the future to fix the permission lost problem described above. +# Detailed Design + +## Design & Implementation Details +- Add a field `role` in the consumer's stat to show the owner of this consumer. +- Add a new conf `exposeRoleInConsumerStat` to enable or disable this feature. + +## Public-facing Changes + +### Configuration + +- Add a new conf `exposeRoleInConsumerStat` to enable or disable this feature. The default value is false. + + # Backward & Forward Compatibility Fully compatible. From 3180731797ac2ec717317a664a3b668f9bc61f9d Mon Sep 17 00:00:00 2001 From: thetumbled <843221020@qq.com> Date: Mon, 6 May 2024 17:08:27 +0800 Subject: [PATCH 4/7] change doc. --- pip/pip-347.md | 55 +++++++++----------------------------------------- 1 file changed, 10 insertions(+), 45 deletions(-) diff --git a/pip/pip-347.md b/pip/pip-347.md index 369a418990276..33b2fba2de192 100644 --- a/pip/pip-347.md +++ b/pip/pip-347.md @@ -3,66 +3,31 @@ # Background knowledge -- In a typical Pulsar deployment, we set `authenticationEnabled=true` and `authorizationEnabled=true` to enable authentication and authorization. -- When users want to create a subscription in the management platform, we will grant the `consume` permission of topic and the permission of -that subscription to the user's token. so that user can create a subscription successfully. -- We also set config `allowAutoSubscriptionCreation` to be true to allow user creating subscriptions automatically if user's token has the `consume` permission. -But, **auto-creation of subscriptions will not grant the permission of the subscription to the user's token**. -- The permission control flow of subscription: - - we can only grant and revoke the permission of a subscription by the `grant-permission` and `revoke-permission` REST API. - - The permission of subscription is in the namespace level. When a user creates a subscription `sub` for topic `persistent://public/default/test`, - broker will check if user's token has the `consume` permission of `persistent://public/default/test`, if yes, broker will retrieve the permission of all subscriptions under the namespace `public/default` and get the role list of subscription `sub`. - - **If the role list is empty or null, broker allow to create the subscription.** However, the permission of the subscription is still not granted to the user's token. - - If the role list is not empty, broker will check if the role corresponding to the user's token is in the role list. If yes, broker allow to create the subscription. - - If the role corresponding to the user's token is not in the role list, broker will reject the request. - -There is a permission hole in the current design. If a user obtain the `consume` permission of a topic in management platform, he can create infinite subscriptions without notifying the administrator. -For example, user `jack` can obtain the `consume` permission of topic `persistent://public/default/test` and the permission of subscription `sub` in management platform. -At this time, the role list under the namespace `public/default` is -``` -'sub' -> ['jack'] -``` -Then, user `jack` can create other subscriptions `sub1`, `sub2`,... for topic `persistent://public/default/test` without notifying the administrator. The role list under the namespace `public/default` remain unchanged as -``` -'sub' -> ['jack'] -``` -The administrator can't know who is the owner of the subscriptions `sub1`, `sub2`,... - -What is worse, if another user `tom` obtain the permission of subscription `sub1` under the same namespace `public/default` in management platform, that result into the role list under the namespace `public/default` become to -``` -'sub' -> ['jack'] -'sub1' -> ['tom'] -``` -`jack` lose the permission of subscription `sub1` immediately. Because jack do not have the permission of subscription `sub1` really. - -He just uses an unregistered subscription `sub1` owned by nobody to consume the messages. Once there are any people obtain the permission of subscription `sub1`, his task consuming `sub1` will be shutdown immediately. +During the operation and maintenance process, there are many users asking administrator for help to find out the consumers of a topic and notify them about the business change. +Administrators can call `bin/pulsar-admin topics partitioned-stats` to find out the `ip:port` of the consumers, but no role info. So administrators need to take a lot of time to +communicate with users to find out the owner based on the `ip:port`. It's a troublesome work and low efficiency. # Motivation -Such kind of permission lost problem cause us lots of problem. We need a tool to find out the owner of these unregistered subscriptions first, and then we can grant the permission of these subscriptions to the real owner, in case of the permission lost problem. +This pip can help to solve such kind of problem. By adding a field `role` in the consumer's stat, the administrator can find out the owner of the consumer directly. +It can save a lot of time and improve the efficiency of the operation and maintenance process. # Goals -## In Scope - - help administrator to find out the owner of the consumer -## Out of Scope - -Maybe we improve the permission control of subscriptions in the future to fix the permission lost problem described above. - - # Detailed Design ## Design & Implementation Details - Add a field `role` in the consumer's stat to show the owner of this consumer. -- Add a new conf `exposeRoleInConsumerStat` to enable or disable this feature. +- For some use cases, the role info is sensitive, so we only expose the role info when the cli tool option `--expose-role` is set + and the user is a super-user or a tenant admin. ## Public-facing Changes -### Configuration - -- Add a new conf `exposeRoleInConsumerStat` to enable or disable this feature. The default value is false. +### CLI +Add a new option `--expose-role` for command `bin/pulsar-admin topics partitioned-stats` +and `bin/pulsar-admin topics stats` to show the role info of the consumer. # Backward & Forward Compatibility From d35627199011719540b85844957eec95ff78c89b Mon Sep 17 00:00:00 2001 From: thetumbled <843221020@qq.com> Date: Mon, 6 May 2024 18:33:30 +0800 Subject: [PATCH 5/7] update doc. --- pip/pip-347.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/pip/pip-347.md b/pip/pip-347.md index 33b2fba2de192..af07393da1705 100644 --- a/pip/pip-347.md +++ b/pip/pip-347.md @@ -25,10 +25,22 @@ It can save a lot of time and improve the efficiency of the operation and mainte ## Public-facing Changes +### Public API +Add a new option `exposeRole` for following endpoints: +- `GET /admin/v2/persistent/{tenant}/{namespace}/{topic}/partitioned-stats` +- `GET /admin/v2/persistent/{tenant}/{namespace}/{topic}/stats` +- `GET /admin/v2/non-persistent/{tenant}/{namespace}/{topic}/partitioned-stats` +- `GET /admin/v2/non-persistent/{tenant}/{namespace}/{topic}/stats` + +`exposeRole` is a boolean value, default is false. If it's true and the user is a super-user or a tenant admin, the role info will be exposed. + ### CLI Add a new option `--expose-role` for command `bin/pulsar-admin topics partitioned-stats` and `bin/pulsar-admin topics stats` to show the role info of the consumer. +### ADMIN API +Add a new input parameter `exposeRole` in the `admin.topics().getPartitionedStats` and `admin.topics().getStats` to show the role info of the consumer. + # Backward & Forward Compatibility From 9f351fc540f4aa7a160e4e70f376018e1a7d79fd Mon Sep 17 00:00:00 2001 From: thetumbled <843221020@qq.com> Date: Mon, 6 May 2024 19:45:57 +0800 Subject: [PATCH 6/7] update doc. --- pip/pip-347.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/pip/pip-347.md b/pip/pip-347.md index af07393da1705..ca82202031f7b 100644 --- a/pip/pip-347.md +++ b/pip/pip-347.md @@ -14,7 +14,7 @@ It can save a lot of time and improve the efficiency of the operation and mainte # Goals -- help administrator to find out the owner of the consumer +- help administrator to find out the owner of the consumer. # Detailed Design @@ -34,12 +34,18 @@ Add a new option `exposeRole` for following endpoints: `exposeRole` is a boolean value, default is false. If it's true and the user is a super-user or a tenant admin, the role info will be exposed. + ### CLI Add a new option `--expose-role` for command `bin/pulsar-admin topics partitioned-stats` and `bin/pulsar-admin topics stats` to show the role info of the consumer. + ### ADMIN API -Add a new input parameter `exposeRole` in the `admin.topics().getPartitionedStats` and `admin.topics().getStats` to show the role info of the consumer. +Add new methods: +- `org.apache.pulsar.client.admin.Topics#getPartitionedStats(String topic, boolean perPartition, boolean getPreciseBacklog, boolean subscriptionBacklogSize, boolean exposeRole)` +- `org.apache.pulsar.client.admin.Topics#getPartitionedStatsAsync(String topic, boolean perPartition, boolean getPreciseBacklog, boolean subscriptionBacklogSize, boolean exposeRole)` +- `org.apache.pulsar.client.admin.Topics#getStats(String topic, boolean getPreciseBacklog, boolean subscriptionBacklogSize, boolean exposeRole)` +- `org.apache.pulsar.client.admin.Topics#getStatsAsync(String topic, boolean getPreciseBacklog, boolean subscriptionBacklogSize, boolean exposeRole)` # Backward & Forward Compatibility From dd7865dbbb0db12002b69d469e53be9a51bc2583 Mon Sep 17 00:00:00 2001 From: thetumbled <843221020@qq.com> Date: Mon, 13 May 2024 18:13:24 +0800 Subject: [PATCH 7/7] not sensitive. --- pip/pip-347.md | 36 +++++------------------------------- 1 file changed, 5 insertions(+), 31 deletions(-) diff --git a/pip/pip-347.md b/pip/pip-347.md index ca82202031f7b..5326fed353374 100644 --- a/pip/pip-347.md +++ b/pip/pip-347.md @@ -5,48 +5,22 @@ During the operation and maintenance process, there are many users asking administrator for help to find out the consumers of a topic and notify them about the business change. Administrators can call `bin/pulsar-admin topics partitioned-stats` to find out the `ip:port` of the consumers, but no role info. So administrators need to take a lot of time to -communicate with users to find out the owner based on the `ip:port`. It's a troublesome work and low efficiency. +communicate with users to find out the owner based on the `ip:port`. It's a troublesome work and low efficiency, or even can't find out the owner. # Motivation -This pip can help to solve such kind of problem. By adding a field `role` in the consumer's stat, the administrator can find out the owner of the consumer directly. +This pip can help to solve such kind of problem. By adding a field `appId` in the consumer's stat. +For cluster with JWT-based authentication, the administrator can find out the owner of the consumer directly. It can save a lot of time and improve the efficiency of the operation and maintenance process. # Goals -- help administrator to find out the owner of the consumer. +- help administrator to find out the owner of the consumer for cluster with JWT-based authentication. # Detailed Design ## Design & Implementation Details -- Add a field `role` in the consumer's stat to show the owner of this consumer. -- For some use cases, the role info is sensitive, so we only expose the role info when the cli tool option `--expose-role` is set - and the user is a super-user or a tenant admin. - -## Public-facing Changes - -### Public API -Add a new option `exposeRole` for following endpoints: -- `GET /admin/v2/persistent/{tenant}/{namespace}/{topic}/partitioned-stats` -- `GET /admin/v2/persistent/{tenant}/{namespace}/{topic}/stats` -- `GET /admin/v2/non-persistent/{tenant}/{namespace}/{topic}/partitioned-stats` -- `GET /admin/v2/non-persistent/{tenant}/{namespace}/{topic}/stats` - -`exposeRole` is a boolean value, default is false. If it's true and the user is a super-user or a tenant admin, the role info will be exposed. - - -### CLI -Add a new option `--expose-role` for command `bin/pulsar-admin topics partitioned-stats` -and `bin/pulsar-admin topics stats` to show the role info of the consumer. - - -### ADMIN API -Add new methods: -- `org.apache.pulsar.client.admin.Topics#getPartitionedStats(String topic, boolean perPartition, boolean getPreciseBacklog, boolean subscriptionBacklogSize, boolean exposeRole)` -- `org.apache.pulsar.client.admin.Topics#getPartitionedStatsAsync(String topic, boolean perPartition, boolean getPreciseBacklog, boolean subscriptionBacklogSize, boolean exposeRole)` -- `org.apache.pulsar.client.admin.Topics#getStats(String topic, boolean getPreciseBacklog, boolean subscriptionBacklogSize, boolean exposeRole)` -- `org.apache.pulsar.client.admin.Topics#getStatsAsync(String topic, boolean getPreciseBacklog, boolean subscriptionBacklogSize, boolean exposeRole)` - +- Add a field `appId` in the consumer's stat, which can show the owner of this consumer for JWT-based authentication users. # Backward & Forward Compatibility