Skip to content

KNOX-3337: Add optional LDAP role lookup support#1247

Open
smolnar82 wants to merge 1 commit into
apache:masterfrom
smolnar82:KNOX-3337
Open

KNOX-3337: Add optional LDAP role lookup support#1247
smolnar82 wants to merge 1 commit into
apache:masterfrom
smolnar82:KNOX-3337

Conversation

@smolnar82
Copy link
Copy Markdown
Contributor

KNOX-3337 - Introduce LDAP Role Lookup Service

What changes were proposed in this pull request?

This PR introduces a new LDAPRolesLookupService to Apache Knox, allowing for dynamic mapping of users and groups to specific roles during the authentication process. This is particularly useful for downstream services that require fine-grained role information (e.g., platform:admin, workspace:viewer) rather than just raw LDAP groups.

Key changes include:

  • New Service Architecture: Implementation of LDAPRolesLookupService with a pluggable strategy pattern.
  • Lookup Strategies:
    • FileBasedLdapRolesLookup: Maps roles based on a local JSON configuration file.
    • RestApiLdapRolesLookup: Queries an external REST API to retrieve role assignments.
  • KnoxLDAPService Integration: Updated the embedded LDAP service to optionally utilize the role lookup service, allowing it to return resolved roles in place of groups.
  • Header Propagation: Updated AbstractAuthResource (used by PreAuth and ExtAuthz resources) to populate a new X-Knox-Actor-Roles HTTP header when role lookups are successful.
  • Configuration: Added new GatewayConfig properties to configure the lookup strategy, file paths, and API endpoints.

How was this patch tested?

The changes were verified through both existing and newly added unit tests:

  1. Unit Tests for Lookup Logic:
  • FileBasedLdapRolesLookupTest: Verified JSON mapping logic for users and groups.
  • RestApiLdapRolesLookupTest: Verified REST API integration using mocked HTTP clients.
  1. Service Integration Tests:
  • KnoxLDAPServiceTest: Updated to ensure getUserGroups correctly integrates with the lookup service when enabled.
  1. Authentication Resource Tests:
  • PreAuthResourceTest: Added testPopulatingRolesHeader to verify X-Knox-Actor-Roles is set correctly.
  • ExtAuthzResourceTest: Added testPopulatingRolesHeader to verify role propagation in external authorization flows.

Integration Tests

No automated test added this time, but I configured Knox with MockServer as role lookup provider

$ curl -X PUT "http://localhost:55000/mockserver/expectation" \
-H "Content-Type: application/json" \
-d '{
  "httpRequest": {
    "method": "POST",
    "path": "/auth/roles"
  },
  "httpResponse": {
    "statusCode": 200,
    "headers": {
      "Content-Type": ["application/json"]
    },
    "body": {
      "user_id": "alice",
      "roles": [
        {
          "scope": "platform",
          "name": "awc-admin"
        },
        {
          "scope": "ml-workspace-abc",
          "name": "viewer"
        }
      ]
    }
  }
}'

and set the following configs in gateway-reloadable.xml:

<configuration>
    <property>
        <name>gateway.ldap.roles.lookup.strategy</name>
        <value>rest</value>
    </property>
    <property>
        <name>gateway.ldap.roles.lookup.rest.api.endpoint</name>
        <value>http://localhost:55000/auth/roles</value>
    </property>
</configuration>

Verified that the LDAP roles loookup service (as well as the Knox LDAP service) was reconfigured properly:


2026-06-03 11:31:25,224  INFO  knox.gateway (GatewayServer.java:refreshGatewayConfig(275)) - Refreshed gateway config
2026-06-03 11:31:25,225  INFO  services.ldap (DefaultLDAPRolesLookupService.java:onGatewayConfigChanged(66)) - Reloading LDAP roles lookup configuration...
2026-06-03 11:31:25,235  INFO  services.ldap (DefaultLDAPRolesLookupService.java:logStatus(50)) - LDAP roles lookup is disabled
2026-06-03 11:31:25,235  INFO  services.ldap (KnoxLDAPService.java:onGatewayConfigChanged(99)) - Reloading LDAP configuration
2026-06-03 11:31:25,235  INFO  services.ldap (KnoxLDAPServerManager.java:stop(193)) - Stopping LDAP service on port 33,390
2026-06-03 11:31:25,246  INFO  services.ldap (KnoxLDAPServerManager.java:stop(211)) - LDAP service stopped successfully
2026-06-03 11:31:25,247  INFO  services.ldap (BackendFactory.java:createBackend(39)) - Loading backend: ldap (via ServiceLoader)
2026-06-03 11:31:25,247  INFO  services.ldap (LdapProxyBackend.java:initialize(148)) - Loading backend: ldap (via Proxying dc=hadoop,dc=apache,dc=org to ldap://localhost:33389 (dc=hadoop,dc=apache,dc=org) with uid attribute using group searches with recursive group resolution (max depth: 3))
...
2026-06-03 11:32:25,223  INFO  knox.gateway (GatewayServer.java:refreshGatewayConfig(275)) - Refreshed gateway config
2026-06-03 11:32:29,362  INFO  services.ldap (DefaultLDAPRolesLookupService.java:onGatewayConfigChanged(66)) - Reloading LDAP roles lookup configuration...
2026-06-03 11:32:53,041  INFO  services.ldap (DefaultLDAPRolesLookupService.java:logStatus(48)) - LDAP roles lookup is enabled with strategy: rest
2026-06-03 11:32:55,655  INFO  services.ldap (KnoxLDAPService.java:onGatewayConfigChanged(99)) - Reloading LDAP configuration
2026-06-03 11:32:55,655  INFO  services.ldap (KnoxLDAPServerManager.java:stop(193)) - Stopping LDAP service on port 33,390
2026-06-03 11:32:55,659  INFO  services.ldap (KnoxLDAPServerManager.java:stop(211)) - LDAP service stopped successfully
2026-06-03 11:32:55,661  INFO  services.ldap (BackendFactory.java:createBackend(39)) - Loading backend: ldap (via ServiceLoader)
2026-06-03 11:32:55,661  INFO  services.ldap (LdapProxyBackend.java:initialize(148)) - Loading backend: ldap (via Proxying dc=hadoop,dc=apache,dc=org to ldap://localhost:33389 (dc=hadoop,dc=apache,dc=org) with uid attribute using group searches with recursive group resolution (max depth: 3))

Then issued the following curl commands:

$ curl -iu recursiveUser:recursiveUser-password http://localhost:8443/gateway/sandbox/auth/api/v1/pre
HTTP/1.1 200 OK
Date: Wed, 03 Jun 2026 09:33:14 GMT
Set-Cookie: KNOXSESSIONID=node016p2r5x9xtju1sxddl59m3a210.node0; Path=/gateway/sandbox; Secure; HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/sandbox; Max-Age=0; Expires=Tue, 02-Jun-2026 09:33:14 GMT; SameSite=lax
X-Knox-Actor-ID: recursiveUser
X-Knox-Actor-Roles: platform:awc-admin
X-Knox-Actor-Roles: ml-workspace-abc:viewer
Content-Length: 0

$ curl -iku admin:admin-password http://localhost:8443/gateway/sandbox/auth/api/v1/pre
HTTP/1.1 200 OK
Date: Wed, 03 Jun 2026 09:36:34 GMT
Set-Cookie: KNOXSESSIONID=node01tgpz4zn4diz91vp1eiwyymtud1.node0; Path=/gateway/sandbox; Secure; HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/sandbox; Max-Age=0; Expires=Tue, 02-Jun-2026 09:36:34 GMT; SameSite=lax
X-Knox-Actor-ID: admin
X-Knox-Actor-Roles: platform:awc-admin
X-Knox-Actor-Roles: ml-workspace-abc:viewer
Content-Length: 0

Checked the logs:

2026-06-03 11:33:14,804 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG knox.gateway (GatewayFilter.java:doFilter(130)) - Received request: GET /auth/api/v1/pre
2026-06-03 11:33:14,813 58981090-581a-40cb-b5c8-782b8fa6455d INFO  knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn: uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: recursiveUser
2026-06-03 11:33:14,839  DEBUG services.ldap (GroupLookupInterceptor.java:bind(144)) - LDAP Bind: uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org
2026-06-03 11:33:14,848  DEBUG services.ldap (LdapProxyBackend.java:authenticate(278)) - LDAP authentication succeeded for user: uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org
2026-06-03 11:33:19,007 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG knox.gateway (AuthFilterUtils.java:saveImpersonationProvider(138)) - Creating impersonation provider in sandbox / identity-assertion with prefix hadoop.proxyuser and config {admin.groups=*, admin.hosts=*, impersonation.enabled=true}
2026-06-03 11:33:19,009 58981090-581a-40cb-b5c8-782b8fa6455d INFO  knox.gateway (HadoopGroupProviderFilter.java:hadoopGroups(135)) - Using Knox LDAP service to fetch groups...
2026-06-03 11:33:19,018 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(400)) - Recursive group search enabled: true, max depth: 3
2026-06-03 11:33:19,018 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user recursiveUser found 1 group(s) (level1) at depth 0
2026-06-03 11:33:19,019 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:updateCache(488)) - Added parent cn=level2,ou=groups,dc=hadoop,dc=apache,dc=org to cache for group cn=level1,ou=groups,dc=hadoop,dc=apache,dc=org
2026-06-03 11:33:19,020 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user recursiveUser found 1 group(s) (level2) at depth 1
2026-06-03 11:33:19,020 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:updateCache(488)) - Added parent cn=level3,ou=groups,dc=hadoop,dc=apache,dc=org to cache for group cn=level2,ou=groups,dc=hadoop,dc=apache,dc=org
2026-06-03 11:33:19,021 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user recursiveUser found 1 group(s) (level3) at depth 2
2026-06-03 11:33:19,021 58981090-581a-40cb-b5c8-782b8fa6455d WARN  services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(458)) - Recursive group search for user recursiveUser reached max depth 3
2026-06-03 11:33:19,021 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(462)) - Recursive group search for user recursiveUser completed. Total groups found: 3
2026-06-03 11:33:19,086 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG services.ldap (KnoxLDAPService.java:getUserGroups(149)) - LDAP roles lookup for user recursiveUser and groups level1,level2,level3 returned roles: platform:awc-admin,ml-workspace-abc:viewer
2026-06-03 11:33:19,086 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG knox.gateway (HadoopGroupProviderFilter.java:mapGroupPrincipals(115)) - Found groups for principal recursiveUser : [platform:awc-admin, ml-workspace-abc:viewer]
2026-06-03 11:33:19,087 58981090-581a-40cb-b5c8-782b8fa6455d DEBUG knox.gateway (VirtualGroupMapper.java:mapGroups(59)) - User recursiveUser (with group(s) [platform:awc-admin, ml-workspace-abc:viewer]) added to group(s) []
2026-06-03 11:34:07,954  DEBUG services.ldap (GroupLookupInterceptor.java:bind(144)) - LDAP Bind: uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org
2026-06-03 11:34:07,956  DEBUG services.ldap (LdapProxyBackend.java:authenticate(278)) - LDAP authentication succeeded for user: uid=recursiveUser,ou=people,dc=hadoop,dc=apache,dc=org
2026-06-03 11:34:07,958  DEBUG services.ldap (GroupLookupInterceptor.java:search(79)) - LDAP Search: dc=hadoop,dc=apache,dc=org | (|(uid=recursiveUser)(objectClass=referral))
2026-06-03 11:34:07,959  INFO  services.ldap (GroupLookupInterceptor.java:search(119)) - Loaded user from backend: recursiveUser
2026-06-03 11:34:07,962  DEBUG services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(400)) - Recursive group search enabled: true, max depth: 3
2026-06-03 11:34:07,962  DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user recursiveUser found 1 group(s) (level1) at depth 0
2026-06-03 11:34:07,962  DEBUG services.ldap (LdapProxyBackend.java:updateCache(488)) - Added parent cn=level2,ou=groups,dc=hadoop,dc=apache,dc=org to cache for group cn=level1,ou=groups,dc=hadoop,dc=apache,dc=org
2026-06-03 11:34:07,963  DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user recursiveUser found 1 group(s) (level2) at depth 1
2026-06-03 11:34:07,963  DEBUG services.ldap (LdapProxyBackend.java:updateCache(488)) - Added parent cn=level3,ou=groups,dc=hadoop,dc=apache,dc=org to cache for group cn=level2,ou=groups,dc=hadoop,dc=apache,dc=org
2026-06-03 11:34:07,963  DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user recursiveUser found 1 group(s) (level3) at depth 2
2026-06-03 11:34:07,963  WARN  services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(458)) - Recursive group search for user recursiveUser reached max depth 3
2026-06-03 11:34:07,963  DEBUG services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(462)) - Recursive group search for user recursiveUser completed. Total groups found: 3
...
2026-06-03 11:36:34,540 e394d849-9a41-4313-9e48-2505599077c4 DEBUG knox.gateway (GatewayFilter.java:doFilter(130)) - Received request: GET /auth/api/v1/pre
2026-06-03 11:36:34,541 e394d849-9a41-4313-9e48-2505599077c4 INFO  knox.gateway (KnoxLdapRealm.java:getUserDn(688)) - Computed userDn: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: admin
2026-06-03 11:36:34,543  DEBUG services.ldap (GroupLookupInterceptor.java:bind(144)) - LDAP Bind: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
2026-06-03 11:36:34,546  DEBUG services.ldap (LdapProxyBackend.java:authenticate(278)) - LDAP authentication succeeded for user: uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
2026-06-03 11:36:34,547 e394d849-9a41-4313-9e48-2505599077c4 INFO  knox.gateway (HadoopGroupProviderFilter.java:hadoopGroups(135)) - Using Knox LDAP service to fetch groups...
2026-06-03 11:36:34,549 e394d849-9a41-4313-9e48-2505599077c4 DEBUG services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(400)) - Recursive group search enabled: true, max depth: 3
2026-06-03 11:36:34,549 e394d849-9a41-4313-9e48-2505599077c4 DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user admin found 1 group(s) (admin) at depth 0
2026-06-03 11:36:34,550 e394d849-9a41-4313-9e48-2505599077c4 DEBUG services.ldap (LdapProxyBackend.java:logRecursiveSearchProgress(515)) - Recursive group search for user admin found 0 group(s) () at depth 1
2026-06-03 11:36:34,550 e394d849-9a41-4313-9e48-2505599077c4 DEBUG services.ldap (LdapProxyBackend.java:resolveGroupsRecursive(462)) - Recursive group search for user admin completed. Total groups found: 1
2026-06-03 11:36:34,560 e394d849-9a41-4313-9e48-2505599077c4 DEBUG services.ldap (KnoxLDAPService.java:getUserGroups(149)) - LDAP roles lookup for user admin and groups admin returned roles: platform:awc-admin,ml-workspace-abc:viewer
2026-06-03 11:36:34,561 e394d849-9a41-4313-9e48-2505599077c4 DEBUG knox.gateway (HadoopGroupProviderFilter.java:mapGroupPrincipals(115)) - Found groups for principal admin : [platform:awc-admin, ml-workspace-abc:viewer]
2026-06-03 11:36:34,561 e394d849-9a41-4313-9e48-2505599077c4 DEBUG knox.gateway (VirtualGroupMapper.java:mapGroups(59)) - User admin (with group(s) [platform:awc-admin, ml-workspace-abc:viewer]) added to group(s) []

I also ran the new KnoxCLI command:

$ bin/knoxcli.sh ldap-user-groups-test --u recursiveUser
Querying KnoxLDAPService for groups of user: recursiveUser
recursiveUser is a member of: platform:awc-admin, ml-workspace-abc:viewer

UI changes

N/A

@smolnar82 smolnar82 self-assigned this Jun 3, 2026
@smolnar82 smolnar82 added the ldap label Jun 3, 2026
@smolnar82
Copy link
Copy Markdown
Contributor Author

Cc. @handavid

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 3, 2026

Test Results

22 tests   22 ✅  2s ⏱️
 1 suites   0 💤
 1 files     0 ❌

Results for commit 01a5749.

@smolnar82 smolnar82 requested a review from pzampino June 3, 2026 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant