Support for external principals by adutra · Pull Request #5119 · apache/polaris

adutra · 2026-07-21T13:29:21Z

This change introduces support for fully external principals, that is, principals that are not backed by an entity in Polaris metastore.

The new "principal mode" is configurable on a per-realm basis, and by default, internal mode is used (no changes).

When a realm is configured to use external principals instead:

The authentication type for the realm must NOT be internal (i.e., an external IDP must be used).
The authorizer type must NOT be internal (i.e., an external PDP must be used).
The authenticator bypasses the metastore lookup when authenticating the principal.
The authenticator sets the principal roles directly from the credential's claims, as-is (no prefix stripping, no "ALL" pseudo-role expansion).
The resolver creates synthetic principal and principal role entities if asked by the authorizer to resolve the principal and/or principal roles.

Note: in theory, an external authorizer shouldn't attempt to resolve principal roles, but for now it can happen, until the Authorizer SPI is fully refactored to allow more fine-grained resolutions.

See design doc:

https://docs.google.com/document/d/1VSoN1-QsAJGaM40oWTxeLIoYTp-XlfQM9nG-jqfb_-E/edit

Prior work:

#3250

GitHub issue:

#441

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

Copilot

Pull request overview

This pull request adds a configurable “principal mode” per realm to support external principals (principals not backed by Polaris metastore entities), enabling authentication purely from external IDP credentials and synthetic principal/role resolution for external authorizers (e.g., OPA).

Changes:

Introduces PrincipalMode (INTERNAL/EXTERNAL) and wires it into realm authentication config, readiness checks, and default auth flow.
Updates DefaultAuthenticator and Resolver to support external principals (skip metastore lookup; synthesize principal and role entities during resolution).
Adds OPA testcontainer support plus an end-to-end Keycloak+OPA integration test, and updates related docs/config reference.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tools/testcontainers/opa/src/main/resources/org/apache/polaris/test/opa/Dockerfile-opa-version	Adds Renovate-managed OPA image tag source for tests.
tools/testcontainers/opa/src/main/java/org/apache/polaris/test/opa/OpaTestResource.java	Quarkus test resource to start OPA and inject Polaris authz config.
tools/testcontainers/opa/src/main/java/org/apache/polaris/test/opa/OpaContainer.java	Testcontainers wrapper for OPA plus policy upload helper.
tools/testcontainers/opa/build.gradle.kts	New Gradle module for the OPA testcontainer utilities.
site/content/in-dev/unreleased/managing-security/external-idp/idp-dev-notes.md	Documents how external principal mode changes the auth flow.
site/content/in-dev/unreleased/managing-security/external-idp/_index.md	Adds user-facing documentation for enabling/configuring external principals.
site/content/in-dev/unreleased/configuration/config-sections/smallrye-polaris_authentication.md	Updates generated config reference for `principal-mode`.
runtime/service/src/test/java/org/apache/polaris/service/config/ProductionReadinessChecksTest.java	Adds tests for external principal readiness constraints.
runtime/service/src/test/java/org/apache/polaris/service/auth/DefaultAuthenticatorTest.java	Adds unit coverage for external principal authentication behavior.
runtime/service/src/main/java/org/apache/polaris/service/config/ProductionReadinessChecks.java	Adds readiness checks enforcing external principal configuration constraints.
runtime/service/src/main/java/org/apache/polaris/service/auth/PrincipalMode.java	New enum modeling internal vs external principal mode.
runtime/service/src/main/java/org/apache/polaris/service/auth/internal/broker/InternalPolarisToken.java	Makes internal token type public for cross-package detection.
runtime/service/src/main/java/org/apache/polaris/service/auth/external/mapping/PrincipalMapper.java	Clarifies mapping contract docs for external principals.
runtime/service/src/main/java/org/apache/polaris/service/auth/DefaultAuthenticator.java	Implements external principal authentication path and role handling changes.
runtime/service/src/main/java/org/apache/polaris/service/auth/AuthenticationRealmConfiguration.java	Adds `principalMode()` realm configuration option.
runtime/service/src/intTest/java/org/apache/polaris/service/it/ExternalPrincipalKeycloakOpaIT.java	New E2E integration test for Keycloak-authenticated external principals with OPA authz.
runtime/service/build.gradle.kts	Wires in OPA auth extension and OPA testcontainer dependency.
runtime/defaults/src/main/resources/application.properties	Adds default `polaris.authentication.principal-mode` setting and per-realm example.
polaris-core/src/test/java/org/apache/polaris/core/persistence/ResolverTest.java	Adds coverage for synthetic resolution of external principals/roles.
polaris-core/src/main/java/org/apache/polaris/core/persistence/resolver/Resolver.java	Synthesizes caller principal and principal roles for external principals.
gradle/projects.main.properties	Registers the new `polaris-opa-testcontainer` module.
bom/build.gradle.kts	Exposes the new OPA testcontainer module in the BOM.

Comments suppressed due to low confidence (1)

runtime/service/src/main/java/org/apache/polaris/service/auth/external/mapping/PrincipalMapper.java:49

Javadoc return description for mapPrincipalId() still refers to returning a "Polaris principal" even though the method returns an OptionalLong principal ID. This is confusing for implementers and readers.

   *
   * @param identity the {@link SecurityIdentity} of the user
   * @return the Polaris principal, or an empty optional if no mapping is available
   */
  OptionalLong mapPrincipalId(SecurityIdentity identity);

+    var principalName = entity != null ? entity.getName() : credentials.getPrincipalName();
+    var polarisPrincipal = PolarisPrincipal.of(principalName, attributes, roleSelection.roles());
+
+    LOGGER.debug("Resolved internal principal: {}", polarisPrincipal);
    return polarisPrincipal;


+      URL url = getExternalUrl().resolve("v1/policies/" + name).toURL();
+      HttpURLConnection conn = (HttpURLConnection) url.openConnection();
+      conn.setRequestMethod("PUT");
+      conn.setDoOutput(true);
+      conn.setRequestProperty("Content-Type", "text/plain");
+      try (OutputStream os = conn.getOutputStream()) {
+        os.write(rego.getBytes(StandardCharsets.UTF_8));
+      }
+      int code = conn.getResponseCode();
+      if (code < 200 || code >= 300) {
+        throw new IllegalStateException("OPA policy upload failed, HTTP " + code);
+      }


This change introduces support for fully external principals, that is, principals that are not backed by an entity in Polaris metastore. The new "principal mode" is configurable on a per-realm basis, and by default, internal mode is used (no changes). When a realm is configured to use external principals instead: - The authentication type for the realm must NOT be `internal` (i.e., an external IDP must be used). - The authorizer type must NOT be `internal` (i.e., an external PDP must be used). - The authenticator bypasses the metastore lookup when authenticating the principal. - The authenticator sets the principal roles directly from the credential's claims, as-is (no prefix stripping, no "ALL" pseudo-role expansion). - The resolver creates synthetic principal and principal role entities if asked by the authorizer to resolve the principal and/or principal roles. Note: in theory, an external authorizer shouldn't attempt to resolve principal roles, but for now it can happen, until the Authorizer SPI is fully refactored to allow more fine-grained resolutions. See design doc: https://docs.google.com/document/d/1VSoN1-QsAJGaM40oWTxeLIoYTp-XlfQM9nG-jqfb_-E/edit Prior work: apache#3250 GitHub issue: apache#441

dimas-b

LGTM 👍 Thanks for picking this feature up, @adutra !

flyrain

Can we add a changelog for this PR?

flyrain · 2026-07-21T20:03:39Z

+        polarisPrincipal
+            .getAttribute(PolarisPrincipal.PRINCIPAL_ENTITY_ATTRIBUTE_KEY, PrincipalEntity.class)
+            .isEmpty();
+    if (externalPrincipal) {


External-vs-internal here is decided purely by whether the principal carries a PRINCIPAL_ENTITY_ATTRIBUTE_KEY. Can we use a more deterministic way? like a flag

Introducing another flag seems unnecessary to me: the existence (or not) of a principal entity is precisely what distinguishes an external principal from an internal one.

Copilot AI review requested due to automatic review settings July 21, 2026 13:29

github-project-automation Bot added this to Basic Kanban Board Jul 21, 2026

github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Jul 21, 2026

Copilot started reviewing on behalf of adutra July 21, 2026 13:29 View session

adutra requested review from dimas-b and sungwy July 21, 2026 13:30

Copilot AI reviewed Jul 21, 2026

View reviewed changes

adutra added 2 commits July 27, 2026 20:28

review

aa24ef2

adutra force-pushed the external-principals branch from a37ae41 to aa24ef2 Compare July 27, 2026 18:29

dimas-b previously approved these changes Jul 27, 2026

View reviewed changes

github-project-automation Bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jul 27, 2026

flyrain reviewed Jul 28, 2026

View reviewed changes

add changelog

2b22b08

adutra dismissed dimas-b’s stale review via 2b22b08 July 29, 2026 16:27

adutra mentioned this pull request Aug 3, 2026

Switch principal attributes to AttributeMap #5140

Open

6 tasks

Navigation Menu

Uh oh!

Support for external principals - #5119

Support for external principals#5119
adutra wants to merge 3 commits into
apache:mainfrom
adutra:external-principals

adutra commented Jul 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

dimas-b left a comment

Uh oh!

flyrain left a comment

Uh oh!

flyrain Jul 21, 2026

Uh oh!

adutra Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

adutra commented Jul 21, 2026

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

flyrain Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

adutra Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants