Skip to content

[#10960] feat(authn): wire built-in IdP storage#11167

Closed
lasdf1234 wants to merge 3 commits into
apache:mainfrom
lasdf1234:add-built-in-storage
Closed

[#10960] feat(authn): wire built-in IdP storage#11167
lasdf1234 wants to merge 3 commits into
apache:mainfrom
lasdf1234:add-built-in-storage

Conversation

@lasdf1234
Copy link
Copy Markdown
Collaborator

@lasdf1234 lasdf1234 commented May 20, 2026

What changes were proposed in this pull request?

This PR wires built-in IdP storage into Gravitino runtime on top of main, including:

  • Core metadata services (IdpUserMetaService, IdpGroupMetaService) and JDBC backend integration
  • IdpManagerFactory / IdpManager and GravitinoEnv lifecycle wiring
  • idp-basic plugin providers and SPI registration

Rebased onto latest apache/gravitino main (merged with -X theirs so upstream changes take precedence on conflicts).

Part of #10960 / subtask #11044

Why are the changes needed?

The relational storage classes and plugin mappers landed on main via #11127 and #11140, but runtime wiring (env initialization, JDBC backend delegation, provider loading) still needs to land so the built-in IdP storage is usable.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

./gradlew :core:compileJava :plugins:idp-basic:compileJava :plugins:idp-basic:compileTestJava -PskipITs

./gradlew :core:test :plugins:idp-basic:test -PskipITs -PskipDockerTests=true -PskipWeb=true \
  --tests org.apache.gravitino.TestGravitinoEnv \
  --tests org.apache.gravitino.TestIdpManagerFactory \
  --tests org.apache.gravitino.storage.relational.service.TestIdpGroupMetaService \
  --tests org.apache.gravitino.storage.relational.service.TestIdpUserMetaService \
  --tests org.apache.gravitino.storage.provider.TestIdpBasicGroupMetaProvider \
  --tests org.apache.gravitino.storage.provider.TestIdpBasicUserMetaProvider

Made with Cursor

lasdf1234 and others added 3 commits May 11, 2026 23:14
@lasdf1234
[apache#10960] feat(authn): add idp-basic storage relational models
5998dfa 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lasdf1234
[apache#10960] feat(authn): wire built-in IdP storage
ee42688 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@lasdf1234
Merge apache/gravitino main into add-built-in-storage (prefer main on…
84c0d4b 
… conflicts)
Copilot AI review requested due to automatic review settings May 20, 2026 08:13
@lasdf1234 lasdf1234 mentioned this pull request May 20, 2026
Closed
Copilot AI reviewed May 20, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR adds built-in IdP (user/group) relational persistence and SPI-based loading so core can delegate IdP operations to the idp-basic plugin, including SQL providers for multiple JDBC backends and associated tests.

Changes:

  • Introduced IdP user/group POs, MyBatis mappers, and SQL provider factories with backend-specific implementations (MySQL/H2/PostgreSQL).
  • Added core-side SPI interfaces, service facades, and a ServiceLoader-based provider loader for IdP metadata operations.
  • Wired IdP support into core environment lifecycle and legacy GC paths, plus extensive new unit tests and service registrations.

Reviewed changes

Copilot reviewed 63 out of 63 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/po/TestIdpUserPO.java Adds unit tests for IdP user PO builder/validation/equality.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/po/TestIdpGroupUserRelPO.java Adds unit tests for group-user relation PO builder/validation/equality.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/po/TestIdpGroupPO.java Adds unit tests for IdP group PO builder/validation/equality.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/postgresql/TestIdpUserMetaPostgreSQLProvider.java Adds PostgreSQL-specific SQL expectations for IdP user meta provider.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/postgresql/TestIdpGroupUserRelPostgreSQLProvider.java Adds PostgreSQL-specific SQL expectations for group-user relation provider.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/postgresql/TestIdpGroupMetaPostgreSQLProvider.java Adds PostgreSQL-specific SQL expectations for IdP group meta provider.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/h2/TestIdpUserMetaH2Provider.java Adds H2 provider coverage via base SQL provider tests.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/h2/TestIdpGroupUserRelH2Provider.java Adds H2 provider coverage via base SQL provider tests.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/h2/TestIdpGroupMetaH2Provider.java Adds H2 provider coverage via base SQL provider tests.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestIdpUserMetaBaseSQLProvider.java Adds base SQL provider tests for IdP user meta queries/mutations.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestIdpGroupUserRelBaseSQLProvider.java Adds base SQL provider tests for group-user relation SQL (including empty/null list handling).
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/base/TestIdpGroupMetaBaseSQLProvider.java Adds base SQL provider tests for IdP group meta queries/mutations.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/mapper/provider/TestIdpBasicMapperPackageProvider.java Tests mapper package provider returns expected mapper classes.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/mapper/TestIdpUserMetaMySQLProvider.java Adds MySQL provider coverage via base SQL provider tests.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/mapper/TestIdpGroupUserRelMySQLProvider.java Adds MySQL provider coverage via base SQL provider tests.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/relational/mapper/TestIdpGroupMetaMySQLProvider.java Adds MySQL provider coverage via base SQL provider tests.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/provider/TestIdpBasicUserMetaProvider.java Tests singleton/type bridge and ServiceLoader registration for IdP user provider.
plugins/idp-basic/src/test/java/org/apache/gravitino/storage/provider/TestIdpBasicGroupMetaProvider.java Tests singleton/type bridge and ServiceLoader registration for IdP group provider.
plugins/idp-basic/src/main/resources/META-INF/services/org.apache.gravitino.storage.relational.provider.IdpUserMetaProvider Registers idp-basic IdP user provider for SPI loading.
plugins/idp-basic/src/main/resources/META-INF/services/org.apache.gravitino.storage.relational.provider.IdpGroupMetaProvider Registers idp-basic IdP group provider for SPI loading.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/po/IdpUserPO.java Adds IdP user persistence object with builder + validation.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/po/IdpGroupUserRelPO.java Adds group-user relation persistence object with builder + validation.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/po/IdpGroupPO.java Adds IdP group persistence object with builder + validation.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/postgresql/IdpUserMetaPostgreSQLProvider.java PostgreSQL overrides for soft delete timestamp and legacy deletion SQL.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/postgresql/IdpGroupUserRelPostgreSQLProvider.java PostgreSQL overrides for relation soft delete timestamp and legacy deletion SQL.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/postgresql/IdpGroupMetaPostgreSQLProvider.java PostgreSQL overrides for group soft delete timestamp and legacy deletion SQL.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/h2/IdpUserMetaH2Provider.java H2 provider adapter extending base SQL provider.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/h2/IdpGroupUserRelH2Provider.java H2 provider adapter extending base SQL provider.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/h2/IdpGroupMetaH2Provider.java H2 provider adapter extending base SQL provider.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/base/IdpUserMetaBaseSQLProvider.java Adds base SQL templates for IdP user meta operations.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/base/IdpGroupUserRelBaseSQLProvider.java Adds base SQL templates for group-user relation operations (with IN-list guarding).
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/base/IdpGroupMetaBaseSQLProvider.java Adds base SQL templates for IdP group meta operations.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/IdpBasicMapperPackageProvider.java Exposes plugin mapper classes to core mapper scanning.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/IdpUserMetaSQLProviderFactory.java Adds backend-based SQL provider selection for IdP user meta.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/IdpUserMetaMapper.java Adds MyBatis mapper for IdP user meta operations.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/IdpGroupUserRelSQLProviderFactory.java Adds backend-based SQL provider selection for group-user relations.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/IdpGroupUserRelMapper.java Adds MyBatis mapper for group-user relation operations.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/IdpGroupMetaSQLProviderFactory.java Adds backend-based SQL provider selection for IdP group meta.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/IdpGroupMetaMapper.java Adds MyBatis mapper for IdP group meta operations.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/provider/IdpUserMetaProvider.java Adds plugin-side compatibility bridge for core IdP user provider SPI.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/provider/IdpGroupMetaProvider.java Adds plugin-side compatibility bridge for core IdP group provider SPI.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/provider/IdpBasicUserMetaProvider.java Implements IdP user provider backed by MyBatis mappers.
plugins/idp-basic/src/main/java/org/apache/gravitino/storage/provider/IdpBasicGroupMetaProvider.java Implements IdP group provider backed by MyBatis mappers.
core/src/test/java/org/apache/gravitino/storage/relational/service/TestIdpUserMetaService.java Tests IdP user service delegation and ServiceLoader error cases.
core/src/test/java/org/apache/gravitino/storage/relational/service/TestIdpGroupMetaService.java Tests IdP group service delegation and ServiceLoader error cases.
core/src/test/java/org/apache/gravitino/storage/relational/TestJDBCBackend.java Extends legacy record existence checks to IdP tables.
core/src/test/java/org/apache/gravitino/storage/relational/BackendTestExtension.java Adds metrics sliding window config stubbing required by backend startup.
core/src/test/java/org/apache/gravitino/TestIdpManagerFactory.java Tests IdP manager SPI loading behavior and error handling.
core/src/test/java/org/apache/gravitino/TestGravitinoEnv.java Tests env initialization/shutdown behavior for IdP manager integration.
core/src/main/java/org/apache/gravitino/storage/relational/service/IdpUserMetaService.java Adds core facade delegating IdP user operations to plugin provider.
core/src/main/java/org/apache/gravitino/storage/relational/service/IdpGroupMetaService.java Adds core facade delegating IdP group operations to plugin provider.
core/src/main/java/org/apache/gravitino/storage/relational/provider/IdpUserMetaProvider.java Adds core SPI for IdP user operations.
core/src/main/java/org/apache/gravitino/storage/relational/provider/IdpMetaProviderLoader.java Adds single-implementation ServiceLoader helper for IdP providers.
core/src/main/java/org/apache/gravitino/storage/relational/provider/IdpGroupMetaProvider.java Adds core SPI for IdP group + relation operations.
core/src/main/java/org/apache/gravitino/storage/relational/po/IdpUserMeta.java Adds core model interface for IdP user meta.
core/src/main/java/org/apache/gravitino/storage/relational/po/IdpGroupUserRelMeta.java Adds core model interface for group-user relation meta.
core/src/main/java/org/apache/gravitino/storage/relational/po/IdpGroupMeta.java Adds core model interface for IdP group meta.
core/src/main/java/org/apache/gravitino/storage/relational/JDBCBackend.java Wires IdP entity types into hard delete legacy data paths.
core/src/main/java/org/apache/gravitino/authorization/IdpManager.java Introduces IdP manager API for user/group management in core.
core/src/main/java/org/apache/gravitino/IdpManagerFactory.java Adds ServiceLoader-based factory for IdP manager SPI.
core/src/main/java/org/apache/gravitino/GravitinoEnv.java Integrates IdP manager lifecycle into core env init/shutdown.
core/src/main/java/org/apache/gravitino/Entity.java Adds new entity types for IDP_USER and IDP_GROUP.
core/build.gradle.kts Adds idp-basic plugin as test runtime dependency for core tests.
Comments suppressed due to low confidence (2)

plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/base/IdpUserMetaBaseSQLProvider.java:1

  • This SQL script can render invalid SQL when userNames is empty (e.g., IN ()) and may fail if userNames is null. The group-user-rel providers in this PR handle empty/null lists via <choose> and AND 1 = 0; selectIdpUsers should apply the same pattern (or otherwise guarantee a valid predicate) to prevent runtime SQL errors if callers bypass the current Java-side empty-list guard.
    plugins/idp-basic/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/postgresql/IdpUserMetaPostgreSQLProvider.java:1
  • This override ignores the deletedAt parameter and always uses the database clock. That creates cross-backend semantic differences (MySQL/H2 use the provided parameter) and can surprise API consumers that pass deletedAt expecting it to be persisted. If using DB time is required for PostgreSQL, consider documenting this explicitly (e.g., Javadoc on the override and/or on the SPI/mapper method) and suppressing the unused-parameter warning; alternatively, align behavior across backends by using #{deletedAt} everywhere.

Comment thread core/src/main/java/org/apache/gravitino/storage/relational/provider/IdpMetaProviderLoader.java
Comment on lines +32 to +49
public static <T> T loadService(Class<T> serviceClass) {
List<T> services = new ArrayList<>();
for (T service : ServiceLoader.load(serviceClass)) {
services.add(service);
}

if (services.isEmpty()) {
throw new IllegalStateException(
String.format("No %s implementation found", serviceClass.getSimpleName()));
}

if (services.size() > 1) {
throw new IllegalStateException(
String.format("Multiple %s implementations found", serviceClass.getSimpleName()));
}

return services.get(0);
}
Comment thread core/src/main/java/org/apache/gravitino/GravitinoEnv.java
Comment on lines +463 to +467
/** Get the built-in IdP manager implementation. */
public IdpManager idpManager() {
Preconditions.checkArgument(idpManager != null, "GravitinoEnv is not initialized.");
return idpManager;
}
Comment thread core/src/main/java/org/apache/gravitino/GravitinoEnv.java
try {
idpManager.close();
} catch (Exception e) {
LOG.warn("Failed to close IdpManager", e);
Comment thread core/src/main/java/org/apache/gravitino/IdpManagerFactory.java
Comment on lines +45 to +62
private static <T> T loadService(Class<T> serviceClass) {
List<T> services = new ArrayList<>();
for (T service : ServiceLoader.load(serviceClass)) {
services.add(service);
}

if (services.isEmpty()) {
throw new IllegalStateException(
String.format("No %s implementation found", serviceClass.getSimpleName()));
}

if (services.size() > 1) {
throw new IllegalStateException(
String.format("Multiple %s implementations found", serviceClass.getSimpleName()));
}

return services.get(0);
}
Comment thread .../main/java/org/apache/gravitino/storage/relational/mapper/IdpUserMetaSQLProviderFactory.java
.getDatabaseId();

JDBCBackendType jdbcBackendType = JDBCBackendType.fromString(databaseId);
return IDP_USER_META_SQL_PROVIDER_MAP.get(jdbcBackendType);
@lasdf1234 lasdf1234 marked this pull request as draft May 20, 2026 08:16
@lasdf1234
Copy link
Copy Markdown
Collaborator Author

lasdf1234 commented May 20, 2026

Closing per maintainer request.

@lasdf1234 lasdf1234 closed this May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lasdf1234