Skip to content

[#12297] feat(secret): Support setSecretBinding/setSecretReference on catalog/schema alter - #12674

Merged
roryqi merged 8 commits into
apache:mainfrom
lasdf1234:feat/entity-secrets-alter-catalog-schema-apache
Aug 28, 2026
Merged

[#12297] feat(secret): Support setSecretBinding/setSecretReference on catalog/schema alter#12674
roryqi merged 8 commits into
apache:mainfrom
lasdf1234:feat/entity-secrets-alter-catalog-schema-apache

Conversation

@lasdf1234

@lasdf1234 lasdf1234 commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

What changes were proposed in this pull request?

Follow-up to #12646 (fileset alter). Adds catalog / schema alter support for setSecretBinding / setSecretReference:

  • API / DTO / OpenAPI / Java & Python clients: new CatalogChange and SchemaChange types
  • CatalogManager / SchemaOperationDispatcher: local prepare*SecretChanges loops calling existing SecretManager alter helpers, rewriting secret ops to setProperty / removeProperty (same pattern as FilesetOperationDispatcher)
  • OperationDispatcher: validate alter upserts for schema secret changes
  • Unit tests for alter removeProperty write-through cleanup and JSON serde

Why are the changes needed?

Create-time secretBindings / secretReferences for catalog and schema are already on main (#12420). Catalog and schema alter still need the typed secret update contract from the entity-secrets design (§5.9.4).

Fix: #12297

Does this PR introduce any user-facing change?

Yes. Catalog and schema alter APIs / REST update requests gain setSecretBinding and setSecretReference.

How was this patch tested?

./gradlew spotlessApply
./gradlew :core:test \
  --tests 'org.apache.gravitino.secret.TestSecretManagerAlter' \
  --tests 'org.apache.gravitino.catalog.TestCatalogManager.testAlterRemovePropertyDeletesWriteThroughSecret' \
  --tests 'org.apache.gravitino.catalog.TestCatalogManager.testSecretRollback' \
  --tests 'org.apache.gravitino.catalog.TestSchemaOperationDispatcher.testAlterRemovePropertyDeletesWriteThroughSecret' \
  --tests 'org.apache.gravitino.catalog.TestSchemaOperationDispatcher.testCreateAndAlterSchemaRejectMaskedPlaceholder' \
  :common:test --tests 'org.apache.gravitino.json.TestRequestJsonSerDe' \
  :docs:build \
  -PskipITs

lasdf1234 and others added 6 commits August 27, 2026 19:11
…nce on catalog/schema alter

Wire catalog/schema alter through SecretManager alter helpers and local
prepare*SecretChanges loops, matching fileset and create-path style.

Co-authored-by: Cursor <cursoragent@cursor.com>
…set pattern

Extract alterCatalogUnderLock and alterSchemaUnderLock, prepare secret changes
before persistence, and use the same rollback pattern as FilesetOperationDispatcher
instead of AtomicReference inside store.update.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep secret preparation on the same entity snapshot as the catalog update
to avoid stale-property races, while retaining alterCatalogUnderLock structure.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ecret flow

Extract SecretAlterChanges and SecretMaterialsHolder, prepare secrets inside
store.update for catalog, managed schema, and fileset catalog, while keeping
dispatcher-side prepare for non-fileset test catalogs.

Co-authored-by: Cursor <cursoragent@cursor.com>
Merge alter-remove secret checks into existing create tests, consolidate
JSON secret update serde into one test, and rename masked-placeholder tests.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
roryqi
roryqi previously approved these changes Aug 27, 2026

@roryqi roryqi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Only use managed-schema store.update path when schema ops is
ManagedSchemaOperations, so Kafka default-schema guards still apply.
Also remove unused Dict import in Python schema_change.

Co-authored-by: Cursor <cursoragent@cursor.com>
Tolerate loadSchema failures in external schema alter so catalogs like
Hudi can still throw their own UnsupportedOperationException.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

Copy link
Copy Markdown

Code Coverage Report

Overall Project 68.41% -0.16% 🟢
Files changed 64.3% 🟢

Module Coverage
aliyun 19.74% 🔴
api 52.07% -0.19% 🟢
authorization-common 85.96% 🟢
authorization-ranger 4.38% 🔴
aws 53.54% 🟢
azure 32.1% 🔴
catalog-common 9.9% 🔴
catalog-fileset 80.21% +12.24% 🟢
catalog-glue 69.24% 🟢
catalog-hive 82.96% 🟢
catalog-jdbc-common 45.69% 🟢
catalog-jdbc-doris 82.44% 🟢
catalog-jdbc-mysql 79.33% 🟢
catalog-jdbc-postgresql 83.39% 🟢
catalog-jdbc-starrocks 79.16% 🟢
catalog-kafka 76.99% 🟢
catalog-lakehouse-generic 60.55% 🟢
catalog-lakehouse-hudi 79.1% 🟢
catalog-lakehouse-iceberg 85.86% 🟢
catalog-lakehouse-paimon 84.26% 🟢
catalog-model 77.99% 🟢
cli 44.48% 🟢
client-java 77.44% -3.01% 🟢
common 55.76% -0.17% 🟢
core 83.5% -0.75% 🟢
filesystem-hadoop3 76.45% 🟢
flink 0.0% 🔴
flink-common 50.29% 🟢
flink-runtime 0.0% 🔴
gcp 32.2% 🔴
hadoop-auth 68.0% 🟢
hadoop-common 17.84% 🔴
hive-metastore-common 53.4% 🟢
iceberg-aliyun-bundle 0.0% 🔴
iceberg-common 64.75% 🟢
iceberg-rest-server 75.68% 🟢
idp-basic 85.98% 🟢
integration-test-common 0.0% 🔴
jobs 62.92% 🟢
lance-common 32.72% -2.88% 🔴
lance-rest-server 64.78% -8.51% 🟢
lineage 53.02% 🟢
optimizer 83.17% 🟢
optimizer-api 21.95% 🔴
server 88.04% 🟢
server-common 80.35% 🟢
spark 28.57% 🔴
spark-common 48.92% 🟢
tencent 69.84% 🟢
trino-connector 47.44% 🟢
Files
Module File Coverage
api CatalogChange.java 44.44% 🔴
SchemaChange.java 44.0% 🔴
catalog-fileset FilesetCatalogOperations.java 81.91% 🟢
client-java DTOConverters.java 51.37% 🔴
common CatalogUpdateRequest.java 59.09% 🔴
SchemaUpdateRequest.java 33.33% 🔴
core SecretMaterialsHolder.java 100.0% 🟢
OperationDispatcher.java 88.64% 🟢
FilesetOperationDispatcher.java 88.6% 🟢
SchemaOperationDispatcher.java 71.29% 🟢
CatalogManager.java 70.73% 🟢
SecretAlterChanges.java 58.33% 🔴
ManagedSchemaOperations.java 37.93% 🔴
SchemaEntityChanges.java 0.0% 🔴
lance-common NamespaceWrapper.java 96.0% 🟢
LanceConfig.java 94.74% 🟢
GravitinoLanceNameSpaceOperations.java 1.38% 🔴
ObjectIdentifier.java 0.0% 🔴
lance-rest-server LanceNamespaceOperations.java 88.37% 🟢
LanceRESTService.java 0.0% 🔴

@roryqi
roryqi merged commit d4316e1 into apache:main Aug 28, 2026
38 checks passed
@lasdf1234
lasdf1234 deleted the feat/entity-secrets-alter-catalog-schema-apache branch August 28, 2026 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[EPIC] Entity connection secrets via GravitinoSecretProvider

2 participants