[#10878] feat(core): add relational storage for logical views with versioning by mchades · Pull Request #10879 · apache/gravitino

mchades · 2026-04-27T16:08:12Z

What changes were proposed in this pull request?

Adds the relational storage layer for the Logical View Management feature:

New ViewEntity (implements View, HasIdentifier) with defaultCatalog / defaultSchema aligned with main's view API.
New ViewVersionInfoPO + ViewVersionInfoMapper with base + PostgreSQL SQL providers.
ViewPO / ViewMetaMapper providers extended to read/write audit_info, default_catalog, default_schema and join the current view_version_info row.
ViewMetaService rewritten for full CRUD with version management and JSON serialization of columns/properties/representations; soft-delete support.
JDBCBackend routes ViewEntity inserts through ViewMetaService; bumps CURRENT_SCRIPT_VERSION to 1.3.0.
ViewOperationDispatcher updated with a createView stub matching main's ViewCatalog#createView(..., String defaultCatalog, String defaultSchema, Map) signature; full CRUD dispatchers will land in a follow-up PR.

Why are the changes needed?

This is the storage portion of the Logical View Management EPIC (#604). The API and DDL are already on main; this PR wires the persistence layer so views can be stored in Gravitino's metadata store with versioned snapshots.

Fix: #10878

Does this PR introduce any user-facing change?

No new user-facing APIs in this PR. CURRENT_SCRIPT_VERSION is bumped from 1.2.0 to 1.3.0 to match the schema files already present on main; deployments will need to apply the upgrade-1.2.0-to-1.3.0-*.sql script (already in the repo).

How was this patch tested?

New unit tests in TestViewMetaService covering insert / load / update (with new version row) / soft delete / list-by-namespace / multi-representation round-trip.
Existing TestViewOperationDispatcher extended for the updated createView signature.
Verified locally: ./gradlew :core:test --tests "*TestViewMetaService*" --tests "*TestViewOperationDispatcher*" -PskipITs — all 16 tests pass.

github-actions · 2026-04-27T18:33:18Z

Code Coverage Report

Overall Project	65.48% `+0.17%`	🟢
Files changed	79.08%	🟢

Module	Coverage
aliyun	1.73%	🔴
api	47.13%	🟢
authorization-common	85.96%	🟢
aws	1.1%	🔴
azure	2.6%	🔴
catalog-common	10.2%	🔴
catalog-fileset	80.02%	🟢
catalog-glue	82.47%	🟢
catalog-hive	81.83%	🟢
catalog-jdbc-clickhouse	79.06%	🟢
catalog-jdbc-common	43.93%	🟢
catalog-jdbc-doris	80.28%	🟢
catalog-jdbc-hologres	54.03%	🟢
catalog-jdbc-mysql	79.23%	🟢
catalog-jdbc-oceanbase	78.38%	🟢
catalog-jdbc-postgresql	82.05%	🟢
catalog-jdbc-starrocks	78.27%	🟢
catalog-kafka	77.01%	🟢
catalog-lakehouse-generic	45.07%	🟢
catalog-lakehouse-hudi	79.1%	🟢
catalog-lakehouse-iceberg	86.98%	🟢
catalog-lakehouse-paimon	77.71%	🟢
catalog-model	77.72%	🟢
cli	44.51%	🟢
client-java	77.63%	🟢
common	48.75% `+0.08%`	🟢
core	81.5% `-0.12%`	🟢
filesystem-hadoop3	76.97%	🟢
flink	40.55%	🟢
flink-runtime	0.0%	🔴
gcp	14.2%	🔴
hadoop-common	10.39%	🔴
hive-metastore-common	46.83%	🟢
iceberg-common	55.24%	🟢
iceberg-rest-server	67.05% `+0.49%`	🟢
integration-test-common	0.0%	🔴
jobs	66.17%	🟢
lance-common	23.88%	🔴
lance-rest-server	57.84%	🟢
lineage	53.02%	🟢
optimizer	82.87%	🟢
optimizer-api	21.95%	🔴
server	85.46%	🟢
server-common	70.01%	🟢
spark	32.79%	🔴
spark-common	39.09%	🔴
trino-connector	34.28%	🔴

Files

Module	File	Coverage
common	SQLRepresentationDTO.java	90.0%	🟢
	RepresentationDTO.java	60.0%	🟢
	ConfigConstants.java	0.0%	🔴
core	DefaultMapperPackageProvider.java	100.0%	🟢
	ViewMetaPostgreSQLProvider.java	100.0%	🟢
	ViewPO.java	92.97%	🟢
	ViewOperationDispatcher.java	82.54%	🟢
	ViewVersionInfoPO.java	79.55%	🟢
	ViewEntity.java	76.67%	🟢
	ViewMetaBaseSQLProvider.java	76.47%	🟢
	ViewMetaService.java	76.04%	🟢
	JDBCBackend.java	74.93%	🟢
	ViewVersionInfoSQLProviderFactory.java	73.68%	🟢
	EntityCombinedView.java	70.0%	🟢
	ViewVersionInfoPostgreSQLProvider.java	57.14%	🔴
	ViewVersionInfoBaseSQLProvider.java	55.56%	🔴
	ViewMetaMapper.java	0.0%	🔴
	ViewVersionInfoMapper.java	0.0%	🔴
iceberg-rest-server	IcebergViewHookDispatcher.java	85.19%	🟢

Introduce the storage layer for the Logical View Management feature. Each view has a versioned history tracked in a new view_version_info table; view_meta is extended with audit_info and points to the current version. Snapshots include columns, properties, default catalog/schema and multi-dialect SQL representations serialized as JSON. Main changes: * PO/Mapper: ViewVersionInfoPO + mapper with base/PostgreSQL SQL providers; ViewPO and ViewMetaMapper providers extended with audit_info and default_catalog/default_schema. * Service: ViewMetaService rewritten for full CRUD with version management and JSON representation serialization. * meta: new ViewEntity implementing View and HasIdentifier. * JDBCBackend routes ViewEntity inserts through ViewMetaService; CURRENT_SCRIPT_VERSION bumped to 1.3.0. * Minimal compilation stubs added in ViewOperationDispatcher and test doubles for the expanded ViewCatalog interface (full implementation lands in Phase 3). The relational scripts (apache#10860) and view API surface (apache#10830) are already on main; this change builds on them. Unit tests cover insert, load, update (with new version row), soft delete, list-by-namespace and multi-representation round-trips. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a relational persistence layer for Logical Views with versioned snapshots, integrating it into the JDBC backend and updating catalog/view dispatch/testing to use the new ViewEntity + version tables.

Changes:

Introduces ViewEntity and view_version_info persistence (PO/Mapper/SQL providers) and joins current version when reading views.
Refactors ViewMetaService to CRUD views with version creation, JSON (de)serialization, and soft-delete cleanup for both meta and versions.
Updates dispatcher/backend/tests to store/import ViewEntity and align createView test signatures with main API.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
core/src/main/java/org/apache/gravitino/meta/ViewEntity.java	Adds a concrete view metadata entity implementing `View` for storage and import.
core/src/main/java/org/apache/gravitino/storage/relational/service/ViewMetaService.java	Reworks view persistence to read/write version rows and handle deletes across meta + versions.
core/src/main/java/org/apache/gravitino/storage/relational/po/ViewPO.java	Adds JSON (de)serialization and conversion helpers between PO and `ViewEntity`.
core/src/main/java/org/apache/gravitino/storage/relational/po/ViewVersionInfoPO.java	Introduces PO for version snapshot rows with required-field validation.
core/src/main/java/org/apache/gravitino/storage/relational/mapper/ViewVersionInfoMapper.java	Adds MyBatis mapper for view version info table operations.
core/src/main/java/org/apache/gravitino/storage/relational/mapper/ViewVersionInfoSQLProviderFactory.java	Adds provider factory for dialect-specific SQL generation for version info.
core/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/base/ViewVersionInfoBaseSQLProvider.java	Adds base SQL provider implementations for version info (MySQL/H2 style).
core/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/postgresql/ViewVersionInfoPostgreSQLProvider.java	Adds PostgreSQL-specific SQL for upserts and limited deletes for version info.
core/src/main/java/org/apache/gravitino/storage/relational/mapper/ViewMetaMapper.java	Updates result mapping to include joined current version info data.
core/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/base/ViewMetaBaseSQLProvider.java	Updates view meta queries to join current version info and include audit fields.
core/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/postgresql/ViewMetaPostgreSQLProvider.java	PostgreSQL overrides for joined view+version selects and audit fields.
core/src/main/java/org/apache/gravitino/storage/relational/mapper/provider/DefaultMapperPackageProvider.java	Registers the new `ViewVersionInfoMapper`.
core/src/main/java/org/apache/gravitino/storage/relational/JDBCBackend.java	Routes `ViewEntity` inserts through `ViewMetaService`.
core/src/main/java/org/apache/gravitino/catalog/ViewOperationDispatcher.java	Imports catalog views into entity store as `ViewEntity` including full view definition.
core/src/main/java/org/apache/gravitino/catalog/EntityCombinedView.java	Switches combined view wrapper to use `ViewEntity` instead of `GenericEntity`.
core/src/test/java/org/apache/gravitino/storage/relational/service/TestViewMetaService.java	Adds/updates unit tests for CRUD + versioning + soft-delete lifecycle.
core/src/test/java/org/apache/gravitino/catalog/TestViewOperationDispatcher.java	Updates tests to use `ViewEntity` and include a non-empty representation.
core/src/test/java/org/apache/gravitino/connector/TestCatalogOperations.java	Updates test connector to satisfy new `ViewCatalog` methods.
common/src/main/java/org/apache/gravitino/config/ConfigConstants.java	Bumps `CURRENT_SCRIPT_VERSION` to `1.3.0`.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replay the local-vs-remote diff as one standalone commit on top of the current PR head so it can be pushed without force. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace GenericEntity(VIEW) write/update paths with ViewEntity, and remove GenericEntity fallback insertion for VIEW in relational backend. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge origin/main into the PR branch and resolve conflict blocks while keeping the ViewEntity-based write path changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jerryshao · 2026-04-29T06:04:14Z

    }
  }
+
+  private ViewEntity buildViewEntityForImport(Long uid, NameIdentifier ident, View view) {


What's the meaning of buildViewEntityForImport?

buildViewEntityForImport is the materialization step in the loadView auto-import path: it converts a catalog View into a persisted ViewEntity with Gravitino-generated id/identifier.

It is similar in purpose to table import, but currently lighter than importTable (no string-identifier reconciliation/rename correction path, and import here is best-effort).

Given ViewOperationDispatcher currently only supports loadView + auto-import, we expect to clean up/refactor this helper when full view CRUD lands in follow-up PRs.

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

jerryshao · 2026-04-29T11:19:16Z

      LOG.info("Successfully imported view {} into entity store with id {}", ident, uid);
-      return EntityCombinedView.of(entityCombinedView.viewFromCatalog(), newViewEntity)
-          .withImported(true);
+      return EntityCombinedView.of(catalogView, newViewEntity).withImported(true);


If we import the view from external source to Gravitino, shall we still need to combine them, I think they're the same, am I right?

Different catalogs support views to varying degrees, so combining them with viewEntity is necessary. Specific handling will be added in subsequent PRs.

jerryshao · 2026-04-29T11:24:47Z

-@Getter
-@EqualsAndHashCode
-@ToString
 public class ViewPO {


Why don't you use lombok to simplify the code?

Handle zero-row optimistic update in ViewMetaService.updateView by failing the transaction and surfacing IOException, preventing orphaned view_version_info rows. Also add a regression test that deletes the view during updater execution to verify no extra version row is persisted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Replace manual getters/equals/hashCode/builder in ViewPO with Lombok while preserving equality semantics by excluding viewVersionInfoPO. Update ViewMetaService to use Lombok-generated ViewPOBuilder and add TestViewPO to verify builder behavior and equals/hashCode compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jerryshao · 2026-04-30T03:23:42Z

+      m.put("type", Representation.TYPE_SQL);
+      m.put("dialect", sql.dialect());
+      m.put("sql", sql.sql());
+      return m.build();


How does this Representation deserialize in DTO tranferring? I think we should align the json serde with DTO. Here the hard-coded toMap is very hard to maintain.

Or, you can figure out a more explicit way to do this.

jerryshao · 2026-04-30T03:26:35Z

+                SessionUtils.getWithoutCommit(
+                    ViewMetaMapper.class, mapper -> mapper.updateViewMeta(newViewPO, oldViewPO)));
+            if (updateResult.get() == 0) {
+              throw new RuntimeException("Failed to update the entity: " + ident);


Shall we throw a NoSuchEntityException here?

updateViewMeta(...) returning 0 here does not necessarily mean the view is missing; it can also be an optimistic-concurrency conflict (current_version mismatch).

We intentionally throw a runtime exception inside the transaction to force rollback, then map it to IOException for consistency with other update paths (e.g., TableMetaService and FilesetMetaService).

Align view representation serialization with FunctionPO style by introducing typed Representation DTOs instead of hard-coded map conversion in ViewPO. Add DTO serde tests in common and round-trip compatibility tests in core, including legacy json without explicit type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add constructor Javadoc for RepresentationDTO to satisfy common:javadoc with -Werror. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mchades force-pushed the view-relational-storage branch from 00086c5 to 894ad76 Compare April 27, 2026 16:29

mchades force-pushed the view-relational-storage branch from 894ad76 to b91803d Compare April 28, 2026 04:08

mchades force-pushed the view-relational-storage branch from b91803d to 6b6207d Compare April 28, 2026 04:16

mchades marked this pull request as ready for review April 28, 2026 05:07

mchades requested a review from Copilot April 28, 2026 05:07

Copilot AI reviewed Apr 28, 2026

View reviewed changes

Copilot started reviewing on behalf of mchades April 28, 2026 05:32 View session

mchades and others added 2 commits April 28, 2026 13:34

fix(iceberg): use ViewEntity for view rename hook

b0979ed

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix(core): remove GenericEntity view insert fallback

2d6ce81

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mchades requested a review from jerryshao April 28, 2026 06:37

mchades and others added 3 commits April 29, 2026 11:55

fix(core): replay local updates for PR apache#10879

2cc7e14

Replay the local-vs-remote diff as one standalone commit on top of the current PR head so it can be pushed without force. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix(core): use ViewEntity for view persistence paths

321aec4

Replace GenericEntity(VIEW) write/update paths with ViewEntity, and remove GenericEntity fallback insertion for VIEW in relational backend. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

merge: resolve conflicts with latest main for PR apache#10879

6bcde2c

Merge origin/main into the PR branch and resolve conflict blocks while keeping the ViewEntity-based write path changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jerryshao reviewed Apr 29, 2026

View reviewed changes

jerryshao requested a review from Copilot April 29, 2026 06:12

Copilot started reviewing on behalf of jerryshao April 29, 2026 06:13 View session

Copilot AI reviewed Apr 29, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/gravitino/storage/relational/service/ViewMetaService.java Outdated

jerryshao reviewed Apr 29, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/gravitino/storage/relational/po/ViewPO.java Outdated

mchades and others added 2 commits April 30, 2026 07:44

mchades requested review from jerryshao and yuqi1129 April 29, 2026 23:55

jerryshao reviewed Apr 30, 2026

View reviewed changes

mchades and others added 2 commits April 30, 2026 12:09

docs(common): add missing javadoc for representation dto ctor

7cfb2ab

Add constructor Javadoc for RepresentationDTO to satisfy common:javadoc with -Werror. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jerryshao approved these changes Apr 30, 2026

View reviewed changes

jerryshao merged commit 2c0711a into apache:main Apr 30, 2026
29 checks passed

jerryshao assigned mchades Apr 30, 2026

mchades deleted the view-relational-storage branch April 30, 2026 06:09

Conversation

mchades commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

github-actions Bot commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mchades commented Apr 27, 2026 •

edited

Loading

github-actions Bot commented Apr 27, 2026 •

edited

Loading