[DISCUSS] Is a Paimon REST Catalog Server still necessary when Gravitino is used as the unified access layer?

[TKilome]

I am evaluating the recommended architecture for integrating Apache Paimon with Gravitino.

If users access Paimon through Gravitino connectors, the request path is:

Flink / Spark
  -> Gravitino connector
  -> Gravitino Server
  -> Gravitino Paimon catalog
  -> Paimon backend

In this model, Gravitino can provide a unified metadata entry point, authentication, authorization, audit, and catalog governance.

My question is: in this architecture, is it still necessary to implement or deploy a separate Paimon REST Catalog Server?

More specifically:

1. If Gravitino already exposes Paimon metadata through its own REST API and engine connectors, what additional value does a Paimon REST Catalog Server provide?

2. Should the recommended deployment be:

   Flink / Spark -> Gravitino -> Paimon filesystem / Hive / JDBC backend

   instead of:

   Flink / Spark -> Paimon REST Catalog Server

3. Is the Paimon REST Catalog Server mainly useful for clients that do not integrate with Gravitino directly, or for compatibility with the Paimon REST catalog protocol?

4. If a Paimon REST Catalog Server is still useful behind Gravitino, should it be treated only as an internal backend service, with users accessing Paimon only through Gravitino?

5. From the Gravitino community's perspective, is there any plan or recommendation around implementing a Gravitino-backed Paimon REST Catalog Server?

Thanks.

[yuqi1129]

Thanks for the thoughtful questions. I think the Paimon REST Catalog Server and Gravitino address different layers of the architecture.
One clarification first: the request path shown above is conceptually correct for catalog and metadata operations, but not every Paimon operation or data I/O request necessarily passes through the Gravitino server. The current Gravitino Flink and Spark connectors route catalog/DDL operations through Gravitino, while native Paimon catalog and table implementations are still used for format-specific behavior and data I/O.

1. What additional value does a Paimon REST Catalog Server provide?

Its primary value is protocol compatibility. It exposes the standard Paimon REST API, allowing Paimon-native clients to connect without depending on a Gravitino-specific connector or API.

It may also expose Paimon-specific capabilities that do not fit naturally into Gravitino's unified metadata model, such as format-specific metadata, snapshot, branch, partition, and commit operations. The Paimon REST specification also allows such technology-specific logic to remain on the server side, while clients remain relatively lightweight and decoupled from the backend implementation.
Therefore, it is an additional access protocol rather than a replacement for Gravitino.

2. Which deployment model should be recommended?

There is no single deployment model for every use case. If all engines can use the Gravitino connectors and the supported Gravitino metadata model is sufficient, the simpler deployment is:

  Flink / Spark
       -> Gravitino connector
       -> Gravitino
       -> Paimon filesystem / Hive / JDBC backend

In that case, a separate Paimon REST server is not required. If users need standard Paimon REST compatibility, another valid deployment is:

  Flink / Spark / other Paimon REST clients
       -> Paimon REST endpoint integrated with Gravitino
       -> Paimon backend

In the second model, Gravitino can remain the catalog and governance control plane, while the Paimon REST endpoint provides the format-specific protocol.

3. Is the Paimon REST server mainly useful for clients that do not integrate with Gravitino directly?

Yes, that is probably its clearest use case. It allows clients that already support the Paimon REST protocol to access Paimon catalogs without adopting a Gravitino-specific connector.

Other benefits include language-independent access, decoupling client and server versions, keeping Paimon-specific logic on the server side, and potentially applying Gravitino-managed authorization, auditing, and credential vending to standard Paimon clients.

4. Should a Paimon REST server behind Gravitino be treated only as an internal backend?

Not necessarily. We should distinguish two directions:

• Existing rest backend: Gravitino -> external Paimon REST server

• Proposed Gravitino-backed endpoint: Paimon REST client -> Paimon REST endpoint integrated with Gravitino

In the first case, the external Paimon REST server can be treated as an internal backend. If Gravitino is intended to be the mandatory governance entry point, clients should not bypass it unless the external server enforces equivalent authorization and auditing policies.

In the second case, the Paimon REST endpoint is intentionally a public, standard-compatible entry point. It should authenticate Paimon REST clients and integrate their identities and operations with Gravitino's catalog management, authorization, and auditing capabilities. Thus, the endpoint can be parallel to the Gravitino REST API from the client's perspective, while still being governed by Gravitino internally.

5. Is there a community plan to implement a Gravitino-backed Paimon REST server?

As far as the current code and discussions show, there is no accepted implementation plan or roadmap yet. There is an open feature discussion in #7663 (#7663), but the community has also raised concerns about the maturity of the Paimon REST specification, actual user demand, and the long-term cost of maintaining another format-specific REST implementation. Before implementation, I think we should first agree on:

• The concrete clients and use cases that cannot be covered by the existing Gravitino connectors.
• The minimum Paimon REST API scope that should be supported.
• Whether the service is standalone, an auxiliary Gravitino service, or both.
• How authentication identities and Paimon operations map to Gravitino authorization and audit events.
• Which operations go through Gravitino's unified metadata APIs and which must operate directly on the Paimon backend.
• The expected compatibility and maintenance policy as the Paimon REST specification evolves.

In summary, a Paimon REST server is not a mandatory extra hop when using Gravitino. It is an optional, format-specific protocol endpoint whose main value is enabling standard Paimon REST clients to benefit from Gravitino-managed catalogs and governance.

[TKilome]

Thanks @yuqi1129 for the detailed explanation. This clarifies the positioning for me: Gravitino can be the unified catalog/governance entry point, while a Paimon REST endpoint is
optional and mainly valuable for standard Paimon REST clients or format-specific operations. The deployment distinction between an internal REST backend and a Gravitino-backed
public REST endpoint is especially helpful.