Proposal: Adopt OpenAPI as the External Contract for Gravitino API V2

[nevzheng]

Two birds with one stone

Abstract

I recommend adopting OpenAPI as the authoritative external contract for Gravitino API V2 and integrating that contract into the server’s development, testing, and release lifecycle.

This addresses two related problems:

• Our error-handling strategy, where caller errors, dependency failures, and Gravitino defects frequently collapse into generic HTTP 500 responses.
• Our OpenAPI contract and validation gap, where the existing description is not authoritative or reliable enough for code generation and compatibility guarantees.

A specification-first V2 would give us a stable external contract, enable non-Java integrations and generated clients, improve error semantics, and support conformance testing, property-based testing, API fuzzing, and automated regression detection.

Context

Hi everyone,

Last week I spent considerable time investigating our error-handling strategy and OpenAPI description. I initially approached them as separate problems, but I now believe we can solve them through the same initiative.

Gravitino already has useful OpenAPI and Java DTO work to build upon. However, we do not yet treat OpenAPI as the authoritative external contract or continuously verify that the running server conforms to it.

My recommended approach is:

Use OpenAPI as the authoritative external contract for Gravitino API V2 and integrate it directly into how we design, implement, test, and release the server.

A specification-first V2

V2 should be designed specification-first:

1. Define and review the external behavior in OpenAPI.
2. Implement or modify the server to conform.
3. Generate clients from the specification.
4. Validate the running server against the contract.
5. Detect regressions and breaking changes continuously.

By “V2,” I mean the next trustworthy and reliable version of the Gravitino external API—not OpenAPI Specification 2.0. The V2 contract should use an appropriate current OpenAPI 3.x version and follow established API design practices.

V2 should be a clean external contract. This gives us room to improve:

• Resource and operation design.
• Request and response representations.
• HTTP status semantics.
• Authentication and authorization behavior.
• Pagination and filtering.
• Idempotency and retry behavior.
• Public error responses.
• Naming and compatibility conventions.

V1 and V2 can coexist while V2 is developed. We can publish the implemented V2 surface as beta, allowing users to generate clients, build integrations, and give us feedback before the API is declared stable.

OpenAPI should define the external boundary

Our Java DTOs remain useful internal and Java-client models. I am not proposing that we remove them.

I recommend that we stop treating Java DTO shapes as the definition of the external wire protocol.

The OpenAPI contract should independently define:

• Resources and operations.
• Wire representations.
• Required and optional fields.
• Validation constraints and nullability.
• HTTP statuses, media types, and headers.
• Authentication requirements.
• Public error semantics.

The server can use a translation layer between the OpenAPI-defined external representations and its internal DTO or domain models.

This allows internal Java code to evolve without automatically changing the public protocol. It also prevents Java-specific types, exception classes, inheritance, and serialization choices from leaking into integrations written in other languages.

Java DTOs may implement the API, but they should not define it.

Error handling as part of the contract

Error handling is observable API behavior, not merely an implementation detail.

Clients need to distinguish:

• Invalid requests.
• Authentication and authorization failures.
• Missing or conflicting resources.
• Unavailable or failing dependencies.
• Timeouts and retryable conditions.
• Genuine Gravitino defects.

V2 should define stable HTTP semantics and a consistent public error model. That model should provide:

• Stable, machine-readable error reasons.
• Correct HTTP statuses.
• Failure-domain attribution.
• Explicit retry semantics.
• Safe, actionable public messages.
• Request identifiers for correlation with internal diagnostics.

Stack traces and internal causes should remain in server logs rather than becoming part of the public protocol.

The error taxonomy defines what failures mean. OpenAPI makes those semantics public and machine-verifiable. Server-side classification preserves the meaning from the catalog adapter to the HTTP response, and conformance testing verifies that the implementation follows the contract.

OpenAPI will therefore become an important pillar of our effort to eliminate undifferentiated HTTP 500 responses.

Why this matters

Adoption and interoperability

Much of the modern data ecosystem is not written in Java. Apache DataFusion, for example, is written in Rust. Other consumers use Python, Go, TypeScript, C++, and languages we may not support directly.

These users cannot easily reuse our Java DTOs. Without a reliable language-neutral contract, every integration must reverse-engineer payloads, maintain handwritten models, or infer behavior from server implementation details.

A code-generation-clean OpenAPI contract allows users to:

• Generate typed clients in their preferred language.
• Use standard mock servers and validators.
• Build integrations without studying Java source code.
• Validate compatibility before deployment.
• Depend on documented error and retry behavior.

This reduces integration friction and expands Gravitino’s potential ecosystem beyond the JVM.

For initial validation, I propose generating Java, Rust, and Python clients. The precise long-term client-support strategy can evolve as we learn from real usage.

Trust through verifiable behavior

A specification alone is a promise. A specification backed by automated conformance, generated-client checks, compatibility analysis, and regression testing provides evidence that the implementation honors that promise.

Users and downstream projects should be able to verify that:

• Requests and responses have stable semantics.
• The running server behaves as documented.
• Failures are predictable and actionable.
• Compatibility is checked before release.
• Regressions are detected before deployment.

This establishes confidence through measurable behavior rather than documentation alone.

Reliability, security, and maintainability

A stable contract enables:

• Safe retry and backoff policies.
• Circuit breakers and correct alert routing.
• Useful operational runbooks.
• Better attribution between Gravitino and its dependencies.
• Consistent authentication and authorization tests.
• Automated checks against internal information leakage.
• Safer API evolution.
• Faster diagnosis of integration and production failures.

It also gives contributors a clear target. Server and client development can proceed against an agreed contract, while CI detects drift before it reaches users.

An executable contract

The V2 OpenAPI specification should be part of our testing and release infrastructure, not only a source for rendered documentation.

A practical pipeline can include:

1. Lint and bundle the specification.
2. Generate and compile representative clients.
3. Run the server and validate its responses against the contract.
4. Execute representative end-to-end customer journeys.
5. Run property-based and schema-guided fuzz tests.
6. Detect and review breaking changes.

End-to-end tests cover known workflows, but they can exercise only scenarios we write manually. Tools such as Schemathesis can generate valid, invalid, boundary, and adversarial requests directly from OpenAPI.

This lets us test properties such as:

• Malformed input does not produce an unexpected 500.
• Responses match their documented schemas.
• Only documented statuses and media types are returned.
• Boundary values behave consistently.
• Authentication failures do not expose internal details.
• Equivalent failures produce consistent error semantics.

CI can begin in advisory mode while we expose and resolve existing issues. As the specification and server reach conformance, we can progressively make the checks blocking.

Incremental adoption

This should be implemented incrementally, use case by use case.

I propose beginning with catalog and table workflows because they represent central Gravitino behavior and exercise meaningful success, validation, dependency, and error cases.

A practical sequence is:

1. Establish the V2 OpenAPI structure and design rules.
2. Define the shared success and error representations.
3. Specify the catalog and table use cases.
4. Implement the external-to-internal translation layer.
5. Generate initial Java, Rust, and Python clients.
6. Add conformance, property, and customer-journey tests.
7. Publish the implemented surface as V2 beta.
8. Gather feedback and expand V2 incrementally.
9. Promote CI checks from advisory to blocking as V2 matures.
10. Declare V2 stable when its supported surface consistently passes the contract gate.

This approach lets users engage early while allowing the API to grow through concrete, tested use cases.

Future proposal: Antithesis

One future direction I am particularly excited about is integrating Antithesis into the Gravitino testing stack.

A formal V2 contract would give us explicit properties and invariants to test under dependency failures, network faults, concurrency, retries, and unusual operation sequences. Antithesis could exercise the complete system in a deterministic simulation environment and make discovered failures reproducible.

This is not required to begin V2. I intend to develop it as a separate future proposal once the contract and initial conformance infrastructure are established.

Feedback requested from maintainers

I would appreciate feedback on this recommended direction:

1. Adopt OpenAPI as the authoritative external contract for V2.
2. Design V2 specification-first as a clean external API.
3. Keep Java DTOs internally, with translation at the external boundary.
4. Make the public error contract part of the initial V2 work.
5. Begin incrementally with catalog and table use cases.
6. Publish the implemented V2 surface as beta for early feedback.
7. Introduce CI checks progressively, moving from advisory to blocking.
8. Use generated Java, Rust, and Python clients as initial contract validation.

If there are viable alternatives to OpenAPI that better serve these goals, I would like to understand and compare them.

References

• OpenAPI Specification
• Zalando RESTful API Guidelines
• Microsoft Azure REST API Guidelines
• Google AIP-193: Errors
• Google AIP-194: Automatic retries
• RFC 9110: HTTP Semantics
• Schemathesis
• OWASP API Security Top 10

[mchades]

Thanks for putting together this thoughtful proposal. As one of the original contributors and maintainers of Gravitino’s OpenAPI module, I agree with the problem statement and support the overall direction of making OpenAPI the authoritative external contract for API V2.

The current OpenAPI module was initially introduced mainly to document and manage REST API definitions. It was not designed as an executable source of truth or as the basis for server/client generation. Evolving it into a stronger contract is reasonable as Gravitino’s API surface and non-Java ecosystem grow.

That said, I do not think the proposal is implementation-ready yet. It currently combines three related but distinct efforts:

• Improving the quality and CI validation of the existing OpenAPI specification.
• Fixing V1 error classification and HTTP semantics.
• Designing a new contract-first V2 architecture.

These efforts should be coordinated, but they should not block one another. In particular, important V1 error-handling improvements should not wait for V2.

I also suggest keeping the model strategy open. Making OpenAPI authoritative does not necessarily require generating all request/response models or introducing a completely separate model hierarchy. For example, Apache Polaris generates JAX-RS service interfaces from OpenAPI while mapping schema types to Iceberg’s maintained REST models. Other approaches are also viable. We should validate the trade-offs before committing to one architecture.

My recommendation is to start with a narrow Catalog/Table vertical slice and use it to answer the remaining design questions:

• V1/V2 coexistence and version negotiation.
• Generated interfaces versus generated or hand-written wire models.
• The boundary between external models and existing DTO/domain models.
• A stable public error model, including reason, domain, retryability, and request ID.
• Code-generation checks for Java, Rust, and Python, without initially committing to officially supporting all generated SDKs.
• Runtime conformance, customer-journey tests, and breaking-change detection.
• Ownership and release lifecycle of the authoritative specification.

Therefore, my position is +1 on the direction and on starting a focused design/PoC, but I would not yet treat the full V2 migration or translation-layer architecture as decided. A small end-to-end prototype should give us the evidence needed to make those decisions confidently.

[nevzheng]

Clarification: Mark correctly pointed out that Gravitino already publishes rendered OpenAPI documentation:

https://gravitino.apache.org/docs/latest/api/rest/gravitino-rest-api/

My point was more specific: the long-term North Star is to expose a stable, versioned, machine-readable OpenAPI document that clients and tooling can consume directly—for example:

• /api/v1/openapi.json
• /api/v2/openapi.json

This complements the existing documentation rather than replacing it.

[jerryshao]

I want to clarify something:

1. Our current Gravitino REST API doesn't have a versioning prefix like api/v1 to define the API version. Instead, we use VersionFilter, which is a version header, to support API versioning. So we can support versioning per API. We need to honor this when we bump the API version.
2. I would still weigh in the values of bumping to V2 version. Do we have multiple strong reasons to upgrade our current API definition? I would like to know more and have more discussions.

[nevzheng]

Hi all — I wrote up a proposal for making OpenAPI the authoritative external contract for Gravitino API V1. I’d appreciate discussion and iteration on the approach.

I’ll try to get an end-to-end PoC implementation going on my fork.

Proposal: Gravitino API V1, governed by OpenAPI