Error taxonomy: we collapse every failure into HTTP 500 — a gateway shouldn't

[nevzheng]

The problem

Gravitino is a gateway that federates external catalogs (Iceberg REST, Hive Metastore, JDBC, Kafka, filesets), but it handles failures with an overly-broad catch discipline. Across the catalog layer, the recurring shape is:

} catch (Exception e) {
  throw new RuntimeException(e);
}

The server then renders that as HTTP 500 Internal Server Error. The consequence: three fundamentally different kinds of failure become indistinguishable to callers and to us in production:

1. A dependency is down — Hive Metastore unreachable, Iceberg REST returning 503, a JDBC connection drop → not our fault, often retriable.
2. The caller made a mistake — bad argument, not-found, already-exists, unauthorized → the caller's fault, never retriable.
3. Gravitino itself has a bug — a broken invariant → our fault, must be investigated.

Today all three look like the same generic 500. This is a recognized anti-pattern (CWE-396: Declaration of Catch for Generic Exception — broad catches "obscure the source of the exception and make debugging more difficult"), not just a style preference.

The data

throw new RuntimeException(...) in catalog code (catalogs/*/src/main, branch main):

Module	sites		Module	sites
catalog-hive	24		catalog-lakehouse-iceberg	5
catalog-fileset	19		catalog-lakehouse-paimon	5
catalog-kafka	15		catalog-glue	5
hive-metastore-common	15		catalog-lakehouse-hudi	4
catalog-model	14		catalog-common	3
catalog-lakehouse-generic	9		catalog-jdbc-doris	2
hadoop-common	5		catalog-jdbc-common / starrocks	1 / 1

~127 sites in catalog code (full file:line list attached). Not every site swallows a backend exception — some are genuine internal invariants — but the ones catching Exception/IOException/SQLException/TException around a backend call are where dependency-vs-us and retriable-vs-terminal information is erased. Concretely today:

• Only JDBC and Hive have real exception converters. Iceberg (operational paths), Paimon, Kafka, Fileset, Model have none and collapse everything — including backend-unavailable — into RuntimeException.
• On the server (server/.../web/rest/ExceptionHandlers.java), ConnectionFailedException → 502 is mapped by exactly one of 22 per-entity handlers (CatalogExceptionHandler); everywhere else it falls through to 500. UnauthorizedException (error code 1011) is emitted by no handler. There is no ExceptionMapper<Throwable> safety net.
• The Java client has no retry, no backoff, no Retry-After awareness; INTERNAL_ERROR becomes a bare RuntimeException.

Why this matters — and why it grows over time

Operationally / for deployments. We are a gateway; per RFC 9110 §15.6 an upstream outage is 502/503/504, not 500. Returning 500 for a dependency being down is a spec misuse that routes the wrong on-call to the wrong system. A 500 should mean "a Gravitino invariant broke" and should be rare — when it's the catch-all, real bugs hide inside a flood of dependency errors and every incident starts from zero.

This is not hypothetical — it is one of the most expensive failure modes at scale. At BigQuery, 500-class errors are the single hardest category to debug: the team has spent the equivalent of years of engineering time quashing them, is still working through the backlog, and continues to burn significant on-call time chasing them. Undifferentiated 500s are a debt that compounds — every generic 500 Gravitino emits today is a deferred instance of that same cost, far cheaper to prevent at the source than to excavate later from logs.

For users / clients. Every backend we federate already ships a structural retriable signal — Kafka RetriableException, JDBC SQLTransientException/SQLRecoverableException, Iceberg ServiceUnavailableException/503, Hive TTransportException — and we discard all of it at the adapter boundary. Clients can't safely auto-retry because they can't tell transient from permanent. (AIP-194: "retrying a request with an invalid argument will never succeed — only transient failures merit automatic retry.")

For enterprise adoption and complex deployments (the commercial argument). Good error taxonomy is not cosmetic — it's what makes Gravitino operable inside someone else's production, which is exactly the bar for enterprise:

• Integration into a control plane. Enterprise platform/SRE teams wire retries, circuit breakers, autoscaling, alerting, and runbooks to status/error codes. If everything is 500, none of that automation can distinguish "retry / back off" from "page a human." A gateway that can't say whose fault a failure is creates integration friction that stalls POCs.
• SLAs / error budgets. Enterprise contracts define availability targets. You cannot honor — or even measure — an SLA if upstream-catalog outages are counted as Gravitino 500s. A clean taxonomy lets us attribute failures (ours vs dependency) and defend an SLA, which is a prerequisite for enterprise commitments.
• MTTR & support cost. Typed, attributed errors cut mean-time-to-resolution and support-ticket volume — the actual cost centers in enterprise support agreements.
• Security / compliance review. Generic 500s that leak stack traces and internal paths are flagged in security reviews (CWE-209: information exposure through an error message). Sanitized, typed errors pass.
• Trust / accountability boundary. As a federation layer between callers and many backends, being able to say "your Hive Metastore is down, not us" is an accountability boundary ops teams demand before they'll put Gravitino on the critical path.

In one line: clean error semantics convert Gravitino from "a black box that returns 500" into "a well-behaved gateway component" — a procurement and integration checkbox, not a nice-to-have.

A possible direction (for discussion, not a decision)

The standard remedy is a small closed taxonomy where the category itself tells the caller who's at fault and whether to retry — the model used by gRPC status codes, Google AIP-193, and HTTP. Mapped to our gateway role, roughly four buckets + a retriable bit:

Bucket	HTTP	Retriable	gRPC analog
Caller error (bad-request / not-found / already-exists / unsupported)	400 / 404 / 409 / 405	no	INVALID_ARGUMENT, NOT_FOUND, ALREADY_EXISTS, FAILED_PRECONDITION
Auth error (authn / authz)	401 / 403	no	UNAUTHENTICATED, PERMISSION_DENIED
Dependency unavailable (upstream down / transient / timeout)	503 (+Retry-After) / 504	yes	UNAVAILABLE
Dependency failure (upstream returned a broken response)	502	no	(upstream INTERNAL)
Internal error (a Gravitino invariant broke — only our bugs)	500	no	INTERNAL

Guiding principle: classify-then-preserve rather than swallow-then-whitelist — classify a failure once, at the adapter boundary where the native exception still carries meaning, and preserve that classification (incl. the retriable bit) all the way to the HTTP response and the client.

Two dimensions the bucket alone doesn't capture

Which bucket is only one axis. An error carries two more independent facets — both straight from the same standard playbook (Google's google.rpc.Status / ErrorInfo, Abseil Status):

• Error domain — which subsystem produced this? gravitino-core vs iceberg-rest vs hive-metastore vs jdbc:<vendor>. A machine-readable domain (AIP-193's ErrorInfo.domain + reason) lets callers and our own tooling attribute and route failures by blast radius instead of grepping message strings. For a gateway this is arguably the central concept — you cannot reason about "whose fault" without naming the domain. (Cf. "Ow! My Foot!" — "think in error domains and blast radius"; "error domains are everything.")
• Public vs internal — who is the audience? Every error has two: a public face (safe, actionable, stable — what the caller sees) and internal detail (full cause chain, stack, possibly sensitive config — logs/traces only). A generic 500 does the opposite of both at once: it leaks internals to the caller (CWE-209) and strips the useful detail from the response. These must be explicitly separated so we can be verbose internally and disciplined externally.

Remediation plan (proposed, incremental)

Not a big-bang rewrite — each phase is independently shippable and ordered by value/cost. The goal is correct classification, not "zero RuntimeExceptions": genuinely-internal failures (invariants, our own EntityStore) should stay 500; we're separating those from dependency failures.

• Phase 0 — Stop the bleeding (server-side, small, no API change):
  • Hoist ConnectionFailedException → 502, UnauthorizedException → 401, ForbiddenException → 403 into BaseExceptionHandler so all 22 per-entity handlers inherit consistent mapping (today only CatalogExceptionHandler maps 502).
  • Add an ExceptionMapper<Throwable> safety net so nothing escapes as an unparseable container 500.
  • Highest leverage for the least code — fixes blast radius for every entity at once.
• Phase 1 — Define the taxonomy (api / core):
  • Introduce a classified-exception model: a marker exposing failure() (caller / dependency-unavailable / dependency-failure / internal), retriable(), and domain.
  • Add the missing HTTP + error codes: 502, 503 (+ Retry-After), 504; wire Utils / ErrorConstants / ErrorResponse.
  • Establish the public vs internal split — sanitized message to the caller, full cause/stack to logs.
• Phase 2 — Classify at the adapter boundary (per catalog, incremental):
  • Reference implementation first: Iceberg REST converter — translate 503 / ServiceFailure / transport-IOException, and encode the traps (CommitStateUnknownException → never retry; CommitFailedException → retry whole commit).
  • Then the converter-less catalogs: Kafka (read RetriableException), Fileset/HDFS, Paimon.
  • Upgrade JDBC / Hive converters from string-matching ("Access denied", simple-name) to marker-based (SQLTransient* / SQLState class / TTransportException).
  • Attach an error domain at each adapter (iceberg-rest, hive-metastore, jdbc:<vendor>, gravitino-core).
• Phase 3 — Client + guardrails:
  • Java client: Retry-After + bounded backoff for idempotent operations; reverse-map the new codes in the base error handler.
  • Establish review / coding guidelines for error handling, optionally backed by a lightweight lint/Checkstyle rule discouraging bare catch (Exception) → RuntimeException in adapters.

Open questions / other categories to work through

• Authorization vs authentication (401 vs 403). Today 401 is effectively dead taxonomy and backend auth failures (Kafka AuthorizationException, JDBC SQLState 28, Iceberg 401/403) collapse to 500. How do we surface upstream authz failures vs Gravitino's own access control?
• Rate limiting / backpressure (429 + Retry-After). Exists only for the lineage sink today — should the gateway propagate upstream throttling?
• Timeouts (504 vs 503). Distinguish "upstream slow" from "upstream down"?
• Non-obvious retriability traps the converters must encode: Iceberg CommitStateUnknownException → never auto-retry (possible double-write); CommitFailedException → retry the whole commit, not the request; Kafka UnknownTopicOrPartitionException is marked retriable but is semantically 404; JDBC — classify on the marker hierarchy before SQLState (class 08 is ambiguous).
• Error-message hygiene — sanitize what crosses the boundary to callers (CWE-209) while keeping full detail in logs.
• Error-code stability — once clients depend on codes they become API surface; how do we version them (per AIP-193's backwards-compat guidance)?
• Scope / sequencing. Is a cheap first win worthwhile (hoist ConnectionFailed/Unauthorized/Forbidden into BaseExceptionHandler + add an ExceptionMapper<Throwable> safety net) before the larger taxonomy + per-catalog converters + client retry?

References

The anti-pattern: CWE-396 / CWE-397 (generic catch/throw) · CWE-209 (error-message info exposure) · Oracle Java Tutorials — Exceptions · Effective Java Item 77 · Nygard, Release It!

Error domains, public/internal split, and cross-language design: "Ow! My Foot!" — a survey of error handling across C, Go, Rust, and Abseil (nevzheng, 2026) — "think in error domains and blast radius"; "the silent swallow and the invisible rethrow"; "error domains are everything."

The taxonomy to adopt: gRPC status codes · AIP-193 Errors / AIP-194 Retry · RFC 9110 §15.6

Related: #11943, #11959 (the Iceberg REST warehouse fail-fast fix that surfaced this).

Data appendix — full file:line list of the ~127 sites (click to expand)

Generated from grep -rn 'throw new RuntimeException' catalogs/*/src/main --include='*.java' on branch claude/dazzling-knuth-703829 @ 799c50c.
Count = 127 across catalog modules. (Not every site swallows a backend exception — some are internal invariants; the ones that catch Exception/IOException/SQLException from a backend call are the ones that erase dependency/retriability semantics.)

catalog-common (3)

catalog-common/src/main/java/org/apache/gravitino/catalog/hive/StorageFormat.java:64:    if (inputFormat == null) throw new RuntimeException("inputFormat must not be null");
catalog-common/src/main/java/org/apache/gravitino/catalog/hive/StorageFormat.java:65:    if (outputFormat == null) throw new RuntimeException("outputFormat must not be null");
catalog-common/src/main/java/org/apache/gravitino/catalog/hive/StorageFormat.java:66:    if (serde == null) throw new RuntimeException("serde must not be null");

catalog-fileset (19)

catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:324:      throw new RuntimeException("Failed to list filesets under namespace " + namespace, e);
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:346:      throw new RuntimeException("Failed to load fileset %s" + ident, ioe);
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:393:      throw new RuntimeException("Failed to list files in fileset" + filesetIdent, e);
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:421:      throw new RuntimeException("Failed to check if fileset " + ident + " exists", ioe);
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:431:      throw new RuntimeException("Failed to load schema " + schemaIdent, ioe);
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:510:              throw new RuntimeException(
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:534:        throw new RuntimeException("Failed to create fileset " + ident, ioe);
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:568:      throw new RuntimeException("Failed to create fileset " + ident, ioe);
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:631:      throw new RuntimeException("Failed to load fileset " + ident, ioe);
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:651:      throw new RuntimeException("Failed to update fileset " + ident, ioe);
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:656:      throw new RuntimeException(
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:713:      throw new RuntimeException("Failed to delete fileset " + ident, ioe);
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:737:      throw new RuntimeException("Failed to check if schema " + ident + " exists", ioe);
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:759:                  throw new RuntimeException(
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:781:              throw new RuntimeException(
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:798:      throw new RuntimeException("Failed to check if schema " + ident + " exists", ioe);
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:935:      throw new RuntimeException("Failed to delete schema " + ident + " location", ioe);
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:1080:                throw new RuntimeException(
catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:1426:      throw new RuntimeException("Interrupted when getting FileSystem for path: " + path, e);

catalog-glue (5)

catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueTableOperations.java:102:      throw new RuntimeException("Failed to list partitions for table " + tableName, e);
catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueTableOperations.java:124:      throw new RuntimeException("Failed to list partitions for table " + tableName, e);
catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueTableOperations.java:144:      throw new RuntimeException("Failed to get partition " + partitionName, e);
catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueTableOperations.java:184:      throw new RuntimeException("Failed to add partition " + partition.name(), e);
catalog-glue/src/main/java/org/apache/gravitino/catalog/glue/GlueTableOperations.java:213:      throw new RuntimeException("Failed to drop partition " + partitionName, e);

catalog-hive (24)

catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java:221:      throw new RuntimeException(e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java:265:      throw new RuntimeException(e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java:286:      throw new RuntimeException(e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java:335:      throw new RuntimeException(e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java:361:      throw new RuntimeException(e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java:412:      throw new RuntimeException(e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java:497:      throw new RuntimeException(
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java:669:      throw new RuntimeException(e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java:767:      throw new RuntimeException(e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveCatalogOperations.java:1066:      throw new RuntimeException(e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveViewCatalogOperations.java:93:      throw new RuntimeException("Failed to list Hive views in " + namespace, e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveViewCatalogOperations.java:160:      throw new RuntimeException("Failed to create Hive view " + ident, e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveViewCatalogOperations.java:162:      throw new RuntimeException("Failed to create Hive view " + ident, e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveViewCatalogOperations.java:267:      throw new RuntimeException("Failed to alter Hive view " + ident, e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveViewCatalogOperations.java:269:      throw new RuntimeException("Failed to alter Hive view " + ident, e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveViewCatalogOperations.java:314:      throw new RuntimeException("Failed to drop Hive view " + ident, e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveViewCatalogOperations.java:316:      throw new RuntimeException("Failed to drop Hive view " + ident, e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveViewCatalogOperations.java:352:      throw new RuntimeException("Failed to load Hive view " + ident, e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveViewCatalogOperations.java:354:      throw new RuntimeException("Failed to load Hive view " + ident, e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveTableOperations.java:59:      throw new RuntimeException(
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveTableOperations.java:73:      throw new RuntimeException(
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveTableOperations.java:84:      throw new RuntimeException(
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveTableOperations.java:131:      throw new RuntimeException(e);
catalog-hive/src/main/java/org/apache/gravitino/catalog/hive/HiveTableOperations.java:175:      throw new RuntimeException(

catalog-jdbc-common (1)

catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/MySQLProtocolCompatibleCatalogOperations.java:57:      throw new RuntimeException(

catalog-jdbc-doris (2)

catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/operation/DorisTableOperations.java:208:        throw new RuntimeException("Failed to get the number of backend servers", e);
catalog-jdbc-doris/src/main/java/org/apache/gravitino/catalog/doris/utils/DorisUtils.java:220:    throw new RuntimeException("Failed to extract distribution info in sql:" + createTableSql);

catalog-jdbc-starrocks (1)

catalog-jdbc-starrocks/src/main/java/org/apache/gravitino/catalog/starrocks/utils/StarRocksUtils.java:193:    throw new RuntimeException("Failed to extract distribution info in sql:" + createTableSql);

catalog-kafka (15)

catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:160:      throw new RuntimeException("Failed to create Kafka AdminClient", e);
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:177:      throw new RuntimeException(
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:183:      throw new RuntimeException("Failed to list topics under the schema " + namespace, e);
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:231:        throw new RuntimeException("Failed to load topic " + ident.name() + " from Kafka", e);
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:234:      throw new RuntimeException("Failed to load topic " + ident.name() + " from Kafka", e);
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:310:        throw new RuntimeException("Failed to create topic in Kafka" + ident, e);
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:313:      throw new RuntimeException("Failed to create topic in Kafka" + ident, e);
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:386:        throw new RuntimeException("Failed to drop topic " + ident.name() + " from Kafka", e);
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:389:      throw new RuntimeException("Failed to drop topic " + ident.name() + " from Kafka", e);
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:402:      throw new RuntimeException("Failed to list schemas under namespace " + namespace, e);
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:431:      throw new RuntimeException("Failed to load schema " + ident, ioe);
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:540:      throw new RuntimeException("Failed to increase partition count for topic " + topicName, e);
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:554:      throw new RuntimeException("Failed to alter topic properties for topic " + topicName, e);
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:595:      throw new RuntimeException("Failed to check if schema " + defaultSchemaIdent + " exists", e);
catalog-kafka/src/main/java/org/apache/gravitino/catalog/kafka/KafkaCatalogOperations.java:622:      throw new RuntimeException("Failed to create default schema for Kafka catalog", ioe);

catalog-lakehouse-generic (9)

catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceTableOperations.java:349:      throw new RuntimeException("Failed to purge Lance dataset for table " + ident, e);
catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceTableOperations.java:383:      throw new RuntimeException("Failed to drop Lance dataset for table " + ident, e);
catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceTableOperations.java:401:        throw new RuntimeException("Failed to delete Lance dataset at " + location, e);
catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceTableOperations.java:479:      throw new RuntimeException("Failed to create Lance dataset at location " + location, e);
catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceTableOperations.java:559:      throw new RuntimeException("Failed to repair table " + ident, e);
catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceTableOperations.java:670:      throw new RuntimeException("Failed to record empty version for table " + ident, e);
catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceTableOperations.java:795:      throw new RuntimeException(
catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/GenericCatalogOperations.java:359:        throw new RuntimeException(
catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/generic/GenericCatalogOperations.java:362:        throw new RuntimeException(

catalog-lakehouse-hudi (4)

catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/lakehouse/hudi/backend/hms/HudiHMSBackendOps.java:103:      throw new RuntimeException(e);
catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/lakehouse/hudi/backend/hms/HudiHMSBackendOps.java:116:      throw new RuntimeException(e);
catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/lakehouse/hudi/backend/hms/HudiHMSBackendOps.java:155:      throw new RuntimeException(e);
catalog-lakehouse-hudi/src/main/java/org/apache/gravitino/catalog/lakehouse/hudi/backend/hms/HudiHMSBackendOps.java:174:      throw new RuntimeException(e);

catalog-lakehouse-iceberg (5)

catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogOperations.java:242:      throw new RuntimeException(e);
catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogOperations.java:338:      throw new RuntimeException(e);
catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogOperations.java:365:      throw new RuntimeException(e);
catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogOperations.java:624:      throw new RuntimeException(e);
catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/ops/IcebergCatalogWrapperHelper.java:259:        throw new RuntimeException("RenameTable shouldn't use tableUpdate interface");

catalog-lakehouse-paimon (5)

catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogOperations.java:203:      throw new RuntimeException(e);
catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogOperations.java:267:      throw new RuntimeException(e);
catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogOperations.java:564:        throw new RuntimeException(e);
catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/PaimonCatalogOperations.java:682:      throw new RuntimeException(e);
catalog-lakehouse-paimon/src/main/java/org/apache/gravitino/catalog/lakehouse/paimon/utils/CatalogUtils.java:91:        throw new RuntimeException("Failed to login with kerberos", e);

catalog-model (14)

catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:109:      throw new RuntimeException("Failed to list models under namespace " + namespace, ioe);
catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:124:      throw new RuntimeException("Failed to get model " + ident, ioe);
catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:154:      throw new RuntimeException("Failed to register model " + ident, e);
catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:171:      throw new RuntimeException("Failed to delete model " + ident, ioe);
catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:188:      throw new RuntimeException("Failed to list model versions for model " + ident, ioe);
catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:205:      throw new RuntimeException("Failed to list model version infos for model " + ident, ioe);
catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:262:      throw new RuntimeException("Failed to link model version " + ident, e);
catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:314:      throw new RuntimeException("Failed to alter model " + ident, ioe);
catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:328:      throw new RuntimeException("Failed to load model " + ident, ioe);
catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:333:      throw new RuntimeException("Model already exist " + ident.name(), eaee);
catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:424:      throw new RuntimeException("Failed to alter model version " + ident, ioe);
catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:438:      throw new RuntimeException("Failed to load model version " + ident, ioe);
catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:554:      throw new RuntimeException("Failed to get model version " + ident, ioe);
catalog-model/src/main/java/org/apache/gravitino/catalog/model/ModelCatalogOperations.java:605:      throw new RuntimeException("Failed to delete model version " + ident, ioe);

hadoop-common (5)

hadoop-common/src/main/java/org/apache/gravitino/catalog/hadoop/fs/FileSystemUtils.java:161:      throw new RuntimeException("Failed to create GravitinoFileSystemCredentialProvider", e);
hadoop-common/src/main/java/org/apache/gravitino/catalog/hadoop/fs/FileSystemUtils.java:223:      throw new RuntimeException("Failed to create configuration", e);
hadoop-common/src/main/java/org/apache/gravitino/catalog/hadoop/fs/FileSystemUtils.java:234:      throw new RuntimeException("Failed to write property: " + key, e);
hadoop-common/src/main/java/org/apache/gravitino/catalog/hadoop/fs/HDFSFileSystemProxy.java:162:                throw new RuntimeException("Failed to invoke method", e);
hadoop-common/src/main/java/org/apache/gravitino/catalog/hadoop/fs/HDFSFileSystemProxy.java:193:      throw new RuntimeException("Failed to login with Kerberos", e);

hive-metastore-common (15)

hive-metastore-common/src/main/java/org/apache/gravitino/hive/kerberos/HmsKerberosClient.java:111:      throw new RuntimeException("Failed to create proxy user for Kerberos Hive client", e);
hive-metastore-common/src/main/java/org/apache/gravitino/hive/dyn/DynConstructors.java:125:        throw new RuntimeException(e.getCause());
hive-metastore-common/src/main/java/org/apache/gravitino/hive/dyn/DynConstructors.java:135:        throw new RuntimeException(e);
hive-metastore-common/src/main/java/org/apache/gravitino/hive/dyn/DynFields.java:59:        throw new RuntimeException(e);
hive-metastore-common/src/main/java/org/apache/gravitino/hive/dyn/DynFields.java:68:        throw new RuntimeException(e);
hive-metastore-common/src/main/java/org/apache/gravitino/hive/dyn/DynFields.java:378:        throw new RuntimeException(
hive-metastore-common/src/main/java/org/apache/gravitino/hive/dyn/DynMethods.java:103:        throw new RuntimeException(e.getCause());
hive-metastore-common/src/main/java/org/apache/gravitino/hive/dyn/DynMethods.java:113:        throw new RuntimeException(e);
hive-metastore-common/src/main/java/org/apache/gravitino/hive/dyn/DynMethods.java:446:        throw new RuntimeException("Cannot find method: " + name);
hive-metastore-common/src/main/java/org/apache/gravitino/hive/client/HiveClientFactory.java:112:      throw new RuntimeException("Failed to connect to Hive Metastore", e);
hive-metastore-common/src/main/java/org/apache/gravitino/hive/client/HiveClientFactory.java:249:      throw new RuntimeException("Failed to initialize kerberos client", e);
hive-metastore-common/src/main/java/org/apache/gravitino/hive/client/ProxyHiveClientImpl.java:69:      throw new RuntimeException("Failed to create Kerberos Hive client", ex);
hive-metastore-common/src/main/java/org/apache/gravitino/hive/client/Util.java:56:      throw new RuntimeException("Failed to create configuration", e);
hive-metastore-common/src/main/java/org/apache/gravitino/hive/client/HiveClientImpl.java:183:      throw new RuntimeException("Failed to close HiveClient", e);
hive-metastore-common/src/main/java/org/apache/gravitino/hive/client/HiveClientImpl.java:192:      throw new RuntimeException("Failed to get current user", e);

[nevzheng]

Horizon / timeline

I don't see this as a fire drill — but it's a compounding issue we should commit to resolving over roughly the next year, in three registers:

• Immediately — discipline. Integrate error design into how we write and review code: make "what domain is this failure, is it retriable, is it ours or a dependency's?" a standard review question. Costs nothing and stops the pattern from spreading further.
• Short-to-medium term — policy. Adopt a lightweight error-handling + review policy (fault domains, retriability, public vs internal, our-errors vs dependency-errors), optionally nudged by a lint/Checkstyle rule.
• Medium term — remediation. Work the remediation plan above: the server-side quick win, the taxonomy, per-catalog converters, then client retry.
• Long term. I don't expect this to become a long-term problem unless we ignore it — the cost only compounds if we let it sit.

Why a real horizon and not "someday": I believe this will affect enterprise engagements. Large customers need proper error handling to deploy Gravitino into — and integrate it with — their existing systems; their automation and their on-call depend on it. On-call's whole job is to firefight, and you can't firefight when every failure is the same generic 500 with no signal of whose fault it is or whether to retry. Clean error semantics are what let their operators (and ours) act instead of guess. (Checking with Mark on how this has shown up in real engagements.)

[markhoerth]

Weighing in from the product side, since Nevin asked how this shows up in real engagements. The short answer: it shows up early, and it shows up as a trust problem rather than a technical one.

In enterprise evaluations, especially at large financial services organizations, the platform and SRE teams wire retries, circuit breakers, alerting, and on-call routing to status codes before anything goes near production. When every failure is a 500, none of that automation can distinguish "back off and retry" from "page a human" from "open a ticket with the vendor." We have already seen the adjacent version of this ask: the health endpoints added in #10840 came directly from a customer's load balancer requirements. Error semantics are the same class of requirement, and I expect it to be raised explicitly as deployments deepen. A gateway that cannot say whose fault a failure is creates integration friction at exactly the wrong moment in an evaluation.

I also looked at how our own Web UI V2 handles this today, and it makes the case better than any hypothetical:

1. The UI's fallback message for a 500 is "Server error, unable to connect Gravitino!" (web-v2/web/src/lib/utils/axios/checkStatus.js). So when a customer's Hive Metastore is down and the server collapses it into a 500, our own UI tells the user that Gravitino is broken. We make the misattribution for ourselves.
2. The counterfactual already exists in the codebase. The one status the server emits with real meaning is 401, and the UI recovers from it gracefully: the response interceptor clears tokens and triggers re-auth. Where the taxonomy exists, the client recovers. Where everything is 500, it cannot. checkStatus even has a reasonable 503 message that users essentially never see, because we almost never send 503.
3. The UI's retry machinery (axiosRetry.js) is disabled by default (isOpenRetry: false), GET-only, and blind to status codes. Even if we turned it on, there is no signal from the server worth keying it on. Nevin's observation about the Java client extends to the UI for the same root cause.
4. Server exception text flows verbatim into user-facing toasts. checkStatus even does string surgery on the message body (searching for the substring "reason" and stripping brackets) to clean it up after the fact. Raw internal messages are effectively UI copy today, which strengthens the case for the public vs internal split.

On sequencing, two additions:

First, +1 on doing Phase 0 now. It is small, touches one layer, changes no client contracts, and fixes the worst misattribution for every entity at once. I will advocate for prioritizing it.

Second, PR #10387 (log and error message quality polish, currently open) touches many of the same message sites the Phase 2 converters will rewrite. Merging it first avoids a painful rebase later, and I would suggest treating it as the reference for message standards going forward.

Which brings me to the "discipline" register of Nevin's timeline. Since server error messages are user-visible copy, I would like to see us codify message content standards alongside the taxonomy: consistent phrasing conventions (the ones #10387 establishes), plus a simple template of what failed, in which domain, and what the caller can do about it. That belongs in CONTRIBUTING.md and in the standard review questions, next to "is it retriable" and "is it ours or a dependency's." It costs nothing and stops both problems, semantic and linguistic, from spreading further.

Happy to help work through the error-code stability question as well, since once clients depend on codes they become API surface, and that is a product commitment as much as an engineering one.

[jerryshao]

Authorization vs authentication (401 vs 403). Today 401 is effectively dead taxonomy and backend auth failures (Kafka AuthorizationException, JDBC SQLState 28, Iceberg 401/403) collapse to 500. How do we surface upstream authz failures vs Gravitino's own access control?

It depends on how we treat such data source issues. If we return 401/403, 1) It will mix with Gravitino's own authn/authz error, making it hard for the user to understand; 2) some systems will treat 401 as a specially case to do some additional process like relogin, reissue the tickets, which will not work for the data sources Gravitino connected. So it is debatable to change this semantics.

Rate limiting / backpressure (429 + Retry-After). Exists only for the lineage sink today — should the gateway propagate upstream throttling?

I think it makes sense. But, we don't support rate control / backpressure for now. We'd better adding rate control before we support these two error codes.

Timeouts (504 vs 503). Distinguish "upstream slow" from "upstream down"?

Again, itself makes sense. If we can differentiate these two errors, we can have two different codes.

Non-obvious retriability traps the converters must encode: Iceberg CommitStateUnknownException → never auto-retry (possible double-write); CommitFailedException → retry the whole commit, not the request; Kafka UnknownTopicOrPartitionException is marked retriable but is semantically 404; JDBC — classify on the marker hierarchy before SQLState (class 08 is ambiguous).

This should be treated case by case. We'd better not add the retry codes for all the cases. We need to separate into case by case thing.

Error-message hygiene — sanitize what crosses the boundary to callers (CWE-209) while keeping full detail in logs.

This is a good thing.

Error-code stability — once clients depend on codes they become API surface; how do we version them (per AIP-193's backwards-compat guidance)?

We can use OpenAPI to track the stability. But I cannot figure out a better solution. Maybe we can have more discussion on this.