feat(rest): support rest catalog for databases, tables and snapshots

[SteNicholas]

Purpose

Linked issue: #164

Add a REST catalog implementation, selected via the metastore=rest option, that talks the Paimon REST catalog open API.

New components

• RestHttpClient — blocking libcurl client with exponential-backoff retries:
  • 429/503 responses are retried for all methods, transport errors only for idempotent methods via an explicit allowlist of transient failures (name-resolution and connect failures, timeouts, TLS failures and a server closing the connection without responding are never retried).
  • A Retry-After response header — delta-seconds or HTTP-date form, parsed locale-independently — takes precedence over the backoff when it yields a positive delay. Every retry sleep is bounded by a per-sleep cap and an overall retry budget; a Retry-After beyond either bound stops retrying rather than sleeping less than the server requested.
  • Redirects keep the method and body of POST/DELETE requests (CURLOPT_POSTREDIR) and are restricted to http(s) targets.
  • Header names must be valid HTTP tokens and header values must not contain CR/LF/NUL; a violating header fails the request before anything is sent.
  • Easy handles are pooled and reset between requests, so a request reuses the connection opened by the previous one instead of paying a TCP and TLS handshake every time; TLS below 1.2 is refused and requests carry a default paimon-cpp user agent that a header.User-Agent option overrides.
  • Per-request logging carries the request id (debug on success, warn on a retry or a final failure).
• RestApi — HTTP + JSON layer:
  • /v1/config option merging (overrides > client options > defaults), header. options sent as request headers, and paged listing.
  • Error mapping to Status (404 → NotExist, 409 → Exist, 400 → Invalid, 501 → NotImplemented) that prefers the code of the parsed error body over the http status.
  • Sensitive server messages are redacted, and response bodies are never echoed into error messages (they may carry credentials), so a body that is not an error object is reported as unparsable, distinct from an error object carrying no message.
  • Errors carry the request id (x-request-id, falling back to any request-id header) and attach a RestErrorDetail with the mapped code, so callers can distinguish e.g. authentication failures programmatically.
• RestCatalog — database/table create/drop/rename/list/get, snapshot listing, table-default. option defaults on table creation, and system/branch table checks:
  • The server table response is converted into a TableSchema with the computed highestFieldId: ids at or above SpecialFieldIds::SYSTEM_FIELD_ID_START are excluded, a duplicated field id at any nesting level is rejected, and a missing or wrong-typed partitionKeys/primaryKeys/options member fails instead of silently defaulting.
  • Branch identifiers are passed through to the server, which resolves the branch and returns the branch's own schema (the default branch main maps case-insensitively to the bare table); snapshots of a branch are listed under the branch object name.
  • The sys database serves the local global system tables like FileSystemCatalog.
• Bear token authentication provider (token.provider=bear, the protocol's historical spelling of "bearer").

Shared code

• The pieces overlapping with the object-store file systems are unified under common/utils: the generic HTTP client moved there from common/fs and is shared by the S3 file system and the REST catalog, UrlUtils carries both URL-encoding flavors (form-urlencoded for the REST api, RFC 3986 for S3) with the S3 client reusing it, and libcurl detection in CMake is a single find_package(CURL) gated on PAIMON_ENABLE_S3 OR PAIMON_ENABLE_REST.
• CatalogUtils holds the system-database and system/branch table checks shared by FileSystemCatalog and RestCatalog, with the check order and messages aligned with the Java CatalogUtils (Cannot 'createTable' for system table ...). FileSystemCatalog now also rejects branch identifiers on create/drop/rename — a branch identifier resolves to the main table directory, so dropping t$branch_b would otherwise delete the whole table.
• The new SensitiveConfigUtils centralizes credential redaction with the Java SensitiveConfigUtils key markers and masking rule (matched on the key lower-cased with separators removed; a key naming a true secret is masked as a whole, while an identifier-like key such as dlf.access-key-id keeps at most a four-character tail of a long enough value). sys.catalog_options masks every row through it so credentials such as the bearer token never surface to users who can query that table, and a server error message carrying a secret marker is redacted as a whole.
• RestUtil::ExtractRequestId and RestAuthParameter::Create likewise hold the request-id lookup and the query-parameter encoding in one place, so the log line and the error message report the same id and no call site can sign an unencoded parameter.

Build

libcurl is used from the system rather than bundled, so the new CMake option PAIMON_ENABLE_REST defaults to OFF, consistent with the other optional components that need something outside the bundled third-party toolchain (jindo, lance, lumina, lucene, tantivy). CI enables it explicitly in ci/scripts/build_paimon.sh and installs the libcurl development package, so the REST code and its tests are still built and run on every job. With the option off, Catalog::Create reports that metastore=rest requires a build with PAIMON_ENABLE_REST=ON.

Follow-ups (not in this PR): DLF signing, data-token file IO, paged listing parameters (maxResults/name patterns), and the remaining endpoints (alter/commit/views/partitions/tags/functions).

Tests

New rest_test binary (62 cases, registered under the unittest label so it runs in CI), all against an in-process mock HTTP/REST server:

• RestHttpClientTest: uri normalization, query encoding, a pooled handle carrying no method or body over from the previous request, retry classification (429/503 retried, 404 and non-allowlisted transport errors such as a refused connection or a redirect loop not, truncated responses retried only for idempotent methods), Retry-After precedence in both forms plus the per-sleep and overall budget bounds, retry exhaustion, redirect following with POST keeping its method and body, rejection of invalid header names/values, and transport errors omitting the url and query.
• RestUtilTest / ResourcePathsTest / RestMessagesTest, each next to its implementation: prefix extraction and the request-id lookup including the gateway-header fallback, resource path building with url-encoded segments, JSON round trips of all request/response messages, and config merge with null-value filtering (a key present in defaults, client options and overrides at once, plus the null-override and null-default paths).
• RestCatalogTest: client-side option validation (missing uri/token, unsupported token provider), end-to-end catalog operations including pagination, ignore_if_exists / ignore_if_not_exists paths, client/server headers and the json content type, table-default. defaults, system tables, branch tables resolved by the server, broken-schema rejection (missing and wrong-typed partitionKeys/primaryKeys/options members), a non-empty partition-key round trip, server errors surfacing as errors instead of "does not exist", and snapshot listing with sorting.
• RestApiErrorTest: http-status-to-Status mapping including the request-id fallback, redaction and the RestErrorDetail code.

The shared pieces are covered next to their implementations:

• UrlUtilsTest: both encoding flavors, including that a literal % is encoded even when / is preserved, which is what the S3 client relies on.
• SensitiveConfigUtilsTest: key classification, whole-value and tail-preserving masking, and free-form text redaction.
• HttpClientUtilTest in the common utils suite.
• TableSchemaTest.TestComputeHighestFieldId: the highest-field-id computation.
• SnapshotTest.TestToSnapshotInfo: the snapshot-to-info conversion.
• FileSystemCatalogTest.TestBranchIdentifierRejectedForTableOperations: the branch rejection.
• The sys.catalog_options integration test: credential-carrying options (token, access-key secret, account key, credential, SAS) are masked in every emitted row, and an access-key id keeps only its four-character tail.

API and Format

• New public header include/paimon/catalog_options.h with a PAIMON_EXPORTed CatalogOptions struct holding the catalog-level option keys METASTORE, URI, TOKEN, TOKEN_PROVIDER and the TABLE_DEFAULT_OPTION_PREFIX (table-default.) key prefix; table-level keys stay in Options (defs.h), mirroring the CatalogOptions/CoreOptions separation of the Java implementation.
• RestCatalog implements the Catalog interface including GetOptions(), which returns the client options merged with the server-side /v1/config.
• Catalog::Create now dispatches on the metastore option (filesystem remains the default; unknown values are rejected). For metastore=rest, root_path is not a filesystem path but the warehouse (instance) name registered on the REST server, documented on the API.
• No storage format changes. The wire protocol is the Paimon REST catalog open API.

Documentation

• docs/source/user_guide/catalog.rst now documents both metastores: the existing filesystem metastore section and a new REST Catalog section covering the PAIMON_ENABLE_REST build requirement, the root_path semantics (warehouse/instance name), the CatalogOptions keys (metastore, uri, token.provider, token, table-default.<key>) and a configuration example, replacing the note that claimed REST catalog support is future work. It also states which parts of the REST catalog are not implemented yet (altering a database or a table, views, functions, partitions, tags, branch management, consumers and the dlf token provider), so the guide is not read as promising them.
• docs/source/building.rst lists -DPAIMON_ENABLE_REST=ON under the optional components, including its libcurl requirement.

Generative AI tooling

Generated-by: Claude Code (claude-fable-5, claude-opus-5)

🤖 Generated with Claude Code

[SteNicholas]

Cherry pick alibaba/paimon-cpp#441.

[zjw1111]

+1

[JingsongLi]

+1