Support pluggable authentication providers for REST Namespace (e.g. AWS SigV4)

[shiwk]

(Note: Migrated from lance-format/lance-namespace#327 for better visibility.)

Motivation

Cloud-managed catalog services are becoming the norm. AWS S3 Tables provides an Iceberg-compatible catalog directly from the storage layer. GCP BigLake offers similar capabilities. These services use their native IAM authentication (e.g. AWS SigV4) rather than OAuth2 or static Bearer tokens.

For the Lance REST Namespace to integrate with these services — or any catalog service fronted by cloud API gateways with IAM auth — it needs to support request-signing authentication at the protocol level. This is essentially an enrichment of the Credential Vending mechanism in Lance Namespace: beyond vending storage credentials for data access, the namespace itself needs flexible authentication when talking to catalog endpoints.

This was discussed in lance-format/lance-namespace#10 where the conclusion was that REST only needs OAuth2 since native implementations handle other auth flows. I believe this deserves reconsideration given how the ecosystem has evolved — expecting every cloud catalog deployment to set up an OAuth2 token exchange layer creates unnecessary friction.

Current State

The OpenAPI spec (docs/src/rest.yaml) defines three securitySchemes: OAuth2 (client credentials), Bearer Token, and API Key. No scheme exists for request-signing mechanisms. The existing client implementations (Rust, Java, Python) all mirror this limitation — none of them support request-level signing.

While native implementations (lance-namespace-impls) can work around this by talking to cloud services directly (e.g. the Glue implementation uses AWS SDK with IAM credentials), relying on native implementations as the sole solution for cloud auth has several drawbacks:

• Misplaced responsibility: Catalog service providers should focus on catalog semantics, not on reimplementing the full Lance Namespace interface. If a cloud catalog already speaks the REST protocol, requiring a dedicated native implementation just for auth is unnecessary overhead.
• Community fragmentation: As more cloud vendors offer catalog services, each requiring its own native implementation, the ecosystem risks becoming fragmented with many parallel implementations that are hard to maintain and keep consistent.
• Undermines the value of REST as a standard: The REST Namespace is designed to be a universal interop layer. If auth limitations push users toward vendor-specific native implementations even when the catalog already speaks REST, it effectively encourages the community to bypass the REST standard.
• Duplicated effort: Each native implementation must reimplement protocol handling, error mapping, pagination, and other concerns that the REST client already handles well. The only missing piece is authentication.

Proposal

Introduce a pluggable, registry-based authentication provider mechanism for the REST Namespace. Rather than hardcoding specific auth schemes, allow users to register an auth provider that can intercept and mutate outgoing HTTP requests.

Key design points:

• Pluggable architecture: Define an auth provider interface (e.g. a RequestSigner or AuthProvider trait/interface) that has access to the full HTTP request and can sign or modify it before execution. Users and cloud vendors can implement this interface for their specific auth mechanism.
• Registry-based: Auth providers should be registerable by name (e.g. sigv4, gcp-iam), similar to how namespace implementations are registered via ConnectBuilder. Configuration would drive which provider is activated and with what parameters.
• Spec-level extensibility: The OpenAPI spec should acknowledge that the REST Namespace security model is extensible beyond the three built-in schemes, potentially via a convention for custom security schemes.
• Built-in providers for major clouds: Ship first-party providers for widely-used mechanisms (starting with AWS SigV4) behind feature flags, so common use cases work out of the box.

Use Cases

• AWS S3 Tables: S3 Tables exposes an Iceberg REST catalog interface authenticated via SigV4. Although it does not yet support the Lance format, it represents the direction cloud-managed catalogs are heading. A Lance REST Namespace client with SigV4 support could connect to services like this directly without an intermediate auth proxy.
• AWS API Gateway + IAM Auth: Catalog services deployed behind API Gateway using IAM authorization require SigV4-signed requests from clients.
• GCP BigLake / Managed Catalogs: GCP-managed catalog services that rely on Google IAM and signed requests.
• General cloud-native deployments: Any environment where IAM-based auth is already the standard and adding OAuth2 infrastructure is unnecessary overhead.

References

• Prior discussion: Discuss AuthN Story lance-namespace#10
• AWS SigV4 documentation
• Apache Iceberg REST Catalog AuthManager — prior art for pluggable auth in a REST catalog spec

---

I'm happy to discuss this further and contribute to the design and implementation.

[shiwk]

@jackye1995 @westonpace hi! any quick thoughts on this or a suggested workaround? Thanks!

[jackye1995]

Thanks for pinging me here, I missed the thread in namespace

Introduce a pluggable, registry-based authentication provider mechanism for the REST Namespace. Rather than hardcoding specific auth schemes, allow users to register an auth provider that can intercept and mutate outgoing HTTP requests.

+1 overall for completing the auth side of the story. As engineer from lancedb, LanceDB is unfortunately a bit primitive at this point and just use api keys. We added the flexibility to allow passing different auth token per-request, which some community members are using. It gives wide flexibility, but expects user to have a custom namespace implementation, instead of directly using rest. If there are other use cases in the community that can extend the auth side of functionalities, feel free to have a try if you are interested in it!

For a pluggable interface, I think Lance's rust-based binding strategy imposes some challenge here. You need to expose the interface all the way in all language bindings, which is going to get pretty complicated. If what you want is full control, why not just do a custom namespace implementation at that point? There is a handful of common auth flows and auth schemes, which I think it might make more sense to just support in rust and allow user to directly configure in java/python/other languages.

[shiwk]

There is a handful of common auth flows and auth schemes, which I think it might make more sense to just support in rust and allow user to directly configure in java/python/other languages.

Thanks for the thoughtful response @jackye1995!

I agree that a fully pluggable interface across all language bindings would introduce significant complexity that
likely isn't justified. There's a clear trade-off between flexibility and code complexity here, and supporting a
well-defined set of common auth schemes directly in Rust — with configuration-driven usage in Python/Java/etc. —
strikes a much more pragmatic balance.

To share a bit more context on why I raised this: in my experience, authentication tends to be one of the last
things considered when selecting or building a catalog system. Catalog providers naturally focus on catalog
semantics — storage layout, metadata management, query integration — and auth is often not their area of expertise.
Yet in practice, especially with cloud-managed services, we've seen customers get blocked by exactly this
seemingly minor piece. A catalog that speaks the right protocol but doesn't support the expected auth mechanism
becomes surprisingly hard to adopt. That friction is really what motivated this issue. In practice, this friction pushes users
away from the REST Namespace toward custom implementations, which undermines its position as the standard interop
layer.

I'd love to contribute on this front. I'm planning to spend the next few days putting together a concrete proposal
and a TODO list — scoping out which auth schemes to prioritize, how they'd fit into the existing REST client
architecture, and what the configuration surface would look like.

Will follow up here once I have something to share.

[jackye1995]

Sounds great!

[shiwk]

Tracking implementation in #7033