Skip to content

Support case-insensitive column matching on reads (read-time parameter) #495

Description

@JunRuiLee

Motivation

Spark + Java Paimon support case-insensitive column matching on reads. In Java this lives at the engine / catalog layer: Spark's spark.sql.caseSensitive (a session setting) drives Catalyst to resolve projection and filter column references case-insensitively and normalize them to the schema's field names before pushing down to the connector; Paimon core exposes it as a per-call parameter (RowType.getFieldIndex(name, caseSensitive)). It is not a table property.

The Rust implementation (paimon-rust) has no equivalent — projection and predicate column resolution are always exact-case. Engines that read Paimon through the Rust core / C bindings / DataFusion / Python (e.g. Apache Doris) therefore cannot offer the case-insensitive behavior users get from Spark. A user querying SELECT NAME or WHERE NAME = ... against a schema field name fails, even though the same query works via Spark.

Proposal

Expose case sensitivity as a read-time parameter (mirroring Java's per-call caseSensitive), controlled by the reader/engine rather than stored on the table. Default stays case-sensitive, preserving today's exact-match behavior; a reader opts in per read. When disabled, column-name resolution on the read path matches by ASCII case-folding and reports an error when a name is ambiguous (multiple schema fields collide under folding), mirroring Spark's AMBIGUOUS behavior.

Surface:

  • Core: ReadBuilder::with_case_sensitive(bool); PredicateBuilder::new_with_case_sensitive(fields, bool).
  • DataFusion: derived from the session (!sql_parser.enable_ident_normalization) for scan and filter pushdown.
  • Python: PyReadBuilder::with_case_sensitive(bool).
  • C binding: paimon_read_builder_with_case_sensitive, plus a case_sensitive parameter on the paimon_predicate_* constructors (used by Doris and the Go binding).

Out of scope for the initial change:

  • Nested/dotted column paths (top-level only).
  • The _ROW_ID system column stays an exact reserved name.
  • Write path / DDL.

Notes

Default stays case-sensitive, so existing behavior is unchanged unless a reader explicitly requests case-insensitive matching. No table option and no storage-format change.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions