Motivation
Spark + Java Paimon support case-insensitive column matching on reads. In Java this lives at the engine / catalog layer: Spark's spark.sql.caseSensitive (a session setting) drives Catalyst to resolve projection and filter column references case-insensitively and normalize them to the schema's field names before pushing down to the connector; Paimon core exposes it as a per-call parameter (RowType.getFieldIndex(name, caseSensitive)). It is not a table property.
The Rust implementation (paimon-rust) has no equivalent — projection and predicate column resolution are always exact-case. Engines that read Paimon through the Rust core / C bindings / DataFusion / Python (e.g. Apache Doris) therefore cannot offer the case-insensitive behavior users get from Spark. A user querying SELECT NAME or WHERE NAME = ... against a schema field name fails, even though the same query works via Spark.
Proposal
Expose case sensitivity as a read-time parameter (mirroring Java's per-call caseSensitive), controlled by the reader/engine rather than stored on the table. Default stays case-sensitive, preserving today's exact-match behavior; a reader opts in per read. When disabled, column-name resolution on the read path matches by ASCII case-folding and reports an error when a name is ambiguous (multiple schema fields collide under folding), mirroring Spark's AMBIGUOUS behavior.
Surface:
- Core:
ReadBuilder::with_case_sensitive(bool); PredicateBuilder::new_with_case_sensitive(fields, bool).
- DataFusion: derived from the session (
!sql_parser.enable_ident_normalization) for scan and filter pushdown.
- Python:
PyReadBuilder::with_case_sensitive(bool).
- C binding:
paimon_read_builder_with_case_sensitive, plus a case_sensitive parameter on the paimon_predicate_* constructors (used by Doris and the Go binding).
Out of scope for the initial change:
- Nested/dotted column paths (top-level only).
- The
_ROW_ID system column stays an exact reserved name.
- Write path / DDL.
Notes
Default stays case-sensitive, so existing behavior is unchanged unless a reader explicitly requests case-insensitive matching. No table option and no storage-format change.
Motivation
Spark + Java Paimon support case-insensitive column matching on reads. In Java this lives at the engine / catalog layer: Spark's
spark.sql.caseSensitive(a session setting) drives Catalyst to resolve projection and filter column references case-insensitively and normalize them to the schema's field names before pushing down to the connector; Paimon core exposes it as a per-call parameter (RowType.getFieldIndex(name, caseSensitive)). It is not a table property.The Rust implementation (
paimon-rust) has no equivalent — projection and predicate column resolution are always exact-case. Engines that read Paimon through the Rust core / C bindings / DataFusion / Python (e.g. Apache Doris) therefore cannot offer the case-insensitive behavior users get from Spark. A user queryingSELECT NAMEorWHERE NAME = ...against a schema fieldnamefails, even though the same query works via Spark.Proposal
Expose case sensitivity as a read-time parameter (mirroring Java's per-call
caseSensitive), controlled by the reader/engine rather than stored on the table. Default stays case-sensitive, preserving today's exact-match behavior; a reader opts in per read. When disabled, column-name resolution on the read path matches by ASCII case-folding and reports an error when a name is ambiguous (multiple schema fields collide under folding), mirroring Spark'sAMBIGUOUSbehavior.Surface:
ReadBuilder::with_case_sensitive(bool);PredicateBuilder::new_with_case_sensitive(fields, bool).!sql_parser.enable_ident_normalization) for scan and filter pushdown.PyReadBuilder::with_case_sensitive(bool).paimon_read_builder_with_case_sensitive, plus acase_sensitiveparameter on thepaimon_predicate_*constructors (used by Doris and the Go binding).Out of scope for the initial change:
_ROW_IDsystem column stays an exact reserved name.Notes
Default stays case-sensitive, so existing behavior is unchanged unless a reader explicitly requests case-insensitive matching. No table option and no storage-format change.