Skip to content

[spark] partition filter pushdown for non-lake log and kv tables #3241

@fresh-borzoni

Description

@fresh-borzoni

Search before asking

  • I searched in the issues and found nothing similar.

Description

When a Spark query on a partitioned non-lake table has a WHERE clause referencing partition columns, the connector shall extracts a partition-only sub-predicate and prune the partition list at planning time.

Partitions that can't match are skipped - no per-partition log or kv-snapshot reads are issued for them.

Willingness to contribute

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No fields configured for Task.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions