Skip to content

Add Query Option to Table Provider #467

@ShimonSte

Description

@ShimonSte

With the Spark catalog, SQL queries go through Spark's query engine, causing two issues:

  1. Limited function support - ClickHouse-specific functions that aren't registered in Spark will fail
  2. Performance degradation - Incomplete query pushdown forces parts of the query to execute in Spark instead of ClickHouse

I propose add a query option to the TableProvider that executes raw SQL directly in ClickHouse and efficiently retrieves results to Spark. This bypasses Spark's query planner, preserving full ClickHouse functionality and performance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions