Does connector-x support filter pushdown with polars?

[chitralverma]

Hi,
I was trying some experiments with polars over my local mysql instance and was wondering if filter/ predicate pushdown are supported.

My query looks something like SELECT * FROM <DB>.<TABLE> and then using polars DSL im forming an expression WHERE <col> > 10 .

Sample code:

import polars as pl

df = pl.read_sql(sql="select * from SELECT * FROM my_table "], connection_uri="mysql://user:pass@localhost:3306/static_data") \
.filter(pl.col("sno") > 4)

print(df)

I observed in the general logs table of mysql that the executed query does not contain the filter part. Am I observing something this incorrectly?

[ritchie46]

That's correct. Polars optimizations are not pushed down into connector-x. Any read_<> function in polars will be eager and will have no optimizations. scan_<> functions do. However we don't push down to sql. That's not trivially implemented.

[chitralverma]

Ok. thanks @ritchie46 for the explanation.

so if these optimisations were to be built for something like a scan_sql where would it be done? In polars or in connector-x?

[ritchie46]

Well we need to be able to update a SQL query with our projections and our predicates. Given that your query can be anything, this to me seems like a whole project on its own.

[chitralverma]

you are actually right,.

I did some research, there is a project for all of this - Apache Calcite. Parses SQL to relation algebra, validates and optimizes queries. No python or rust bindings though