Very interesting project, how does it work? #356
Replies: 1 comment 1 reply
-
Hi @GavinRay97 , thank you for your interest in connectorx! You are basically correct with the workflow. In general, for federated queries we:
As you mentioned, we use Calcite in step 1 to optimize (push down some operators) and split the query into multiple ones (each run on a single data source). Since our main code is written in rust, we use For step 2, we directly fetch the data from each datasource into arrow format through rust client drivers (note that unlike in Calcite which use JDBC to connect to sources, we use native rust drivers here), leveraging our optimizations in connectorx (Feel free to look at our technical report if you are interested in this part). For this part, the schema is actually refetched in rust for each datasource's implementation of the For step 3, we directly leverage |
Beta Was this translation helpful? Give feedback.
-
Hello, this project is super cool! Especially what looks like federated querying feature.
I took a look at the code, and from what I can understand it looks like:
JdbcSchema
's are created using JVM <-> Rust bridge for the datasourceRecordBatch
objectsIs this correct?
I would love to understand more about the approach, it seems clever🙂
Beta Was this translation helpful? Give feedback.
All reactions