-
-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
(Custom) table engine bindings #52
Comments
This is also a sort of table alias but with interception of the query so you can do magic with the query predicate (e.g. transpiling to get a list of parquet files based on partitions like I do in IceDB, and returning a s3() that reads a list of parquets) |
However the above also does depends on the s3 function NOT using S3 ListObject which it currently does right now every time you give it blob format, whether you specify the entire list or not... |
Some places I would like to use this are: |
Being able to bind a custom table engine (not function) that exhibits dynamic behavior would be really powerful.
For example, similar to how I have the parquet merge engine IceDB where I have an example querying from a custom table function, it would be much nicer if the columns for filtering the metadata could be determined via the query predicate (
WHERE
conditions) rather than function parameters.This would allow you to fully decoupled storage, compute, and metadata to build a serverless OLAP DB that natively support multi-tenancy and dynamic schema (through column name merging, existing clickhouse feature). This could then effectively be aliased to being a list of parquet files to read, with merge on name enabled (can't find the docs in clickhouse to save my life for that setting).
Effectively this would be intercepting the AST in a very DX-friendly way, and returning a custom clickhouse table function in its place before execution.
The text was updated successfully, but these errors were encountered: