Materialized connector inital setup #417
Conversation
|
andresgutgon
force-pushed
the
feature/find-query-files-with-relative-paths
branch
4 times, most recently
from
2d36079
to
8a6b394
Compare
andresgutgon
force-pushed
the
feature/materialize-connector-start
branch
2 times, most recently
from
7bd871f
to
38f2639
Compare
| const compiledSubQuery = await this.source.compileQuery( | ||
| const refSource = | ||
| await this.source.manager.loadFromQuery(fullSubQueryPath) | ||
| const compiledSubQuery = await refSource.compileQuery( |
There was a problem hiding this comment.
This is important. Now runQuery is cross-source. Meaning you can call a query from another source.
| const materialize = compiledSubQuery.config.materialize_query | ||
| const isFalse = materialize === false | ||
|
|
||
| if (isFalse) { |
There was a problem hiding this comment.
The isFalse variable is not really needed... You can just do if (!materialize)
There was a problem hiding this comment.
I know but I want to let users know when they configured with false
| throw new Error( | ||
| `'${referencedQuery}' query can not have parameters to filter the SQL query. You're using ${usedParams} params in the query`, |
There was a problem hiding this comment.
Do we want to show the user which params have they used? I think that can be implicit
There was a problem hiding this comment.
I think it doesn't hurt no?
This commit just introduce the materialize connector but is will need another piece to work that we'll introduce in another PR. For now we don't publish this connector
andresgutgon
force-pushed
the
feature/materialize-connector-start
branch
from
38f2639
to
0da39e0
Compare
What?
We want to allow users cache queries from their sources on disk to improve performance. The queries will be stored in parquet format and read with DuckDB. For that reason, this new connector inherits from our existing DuckDB connector. This PR starts the work in this connector to make it usable in development and writing queries to disk.
Issue ticket number and link
#369
Dependency
#416
Checklist before requesting a review
Sourceto connectors (done in another PR)materializedReffunction