Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Use dbt selectors to decide which Ibis models to run #57

Open
binste opened this issue May 5, 2024 · 0 comments
Open

Feature: Use dbt selectors to decide which Ibis models to run #57

binste opened this issue May 5, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@binste
Copy link
Owner

binste commented May 5, 2024

To speed up the compilation from Ibis models to SQL, we could do it for only the models which are relevant for the current dbt command, i.e. we could use the selector syntax with --select and --exclude. We'd need to cache the schemas of the Ibis expressions between runs so that schemas of upstream Ibis dependencies which are not selected are available.

However, we can't use dbt list --select ... to tell us which models to compile as dbt does not know about the Ibis models until they have been compiled to SQL.

-> This needs to wait until there is a better plugin support in dbt so that Ibis models can directly participate in the dbt dag without being first compiled to sql.

@binste binste added the enhancement New feature or request label May 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant