Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for "Path" select/exclude to DbtDag & DbtTaskGroup parsers #89

Closed
chrishronek opened this issue Jan 23, 2023 · 1 comment · Fixed by #99
Closed

Add support for "Path" select/exclude to DbtDag & DbtTaskGroup parsers #89

chrishronek opened this issue Jan 23, 2023 · 1 comment · Fixed by #99
Assignees
Labels
enhancement New feature or request

Comments

@chrishronek
Copy link
Contributor

chrishronek commented Jan 23, 2023

See dbt docs on cli usage examples here

$ dbt run --select path.to.my.models     # runs all models in a specific directory
$ dbt run --select path/to/models        # run models contained in path/to/models
$ dbt run --select path/to/my_model.sql  # run a specific model by its path

Ultimately, in our parsers, we should be able to have a new parameter that looks something like this:

# (only select or exclude parameter would be specified with staging dir not both)
jaffle_shop = DbtTaskGroup(
    ...
    select={"paths": ["../models/staging"]} # runs all models in the staging directory,
    exclude={"paths": ["../models/staging"]} # runs all models in except the staging directory
)
@chrishronek chrishronek added the enhancement New feature or request label Jan 23, 2023
@chrishronek chrishronek changed the title Create an Airflow DAG/Task Group by using path selector Create an Airflow DAG/Task Group by using Path selector Jan 23, 2023
@chrishronek
Copy link
Contributor Author

chrishronek commented Jan 23, 2023

@jlaneve wanted to discuss the following:

How applicable is this? Isn't this already being done with the dbt_project_name parameter of the parsers? I can't imagine a use case where this is useful?

Jaffle Shop has a staging sub-directory in the project so we'll use that. Also MG gave a good use case of how to do this.

@chrishronek chrishronek self-assigned this Jan 23, 2023
@chrishronek chrishronek changed the title Create an Airflow DAG/Task Group by using Path selector Add support for "Path" select/exclude to DbtDag & DbtTaskGroup parsers Jan 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant