Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support Airflow connections for dbt targets #53

Merged
merged 11 commits into from Apr 29, 2022

Commits on Apr 26, 2022

  1. feat: Support Airflow connections for dbt targets

    The objective of airflow-dbt-python has always been to make dbt a
    first-class citizen of Airflow. As part of this goal, we want to
    integrate dbt with all of Airflow's features. In particular, Airflow
    connections allow us to safely store connection information for
    operators to use, so all dbt operators should be able to leverage them
    too.
    
    The way we achieve this is by manually instantiating a dbt Project and
    Profile. When reading the latter, we also inject any Airflow
    connections that match the given target argument (if any). Moreover,
    if the profile is not defined, we simply create our own with any
    Airflow connection that was passed as a target (of course, missing
    both a profiles file and an Airflow connection raises an error).
    
    Future work may extend this to support custom connection types, at the
    moment we are doing a best effort to include all possible arguments,
    but it's not perfect.
    
    This feature was suggested in the dbt Slack channel as a way to avoid
    having to manage a profiles.yml file, that may contain sensitive
    information, making it a bad target for version control.
    tomasfarias committed Apr 26, 2022
    Copy the full SHA
    a3e7522 View commit details
    Browse the repository at this point in the history
  2. chore: Update dependencies due to conflicts between dbt and Airflow

    Dependency resolution was taking forever with Poetry. To solve this,
    we added Airflow's list of constraints. A few had to be relaxed to
    allow for dbt and Airflow to play well together.
    tomasfarias committed Apr 26, 2022
    Copy the full SHA
    f824d5a View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2022

  1. Copy the full SHA
    fe4a0b9 View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    254109d View commit details
    Browse the repository at this point in the history
  3. fix: Missing dependency

    tomasfarias committed Apr 29, 2022
    Copy the full SHA
    8d72b53 View commit details
    Browse the repository at this point in the history
  4. Copy the full SHA
    977ccc2 View commit details
    Browse the repository at this point in the history
  5. Copy the full SHA
    eef25d1 View commit details
    Browse the repository at this point in the history
  6. Copy the full SHA
    dcd5c8d View commit details
    Browse the repository at this point in the history
  7. Copy the full SHA
    eec0b1c View commit details
    Browse the repository at this point in the history
  8. Copy the full SHA
    be6abc3 View commit details
    Browse the repository at this point in the history
  9. Copy the full SHA
    fa9d991 View commit details
    Browse the repository at this point in the history