Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(dags): Add tests for dbt operators in DAGs #56

Merged
merged 6 commits into from May 4, 2022

Conversation

tomasfarias
Copy link
Owner

Closes #55

This PR adds more integral testing by defining DAGs with Dbt* operators, triggering runs, and checking outputs. We test both the context manager way of defining a dag as well as the TaskFlow API.

As I was working on this I noticed the example DAGs were actually broken 😅. So this PR also fixes them.

@tomasfarias
Copy link
Owner Author

We should add a test that includes a on_failure_callback default parameter and a DbtBuildOperator task to see if we can catch #51

@codecov
Copy link

codecov bot commented May 3, 2022

Codecov Report

Merging #56 (dac0760) into master (c4d13d3) will increase coverage by 0.10%.
The diff coverage is n/a.

❗ Current head dac0760 differs from pull request most recent head 7806761. Consider uploading reports for the commit 7806761 to get more accurate results

@@            Coverage Diff             @@
##           master      #56      +/-   ##
==========================================
+ Coverage   98.58%   98.69%   +0.10%     
==========================================
  Files           8        8              
  Lines         917      917              
==========================================
+ Hits          904      905       +1     
+ Misses         13       12       -1     
Flag Coverage Δ
unittests 98.69% <ø> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
airflow_dbt_python/operators/dbt.py 100.00% <0.00%> (+0.33%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4d13d3...7806761. Read the comment docs.

@tomasfarias
Copy link
Owner Author

Unfortunately, dag tests interact with a lot of Airflow internals that have changed even from version 2.0 to 2.2. Maintaining all 3 versions of Airflow for that set of tests will be too hard, so the older two have been dropped.

Will update the docs to mention examples have been tested with Airflow > 2.2

@tomasfarias tomasfarias marked this pull request as ready for review May 4, 2022 16:39
@tomasfarias tomasfarias merged commit 346f278 into master May 4, 2022
@tomasfarias tomasfarias deleted the integration-testing branch May 4, 2022 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement integration testing pipeline for example DAGs
1 participant