Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add exclusion_condition to expect_row_values_to_have_data_for_every_n_datepart #141

Merged
merged 25 commits into from Apr 4, 2022

Conversation

gofford
Copy link
Contributor

@gofford gofford commented Jan 20, 2022

This PR addresses #140 by adding an exclusion_condition the the row_values_to_have_data_for_every_n_datepart test.

This condition allows the user to specify a criterion for excluding dates from the test output, thereby allowing the test to pass if there are dates that are known to be missing.

exclusion_condition, which defaults to None, is a SQL clause that is appended to the where closing where statement.

Example(s):

tests:
  - dbt_expectations.expect_row_values_to_have_data_for_every_n_datepart:
      date_col: date_day
      date_part: day
      exclusion_condition: date_day not in ('2019-12-25', '2020-12-25', '2021-12-25')

# or equivalently

tests:
  - dbt_expectations.expect_row_values_to_have_data_for_every_n_datepart:
      date_col: date_day
      date_part: day
      exclusion_condition: not(date_part(month, date_day) = 12 and date_part(day, date_day) = 25)

@gofford
Copy link
Contributor Author

gofford commented Jan 20, 2022

@clausherther in the absence of #139, I'm not 100% of the procedure in place for merging into main. E.g., does this need to incur a version bump, etc.?

Can you advise? I'll then update the remaining items (e.g., docs, version) as required.

@gofford gofford marked this pull request as ready for review January 20, 2022 10:29
@clausherther
Copy link
Contributor

Hi @gofford ! I'll take a more detail look on Monday, but this is great, thanks! Please also update README.md with documentation about the new parameter. I will deal with versioning and the CHANGELOG etc. once it's merged to main.

@clausherther
Copy link
Contributor

@gofford we'll also need an integration test for this parameter somewhere here, as a new instance of the test.

@gofford
Copy link
Contributor Author

gofford commented Jan 24, 2022

Will do @clausherther. I'll need to look into integration testing in more detail but I should get it turned around this week. Watch this space.

@gofford
Copy link
Contributor Author

gofford commented Feb 7, 2022

Apologies for the delay. I'll be sorting this out over the next few days.

agusfigueroa-htg and others added 21 commits February 24, 2022 12:26
* Create expect_column_set_to_be_unique_case_insensitive.sql

* adjusting consistent casing test and documentation

* adjusting formatting and adding integration test

* Fix formatting

* Formatting updates

* adjusting Readme and adding test instance

* Fix spelling

Co-authored-by: clausherther <claus@calogica.com>
* Added partition by parameter

* Added documentation

* Modified the test to accept list instead of string

* created data and implemented tests

* corrected test in yml

* Update formatting, test data and test macro

Co-authored-by: clausherther <claus@calogica.com>
* add test for column not to exist

* Update expect_table_columns_not_to_contain_set.sql

* Update README.md

* name change and inner joined

* update readme

* update test
* Add datetime and timestamp test columns

* Convert all evals to timestamps

* Add datatypes

* Update schema tests
* Add github workflow

* Remove pre-commit
* Add .circleci/config.yml

* Remove github action

* Update postgres test

* Add dbt install

* Add env variables

* Install dbt dependencies

* Empty postgres password

* Upgrade postgres image

* Update connection settings

* Fix postgres profile

* Move profiles

* Add BigQuery run

* Fix job name

* Update key path

* Update config

* Update schema

* Add Snowflake tests

* Update profile

* Add back postgres
…#158)

* Show group_by columns in validation errors for column increasing test

* Add github actions workflow (#159)

* Add github workflow

* Remove pre-commit

* Show group_by columns in validation errors for column increasing test

* Add .circleci/config.yml (#161)

* Add .circleci/config.yml

* Remove github action

* Update postgres test

* Add dbt install

* Add env variables

* Install dbt dependencies

* Empty postgres password

* Upgrade postgres image

* Update connection settings

* Fix postgres profile

* Move profiles

* Add BigQuery run

* Fix job name

* Update key path

* Update config

* Update schema

* Add Snowflake tests

* Update profile

* Add back postgres

* Show group_by columns in validation errors for column increasing test

* Jinja whitespace update (to force CI)

Co-authored-by: Claus Herther <claus@calogica.com>
@clausherther clausherther self-requested a review April 4, 2022 01:06
Copy link
Contributor

@clausherther clausherther left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, I added an integration test and ran this through our new CI pipeline. LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants