Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data validation Phase 1 #1239

Merged
merged 36 commits into from
Dec 20, 2022
Merged

Data validation Phase 1 #1239

merged 36 commits into from
Dec 20, 2022

Commits on Dec 20, 2022

  1. Saving work

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    ddb8ea4 View commit details
    Browse the repository at this point in the history

  2. Added test case

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    8996d25 View commit details
    Browse the repository at this point in the history

  3. Add testcases for ColumnCheckOperator

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    595bd02 View commit details
    Browse the repository at this point in the history

  4. Update data types supported by ColumnCheckOperator

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    8640e5e View commit details
    Browse the repository at this point in the history

  5. Add task_id to operator

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    d335c3c View commit details
    Browse the repository at this point in the history

  6. Add testcase for table dataset

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    43d67bd View commit details
    Browse the repository at this point in the history

  7. Add doc string to functions

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    cce4567 View commit details
    Browse the repository at this point in the history

  8. Moved the test_ColumnCheckOperator.py to data_validation.py/test_Colu… 

    …mnCheckOperator.py
    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    488a668 View commit details
    Browse the repository at this point in the history

  9. Add SQLCheckOperator to validate tables via sql

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    518cad3 View commit details
    Browse the repository at this point in the history

  10. Update python-sdk/src/astro/sql/operators/data_validations/SQLCheckOp… 

    …erator.py

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
    @utkarsharma2 @feluelle
    utkarsharma2 and feluelle committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    57a5a00 View commit details
    Browse the repository at this point in the history

  11. Update python-sdk/src/astro/sql/operators/data_validations/SQLCheckOp… 

    …erator.py

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
    @utkarsharma2 @feluelle
    utkarsharma2 and feluelle committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    d2f4dcf View commit details
    Browse the repository at this point in the history

  12. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci @utkarsharma2
    pre-commit-ci[bot] authored and utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    f0140c0 View commit details
    Browse the repository at this point in the history

  13. Update the dataframe check method

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    a2be757 View commit details
    Browse the repository at this point in the history

  14. Change test execution method to run_dag() from operator.execute()

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    016461b View commit details
    Browse the repository at this point in the history

  15. Override GoogleBaseHook with BigqueryHook (#1442) 

    # Description
## What is the current behavior?
Because of below issue:
```
airflow.exceptions.AirflowException: You are trying to use `common-sql` with GoogleBaseHook, but its provider does not support it. Please upgrade the provider to a version that supports `common-sql`. The hook class should be a subclass of `airflow.providers.common.sql.hooks.sql.DbApiHook`. Got GoogleBaseHook Hook with class hierarchy: [<class 'airflow.providers.google.common.hooks.base_google.GoogleBaseHook'>, <class 'airflow.hooks.base.BaseHook'>, <class 'airflow.utils.log.logging_mixin.LoggingMixin'>, <class 'object'>]
```
We are using a work around and using Bigquey Hook
    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    1220883 View commit details
    Browse the repository at this point in the history

  16. Revoke region changes

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    3db2eab View commit details
    Browse the repository at this point in the history

  17. Add apache-airflow-providers-common-sql as dependency

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    12a9334 View commit details
    Browse the repository at this point in the history

  18. Remove unwanted code

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    bc037cb View commit details
    Browse the repository at this point in the history

  19. Remove unwanted codes

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    04d527e View commit details
    Browse the repository at this point in the history

  20. Add location

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    52e3821 View commit details
    Browse the repository at this point in the history

  21. Add google_cloud_platform connection

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    a18e8d8 View commit details
    Browse the repository at this point in the history

  22. Update conn_id

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    3905153 View commit details
    Browse the repository at this point in the history

  23. Change return type

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    a0e6773 View commit details
    Browse the repository at this point in the history

  24. Updated hook

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    686a09d View commit details
    Browse the repository at this point in the history

  25. Update python-sdk/src/astro/sql/operators/data_validations/ColumnChec… 

    …kOperator.py

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
    @utkarsharma2 @feluelle
    utkarsharma2 and feluelle committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    2b91128 View commit details
    Browse the repository at this point in the history

  26. Update python-sdk/src/astro/sql/operators/data_validations/ColumnChec… 

    …kOperator.py

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
    @utkarsharma2 @feluelle
    utkarsharma2 and feluelle committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    ec3b4fd View commit details
    Browse the repository at this point in the history

  27. Update python-sdk/src/astro/sql/operators/data_validations/ColumnChec… 

    …kOperator.py

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
    @utkarsharma2 @feluelle
    utkarsharma2 and feluelle committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    cd828d7 View commit details
    Browse the repository at this point in the history

  28. Update python-sdk/src/astro/sql/operators/data_validations/ColumnChec… 

    …kOperator.py

Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
    @utkarsharma2 @feluelle
    utkarsharma2 and feluelle committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    d1d5c28 View commit details
    Browse the repository at this point in the history

  29. Revert changes to check since nunique() don't count None

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    a0543ad View commit details
    Browse the repository at this point in the history

  30. Fix Deep Source

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    764e08a View commit details
    Browse the repository at this point in the history

  31. Merge branch 'main' into DataValidation

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    c25a736 View commit details
    Browse the repository at this point in the history

  32. Refactored code to remove duplication

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    9673af7 View commit details
    Browse the repository at this point in the history

  33. Refactored code

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    5864a07 View commit details
    Browse the repository at this point in the history

  34. Code refactor

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    493d651 View commit details
    Browse the repository at this point in the history

  35. Refactored ColumnCheckOperator operator

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    8c71f39 View commit details
    Browse the repository at this point in the history

  36. Merge branch 'main' into DataValidation

    @utkarsharma2
    utkarsharma2 committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    ec40143 View commit details
    Browse the repository at this point in the history