Support for Native PySpark #1185
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pyspark engine
updating equivalents to native types
implementing component and container changes for pyspark dataframes
Feature kill switch
change to consistent interface
removing dead code and refactoring as needed
adding setter for errors object in pyspark accessor
reformatting error dict
Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
|
hey @NeerajMalhotra-QB, so there're a considerable number of pylint and mypy errors. I'm going to silence them before merging into the On this front, I want to create an alpha release |
That sounds good to me. Please let me know when you merge it into branch so we can start pulling it and updating in future. Thanks @cosmicBboy |
Quick update: going to use the union-oss/pandera |
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## dev #1185 +/- ##
===========================================
- Coverage 97.23% 75.87% -21.37%
===========================================
Files 65 88 +23
Lines 5067 6674 +1607
===========================================
+ Hits 4927 5064 +137
- Misses 140 1610 +1470
☔ View full report in Codecov by Sentry. |
Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
|
okay, so there are more issues with the tests and linters, which I wasn't able to address @NeerajMalhotra-QB. I'll go ahead and merge this into the dev branch and cut an alpha release. You and the QB team can then iterate from there! |
|
Thanks @cosmicBboy , We will look into it. |
cosmicBboy
added a commit
that referenced
this pull request
Jul 6, 2023
* Support for Native PySpark (#1185) * init * init * init structure * disable imports' * adding structure for pyspark * setting dependency * update class import * fixing datatype cls * adding dtypes for pyspark * keep only bool type * remove pydantic schema * register pyspark data types * add check method * updating equivalents to native types * update column schema * refactor array to column * rename array to column_schema * remove pandas imports * remove index and multiindex functionality * adding pydantic schema class * adding model components * add model config * define pyspark BaseConfig class * removing index and multi-indexes * remove modify schema * Pyspark backend components, base, container, accessor, test file for accessor * Pyspark backend components, base, container, accessor, test file for accessor * Pyspark backend components, base, container, accessor, test file for accessor * Pyspark backend components, base, container, accessor, test file for accessor * add pyspark model components and types * remove hypothesis * remove synthesis and hypothesis * Pyspark backend components, base, container, accessor, test file for accessor * test for pyspark dataframeschema class * test schema with alias types * ensuring treat dataframes as tables types * update container for pyspark dataframe * adding negative test flow * removing series and index on pysparrk dataframes * remove series * revert series from pyspark.pandas * adding checks for pyspark * registering pysparkCheckBackend * cleaning base * Fixing the broken type cast check, validation of schema fix. * define spark level schema * fixing check flow * setting apply fn * add sub sample functionality * adjusting test case against common attributes * need apply for column level check * adding builtin checks for pyspark * adding checks for pyspark df * getting check registered * fixing a bug a in error handling for schema check * check_name validation fixed * implementing dtype checks for pyspark * updating error msg * fixing dtype reason_code * updating builtin checks for pyspark * registeration * Implementation of checks import and spark columns information check * enhancing __call__, checks classes and builtin_checks * delete junk files * Changes to fix the implemtation of checks. Changed Apply function to send list with dataframe and column name, builtin function registers functions with lists which inculdes the dataframe * extending pyspark checks * Fixed builtin check bug and added test for supported builtin checks for pyspark * add todos * bydefault validate all checks * fixing issue with sqlctx * add dtypes pytests * setting up schema * add negative and positive tests * add fixtures and refactor tests * generalize spark_df func * refactor to use conftest * use conftest * add support for decimal dtype and fixing other types * Added new Datatypes support for pyspark, test cases for dtypes pyspark, created test file for error * refactor ArraySchema * rename array to column.py * 1) Changes in test cases to look for summarised error raise instead of fast fail, since default behaviour is changed to summarised. 2) Added functionality to accept and check the precision and scale in Decimal Datatypes. * add neg test * add custom ErrorHandler * Added functionality to DayTimeIntervalType datatype to accept parameters * Added functionality to DayTimeIntervalType datatype to accept parameters * return summarized error report * replace dataframe to dict for return obj * Changed checks input datatype to custom named tuple from the existing list. Also started changing the pyspark checks to include more datatypes * refactor * introduce error categories * rename error categories * fixing bug in schema.dtype.check * fixing error category to by dynamic * Added checks for each datatype in test cases. Reduced the code redundancy of the code in test file. Refactored the name of custom datatype object for checks. * error_handler pass through * add ErrorHandler to column api * removed SchemaErrors since we now aggregate in errorHandler * fixing dict keys * Added Decorator to raise TypeError in case of unexpected input type for the check function. * replace validator with report_errors * cleaning debugs * Support DataModels and Field * Added Decorator to raise TypeError in case of unexpected input type for the check function. Merged with Develop * Fix to run using the class schema type * use alias types * clean up * add new typing for pyspark.sql * Added Decorator to raise TypeError in case of unexpected input type for the check function. Merged with Develop * Added changes to support raising error for use of datatype not supported by the check and support for map and array type. * support bare dtypes for DataFrameModel * remove resolved TODOs and breakpoints * change to bare types * use spark types instead of bare types * using SchemaErrorReason instead of hardcode in container * fixing an issue with error reason codes * minor fix * fixing checks and errors in pyspark * Changes include the following: 1) Updated dtypes test functionality to make it more readable 2) Changes in accessor tests to support the new functionality 3) Changes in engine class to conform to check class everywhere else * enhancing dataframeschema and model classes * Changes to remove the pandas dependency * Refactoring of the checks test functions * Fixing the test case breaking * Isort and Black formatting * Container Test function failure * Isort and black linting * Changes to remove the pandas dependency * Refactoring of the checks test functions * Isort and black linting * Added Changes to refactor the checks class. Fixes to some test cases failures. * Removing breakpoint * fixing raise error * adding metadata dict * Removing the reference of pandas from docstrings * Removing redundant code block in utils * Changes to return dataframe with errors property * add accessor for errorHandler * support errors access on pyspark.sql * updating pyspark error tcs * fixing model test cases * adjusting errors to use pandera.errors * use accessor instead of dict * revert to develop * Removal of imports which are not needed and improved test case. * setting independent pyspark import * pyspark imports * revert comments * store and retrieve metadata at schema levels * adding metadata support * Added changes to support parameter based run. 1) Added parameters.yaml file to hold the configurations 2) Added code in utility to read the config 3) Updated the test cases to support the parameter based run 4) Moved pyspark decorators to a new file decorators.py in backend 5) Type fix in get_matadata property in container.py file * Changing the default value in config * change to consistent interface * cleaning api/pyspark * backend and tests * adding setter on errors accessors for pyspark * reformatting error dict * doc * run black linter Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fix lint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * update pylintrc Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Co-authored-by: jaskaransinghsidana <jaskaran_singh_sidana@mckinsey.com> Co-authored-by: jaskaransinghsidana <112083212+jaskaransinghsidana@users.noreply.github.com> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> * Support native PySpark.sql on Pandera (#1213) * fixing check flow * setting apply fn * add sub sample functionality * adjusting test case against common attributes * need apply for column level check * adding builtin checks for pyspark * adding checks for pyspark df * getting check registered * fixing a bug a in error handling for schema check * check_name validation fixed * implementing dtype checks for pyspark * updating error msg * fixing dtype reason_code * updating builtin checks for pyspark * registeration * Implementation of checks import and spark columns information check * enhancing __call__, checks classes and builtin_checks * delete junk files * Changes to fix the implemtation of checks. Changed Apply function to send list with dataframe and column name, builtin function registers functions with lists which inculdes the dataframe * extending pyspark checks * Fixed builtin check bug and added test for supported builtin checks for pyspark * add todos * bydefault validate all checks * fixing issue with sqlctx * add dtypes pytests * setting up schema * add negative and positive tests * add fixtures and refactor tests * generalize spark_df func * refactor to use conftest * use conftest * add support for decimal dtype and fixing other types * Added new Datatypes support for pyspark, test cases for dtypes pyspark, created test file for error * refactor ArraySchema * rename array to column.py * 1) Changes in test cases to look for summarised error raise instead of fast fail, since default behaviour is changed to summarised. 2) Added functionality to accept and check the precision and scale in Decimal Datatypes. * add neg test * add custom ErrorHandler * Added functionality to DayTimeIntervalType datatype to accept parameters * Added functionality to DayTimeIntervalType datatype to accept parameters * return summarized error report * replace dataframe to dict for return obj * Changed checks input datatype to custom named tuple from the existing list. Also started changing the pyspark checks to include more datatypes * refactor * introduce error categories * rename error categories * fixing bug in schema.dtype.check * fixing error category to by dynamic * Added checks for each datatype in test cases. Reduced the code redundancy of the code in test file. Refactored the name of custom datatype object for checks. * error_handler pass through * add ErrorHandler to column api * removed SchemaErrors since we now aggregate in errorHandler * fixing dict keys * Added Decorator to raise TypeError in case of unexpected input type for the check function. * replace validator with report_errors * cleaning debugs * Support DataModels and Field * Added Decorator to raise TypeError in case of unexpected input type for the check function. Merged with Develop * Fix to run using the class schema type * use alias types * clean up * add new typing for pyspark.sql * Added Decorator to raise TypeError in case of unexpected input type for the check function. Merged with Develop * Added changes to support raising error for use of datatype not supported by the check and support for map and array type. * support bare dtypes for DataFrameModel * remove resolved TODOs and breakpoints * change to bare types * use spark types instead of bare types * using SchemaErrorReason instead of hardcode in container * fixing an issue with error reason codes * minor fix * fixing checks and errors in pyspark * Changes include the following: 1) Updated dtypes test functionality to make it more readable 2) Changes in accessor tests to support the new functionality 3) Changes in engine class to conform to check class everywhere else * enhancing dataframeschema and model classes * Changes to remove the pandas dependency * Refactoring of the checks test functions * Fixing the test case breaking * Isort and Black formatting * Container Test function failure * Isort and black linting * Changes to remove the pandas dependency * Refactoring of the checks test functions * Isort and black linting * Added Changes to refactor the checks class. Fixes to some test cases failures. * Removing breakpoint * fixing raise error * adding metadata dict * Removing the reference of pandas from docstrings * Removing redundant code block in utils * Changes to return dataframe with errors property * add accessor for errorHandler * support errors access on pyspark.sql * updating pyspark error tcs * fixing model test cases * adjusting errors to use pandera.errors * use accessor instead of dict * revert to develop * Removal of imports which are not needed and improved test case. * setting independent pyspark import * pyspark imports * revert comments * store and retrieve metadata at schema levels * adding metadata support * Added changes to support parameter based run. 1) Added parameters.yaml file to hold the configurations 2) Added code in utility to read the config 3) Updated the test cases to support the parameter based run 4) Moved pyspark decorators to a new file decorators.py in backend 5) Type fix in get_matadata property in container.py file * Changing the default value in config * change to consistent interface * Changes to remove config yaml and introduce environment variables for parameterized runs * cleaning api/pyspark * backend and tests * adding setter on errors accessors for pyspark * reformatting error dict * Changes to remove config yaml and introduce environment variables for parameterized runs * Changes to rename the config object and call only in utils.py * Fixing merge conflict issue * Updating the test cases to support new checks types * Added individualized test for each configuration type. * Removing unnecessary prints * The changes include the following: 1) Fixed test case for validating the environment variable 2) Improved docstrings for test cases and few test cases asserts * Fix reference to with wrong key in test_pyspark_schema_data_checks * minor change * Added Support for docstring substitution method. * Removing an extra indent * Removing commented docstring substitution from __new__ method * remove union * cleaning * Feature to add metadata dictionary for pandas schema * Added test to check the docstring substitution decorator * Added test to check the docstring substitution decorator * Feature to add metadata dictionary for pandas schema * Changes to ensure only pandas run does not import pyspark dependencies * Fix of imports for pandas and pyspark for separation * Rename the function from pyspark to pandas * black lint and isort * black lint and isort * Fixes of pyliny issue and suppression wherever necessary * Fixes of mypy failures and redone black linting post changes. * Added new test cases, removed redundant codes and black lint. * Fixed the doc strings, added functionality and test for custom checks * add rst for pyspark.sql * removing rst * Renamed check name and Fixed pylint and mypy issues * add rst for pyspark.sql * Fixed the doc strings, added functionality and test for custom checks * removing rst * Renamed check name and Fixed pylint and mypy issues * add rst for pyspark.sql * Rename for environment variable key name * removing rst * Black lint * Removed daytime interval type * refactor * override pyspark patching of __class_getitem__ Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fixiing mypy error * lint fixes * lint fixes * fixing more lint and type issues * fixing mypy issues * fixing doctest * doctest Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * fixing doctest Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * adding doctest:metadata for pandas container classes Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * doctest * fixing doctest Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * fixing rst Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * black formatting Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * fixing str repr for DataFrameSchema across rst * add ps.DataFrame * fixing tests * fix lint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * use full class name in pandas accessor Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * use os.environ instead of parameters.yaml Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * simplify config Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> Co-authored-by: jaskaransinghsidana <jaskaran_singh_sidana@mckinsey.com> Co-authored-by: jaskaransinghsidana <112083212+jaskaransinghsidana@users.noreply.github.com> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> * support multiple pyspark versions (#1221) support pyspark 3.2 and 3.3 for string representation of dtypes. --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> Co-authored-by: jaskaransinghsidana <jaskaran_singh_sidana@mckinsey.com> Co-authored-by: jaskaransinghsidana <112083212+jaskaransinghsidana@users.noreply.github.com> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> * Refactors for dead code (#1229) * updating builtin checks for pyspark * registeration * Implementation of checks import and spark columns information check * enhancing __call__, checks classes and builtin_checks * delete junk files * Changes to fix the implemtation of checks. Changed Apply function to send list with dataframe and column name, builtin function registers functions with lists which inculdes the dataframe * extending pyspark checks * Fixed builtin check bug and added test for supported builtin checks for pyspark * add todos * bydefault validate all checks * fixing issue with sqlctx * add dtypes pytests * setting up schema * add negative and positive tests * add fixtures and refactor tests * generalize spark_df func * refactor to use conftest * use conftest * add support for decimal dtype and fixing other types * Added new Datatypes support for pyspark, test cases for dtypes pyspark, created test file for error * refactor ArraySchema * rename array to column.py * 1) Changes in test cases to look for summarised error raise instead of fast fail, since default behaviour is changed to summarised. 2) Added functionality to accept and check the precision and scale in Decimal Datatypes. * add neg test * add custom ErrorHandler * Added functionality to DayTimeIntervalType datatype to accept parameters * Added functionality to DayTimeIntervalType datatype to accept parameters * return summarized error report * replace dataframe to dict for return obj * Changed checks input datatype to custom named tuple from the existing list. Also started changing the pyspark checks to include more datatypes * refactor * introduce error categories * rename error categories * fixing bug in schema.dtype.check * fixing error category to by dynamic * Added checks for each datatype in test cases. Reduced the code redundancy of the code in test file. Refactored the name of custom datatype object for checks. * error_handler pass through * add ErrorHandler to column api * removed SchemaErrors since we now aggregate in errorHandler * fixing dict keys * Added Decorator to raise TypeError in case of unexpected input type for the check function. * replace validator with report_errors * cleaning debugs * Support DataModels and Field * Added Decorator to raise TypeError in case of unexpected input type for the check function. Merged with Develop * Fix to run using the class schema type * use alias types * clean up * add new typing for pyspark.sql * Added Decorator to raise TypeError in case of unexpected input type for the check function. Merged with Develop * Added changes to support raising error for use of datatype not supported by the check and support for map and array type. * support bare dtypes for DataFrameModel * remove resolved TODOs and breakpoints * change to bare types * use spark types instead of bare types * using SchemaErrorReason instead of hardcode in container * fixing an issue with error reason codes * minor fix * fixing checks and errors in pyspark * Changes include the following: 1) Updated dtypes test functionality to make it more readable 2) Changes in accessor tests to support the new functionality 3) Changes in engine class to conform to check class everywhere else * enhancing dataframeschema and model classes * Changes to remove the pandas dependency * Refactoring of the checks test functions * Fixing the test case breaking * Isort and Black formatting * Container Test function failure * Isort and black linting * Changes to remove the pandas dependency * Refactoring of the checks test functions * Isort and black linting * Added Changes to refactor the checks class. Fixes to some test cases failures. * Removing breakpoint * fixing raise error * adding metadata dict * Removing the reference of pandas from docstrings * Removing redundant code block in utils * Changes to return dataframe with errors property * add accessor for errorHandler * support errors access on pyspark.sql * updating pyspark error tcs * fixing model test cases * adjusting errors to use pandera.errors * use accessor instead of dict * revert to develop * Removal of imports which are not needed and improved test case. * setting independent pyspark import * pyspark imports * revert comments * store and retrieve metadata at schema levels * adding metadata support * Added changes to support parameter based run. 1) Added parameters.yaml file to hold the configurations 2) Added code in utility to read the config 3) Updated the test cases to support the parameter based run 4) Moved pyspark decorators to a new file decorators.py in backend 5) Type fix in get_matadata property in container.py file * Changing the default value in config * change to consistent interface * Changes to remove config yaml and introduce environment variables for parameterized runs * cleaning api/pyspark * backend and tests * adding setter on errors accessors for pyspark * reformatting error dict * Changes to remove config yaml and introduce environment variables for parameterized runs * Changes to rename the config object and call only in utils.py * Fixing merge conflict issue * Updating the test cases to support new checks types * Added individualized test for each configuration type. * Removing unnecessary prints * The changes include the following: 1) Fixed test case for validating the environment variable 2) Improved docstrings for test cases and few test cases asserts * Fix reference to with wrong key in test_pyspark_schema_data_checks * minor change * Added Support for docstring substitution method. * Removing an extra indent * Removing commented docstring substitution from __new__ method * remove union * cleaning * Feature to add metadata dictionary for pandas schema * Added test to check the docstring substitution decorator * Added test to check the docstring substitution decorator * Feature to add metadata dictionary for pandas schema * Changes to ensure only pandas run does not import pyspark dependencies * Fix of imports for pandas and pyspark for separation * Rename the function from pyspark to pandas * black lint and isort * black lint and isort * Fixes of pyliny issue and suppression wherever necessary * Fixes of mypy failures and redone black linting post changes. * Added new test cases, removed redundant codes and black lint. * Fixed the doc strings, added functionality and test for custom checks * add rst for pyspark.sql * removing rst * Renamed check name and Fixed pylint and mypy issues * add rst for pyspark.sql * Fixed the doc strings, added functionality and test for custom checks * removing rst * Renamed check name and Fixed pylint and mypy issues * add rst for pyspark.sql * Rename for environment variable key name * removing rst * Black lint * Removed daytime interval type * refactor * override pyspark patching of __class_getitem__ Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fixiing mypy error * lint fixes * lint fixes * fixing more lint and type issues * fixing mypy issues * fixing doctest * doctest Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * fixing doctest Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * adding doctest:metadata for pandas container classes Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * doctest * Fix to support pyspark 3.2 and 3.3 both. The string representation of datatype changed in 3.2 and 3.3. Fix ensure both versions are supported. * Black Lint * fixing doctest Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * fixing rst Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * black formatting Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * fixing str repr for DataFrameSchema across rst * add ps.DataFrame * fixing tests * fix lint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * use full class name in pandas accessor Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * use os.environ instead of parameters.yaml Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * simplify config Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * merge with develop * Black Lint * refactor Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * lint fix Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * refactor * remove Column class due to redundancy Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * linting Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> Co-authored-by: jaskaransinghsidana <jaskaran_singh_sidana@mckinsey.com> Co-authored-by: jaskaransinghsidana <112083212+jaskaransinghsidana@users.noreply.github.com> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> * Adding readme for pyspark_sql enhancements (#1236) * add dtypes pytests * setting up schema * add negative and positive tests * add fixtures and refactor tests * generalize spark_df func * refactor to use conftest * use conftest * add support for decimal dtype and fixing other types * Added new Datatypes support for pyspark, test cases for dtypes pyspark, created test file for error * refactor ArraySchema * rename array to column.py * 1) Changes in test cases to look for summarised error raise instead of fast fail, since default behaviour is changed to summarised. 2) Added functionality to accept and check the precision and scale in Decimal Datatypes. * add neg test * add custom ErrorHandler * Added functionality to DayTimeIntervalType datatype to accept parameters * Added functionality to DayTimeIntervalType datatype to accept parameters * return summarized error report * replace dataframe to dict for return obj * Changed checks input datatype to custom named tuple from the existing list. Also started changing the pyspark checks to include more datatypes * refactor * introduce error categories * rename error categories * fixing bug in schema.dtype.check * fixing error category to by dynamic * Added checks for each datatype in test cases. Reduced the code redundancy of the code in test file. Refactored the name of custom datatype object for checks. * error_handler pass through * add ErrorHandler to column api * removed SchemaErrors since we now aggregate in errorHandler * fixing dict keys * Added Decorator to raise TypeError in case of unexpected input type for the check function. * replace validator with report_errors * cleaning debugs * Support DataModels and Field * Added Decorator to raise TypeError in case of unexpected input type for the check function. Merged with Develop * Fix to run using the class schema type * use alias types * clean up * add new typing for pyspark.sql * Added Decorator to raise TypeError in case of unexpected input type for the check function. Merged with Develop * Added changes to support raising error for use of datatype not supported by the check and support for map and array type. * support bare dtypes for DataFrameModel * remove resolved TODOs and breakpoints * change to bare types * use spark types instead of bare types * using SchemaErrorReason instead of hardcode in container * fixing an issue with error reason codes * minor fix * fixing checks and errors in pyspark * Changes include the following: 1) Updated dtypes test functionality to make it more readable 2) Changes in accessor tests to support the new functionality 3) Changes in engine class to conform to check class everywhere else * enhancing dataframeschema and model classes * Changes to remove the pandas dependency * Refactoring of the checks test functions * Fixing the test case breaking * Isort and Black formatting * Container Test function failure * Isort and black linting * Changes to remove the pandas dependency * Refactoring of the checks test functions * Isort and black linting * Added Changes to refactor the checks class. Fixes to some test cases failures. * Removing breakpoint * fixing raise error * adding metadata dict * Removing the reference of pandas from docstrings * Removing redundant code block in utils * Changes to return dataframe with errors property * add accessor for errorHandler * support errors access on pyspark.sql * updating pyspark error tcs * fixing model test cases * adjusting errors to use pandera.errors * use accessor instead of dict * revert to develop * Removal of imports which are not needed and improved test case. * setting independent pyspark import * pyspark imports * revert comments * store and retrieve metadata at schema levels * adding metadata support * Added changes to support parameter based run. 1) Added parameters.yaml file to hold the configurations 2) Added code in utility to read the config 3) Updated the test cases to support the parameter based run 4) Moved pyspark decorators to a new file decorators.py in backend 5) Type fix in get_matadata property in container.py file * Changing the default value in config * change to consistent interface * Changes to remove config yaml and introduce environment variables for parameterized runs * cleaning api/pyspark * backend and tests * adding setter on errors accessors for pyspark * reformatting error dict * Changes to remove config yaml and introduce environment variables for parameterized runs * Changes to rename the config object and call only in utils.py * Fixing merge conflict issue * Updating the test cases to support new checks types * Added individualized test for each configuration type. * Removing unnecessary prints * The changes include the following: 1) Fixed test case for validating the environment variable 2) Improved docstrings for test cases and few test cases asserts * Fix reference to with wrong key in test_pyspark_schema_data_checks * minor change * Added Support for docstring substitution method. * Removing an extra indent * Removing commented docstring substitution from __new__ method * remove union * cleaning * Feature to add metadata dictionary for pandas schema * Added test to check the docstring substitution decorator * Added test to check the docstring substitution decorator * Feature to add metadata dictionary for pandas schema * Changes to ensure only pandas run does not import pyspark dependencies * Fix of imports for pandas and pyspark for separation * Rename the function from pyspark to pandas * black lint and isort * black lint and isort * Fixes of pyliny issue and suppression wherever necessary * Fixes of mypy failures and redone black linting post changes. * Added new test cases, removed redundant codes and black lint. * Fixed the doc strings, added functionality and test for custom checks * add rst for pyspark.sql * removing rst * Renamed check name and Fixed pylint and mypy issues * add rst for pyspark.sql * Fixed the doc strings, added functionality and test for custom checks * removing rst * Renamed check name and Fixed pylint and mypy issues * add rst for pyspark.sql * Rename for environment variable key name * removing rst * Black lint * Removed daytime interval type * refactor * override pyspark patching of __class_getitem__ Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * fixiing mypy error * lint fixes * lint fixes * fixing more lint and type issues * fixing mypy issues * fixing doctest * doctest Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * fixing doctest Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * adding doctest:metadata for pandas container classes Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * doctest * Fix to support pyspark 3.2 and 3.3 both. The string representation of datatype changed in 3.2 and 3.3. Fix ensure both versions are supported. * Black Lint * fixing doctest Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * fixing rst Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * black formatting Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * fixing str repr for DataFrameSchema across rst * add ps.DataFrame * fixing tests * fix lint Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * use full class name in pandas accessor Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * use os.environ instead of parameters.yaml Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * simplify config Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * merge with develop * Black Lint * refactor Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * lint fix Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * readme * refine example * about granular controls * native pyspark.sql documentation Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * docs Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * Add to index * add to index * Delete oryx-build-commands.txt * fix index * revert * fix index * Update pyspark_sql.rst * Update pyspark_sql.rst * update error print * Update pyspark_sql.rst * Update pyspark_sql.rst * adding import in code block Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> * Update pyspark_sql.rst * Update pyspark_sql.rst * Update pyspark_sql.rst * clean up docs Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> Co-authored-by: jaskaransinghsidana <jaskaran_singh_sidana@mckinsey.com> Co-authored-by: jaskaransinghsidana <112083212+jaskaransinghsidana@users.noreply.github.com> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> * fix metadata arg in SeriesSchema Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> * improve coverage * improve coverage in extensions --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: Neeraj Malhotra <neeraj_malhotra@mckinsey.com> Co-authored-by: Neeraj Malhotra <52220398+NeerajMalhotra-QB@users.noreply.github.com> Co-authored-by: jaskaransinghsidana <jaskaran_singh_sidana@mckinsey.com> Co-authored-by: jaskaransinghsidana <112083212+jaskaransinghsidana@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces several enhancements and new features to support native PySpark on Pandera. The key changes implemented include:
SchemaOnly,DataOnlyorSchema_And_DataThese changes significantly enhance the capabilities of Pandera, empowering users leverage Pandera on native PySpark applications.