Skip to content

v0.4.0

Choose a tag to compare

@github-actions github-actions released this 08 Apr 22:14
· 275 commits to main since this release
d0ffb7a
  • Added input spark options and schema for reading from the storage (#312). This commit enhances the data quality framework used for profiling and validating data in a Databricks workspace with new options and functionality for reading data from storage. It allows for the usage of input spark options and schema, and supports fully qualified Unity Catalog or Hive Metastore table names in the format of catalog.schema.table or schema.table. Additionally, the code now includes a new dataclass field, input_schema, and a new dictionary field, input_read_options, to the RunConfig class. The documentation is updated with examples of how to use the new functionality.
  • Added an example of uniqueness check for composite key (#312). Additionally, the code now includes a new dataclass field, input_schema, and a new dictionary field, input_read_options, to the RunConfig class. The documentation is updated with examples of how to use the new functionality.
  • Renamed row checks module for more clarity (#314). This change renames the col_check_functions module to row_checks for clarity and to distinguish it from other types of checks. The import * syntax is removed and unused imports are removed from the demo. This change requires updating import statements that reference col_check_functions to use the new name row_checks. Checks defined using DQX classes require a simple update.

Full Changelog: v0.3.1...v0.4.0