2.0.6

What's Changed

• NEW: Exact Quantile Check
  • Creation of Exact Quantile Check by @jmilis2000 in #512
• Data Synchronization/Matching fixes
  • Delegate to Spark for checking existence of columns in the given dataframes by @rdsharma26 in #515
  • Verify that non key columns exist in each dataset by @rdsharma26 in #517
• Addition of tests
  • Test that exceptions within a check's constraints do not affect other… by @tylermcdaniel0 in #516

New Contributors

• @jmilis2000 made their first contribution in #512
• @tylermcdaniel0 made their first contribution in #516

Full Changelog: 2.0.5...2.0.6

2.0.5

What's Changed

• Spark 3.4 Update
  • Add Spark 3.4 support by @jhchee in #505
  • Update minor version for Spark 3.4 maven release by @eycho-am in #513
• NEW: Custom SQL analyzer
  • Custom SQL Analyzer by @mentekid in #509
  • Fail when CustomSql has syntax errors by @mentekid in #510
  • Fix CustomSQL test syntax by @eycho-am in #511
• Analyzer Improvements
  • Allow all DQ constraints to be generated from an Analyzer by @mentekid in #508

New Contributors

• @jhchee made their first contribution in #505

Full Changelog: 2.0.4...2.0.5

2.0.4

What's Changed

• Row-Level Results:
  • MinLength by @eycho-am in #465
  • Uniqueness by @eycho-am in #471
  • ColumnValues by @zixianzh1 in #476
  • ReferentialIntegrity by @rdsharma26 in #466
  • [Experimental] DataSynchronization by @rdsharma26 in #473
• Referential Integrity:
  • Updated Referential Integrity to support multiple columns by @rdsharma26 in #463
• Constraints and Condition Changes:
  • Add population stability index (PSI) to distance methods by @bevhanno in #480
  • Fix chi-square test conditions by @bevhanno in #482
  • Missing Column Precondition for Compliance Check - issue fix 467 by @samarth-c1 in #478
  • Addition of HasMax/HasMin/HasStandardDeviation/HasMean constraint suggestions by @rdsharma26 in #489
  • Alternative aggregate functions to calculate histogram values. by @akalotkin in #475

New Contributors

• @zixianzh1 made their first contribution in #476
• @samarth-c1 made their first contribution in #478
• @akalotkin made their first contribution in #475

Full Changelog: 2.0.3...2.0.4

2.0.3

What's Changed

• Adding chi-square distance method for categorical variables by @bevhanno in #444
• [WIP] Row Level Results by @mentekid in #451
• [Experimental] Addition of dataset comparison utilities by @rdsharma26 in #449

New Contributors

• @rdsharma26 made their first contribution in #447
• @bevhanno made their first contribution in #444
• @mentekid made their first contribution in #451

Full Changelog: 2.0.2...2.0.3

2.0.2

Adds Spark 3.3 compatibility.

What's Changed

• Upgrade to Spark 3.3.0 by @eycho-am in #442

New Contributors

• @eycho-am made their first contribution in #442

Full Changelog: 2.0.1...2.0.2

2.0.1

Adds Spark 3.2 compatibility.

2.0.0

Add Spark 3.1 compatibility.

Note: this version is no longer compatible with Spark <=3.0. Use previous versions and branch legacy-spark-3.0 instead.

Fix build setup to make artefact importable with maven/sbt

This release updates the build setup (i.e. the pom.xml and the publishing process) so that the artefacts published to maven can now be imported using maven or sbt. There are four branches associated with this new release:

• for spark 2.2: https://github.com/awslabs/deequ/tree/release/1.2.2-spark-2.2
• for spark 2.3: https://github.com/awslabs/deequ/tree/release/1.2.2-spark-2.3
• for spark 2.4: https://github.com/awslabs/deequ/tree/release/1.2.2-spark-2.4
• for spark 2.5: https://github.com/awslabs/deequ/tree/release/1.2.2-spark-2.5

1.1.0

Changes to the build setup to support Spark 2.2.x to 2.4.x and 3.0.x. There now is one maven release available per Spark version:

• spark-3.0-scala-2.12
• spark-2.4-scala-2.11
• spark-2.3-scala-2.11
• spark-2.2-scala-2.11

1.0.5

Upgrades Spark version from 2.2.2 to 2.3.3 (#291)