Skip to content

Releases: awslabs/deequ

2.0.7

02 Jul 17:47
Compare
Choose a tag to compare

What's Changed

Upgrades

New Features

  • New type of MetricsRepository by @VenkataKarthikP:
    • Using Spark tables as the data source in #518
  • Row Level Result Treatment Options by @eycho-am:
    • Uniqueness and Completeness in #532
    • Miminum and Maximum in #535
  • Anomaly Detection Changes by @zeotuan:
    • Add Daily Season with Hourly Interval to HoltWinter in #546
  • New analyzers:

Maintenance/Fixes

  • Fix Breeze dependency conflict in Anomaly Detection Spark 3.4+ by @zeotuan in #545
  • Data Sync / DatasetMatch changes by @VenkataKarthikP:
    • add data synchronization test to verification Suite in #526
    • support col match and change to DatasetMatch in #529
  • Row level results fixes:
    • Add analyzerOption to add filteredRowOutcome for isPrimaryKey Check by @eycho-am in #537
    • Fix bug in MinLength and MaxLength when NullBehavior.EmptyString by @eycho-am in #538
    • [Min/Max] Apply filtered row behavior at the row level evaluation by @rdsharma26 in #543
    • [MinLength/MaxLength] Apply filtered row behavior at the row level evaluation by @rdsharma26 in #547
    • Fix for satisfies row level results bug by @rdsharma26 in #553

New Contributors

Full Changelog: 2.0.6...2.0.7

2.0.6

13 Nov 17:16
54c5e48
Compare
Choose a tag to compare

What's Changed

  • NEW: Exact Quantile Check
  • Data Synchronization/Matching fixes
    • Delegate to Spark for checking existence of columns in the given dataframes by @rdsharma26 in #515
    • Verify that non key columns exist in each dataset by @rdsharma26 in #517
  • Addition of tests
    • Test that exceptions within a check's constraints do not affect other… by @tylermcdaniel0 in #516

New Contributors

Full Changelog: 2.0.5...2.0.6

2.0.5

13 Nov 17:13
94821c2
Compare
Choose a tag to compare

What's Changed

  • Spark 3.4 Update
  • NEW: Custom SQL analyzer
  • Analyzer Improvements
    • Allow all DQ constraints to be generated from an Analyzer by @mentekid in #508

New Contributors

Full Changelog: 2.0.4...2.0.5

2.0.4

10 Aug 17:18
Compare
Choose a tag to compare

What's Changed

New Contributors

Full Changelog: 2.0.3...2.0.4

2.0.3

07 Mar 20:10
c9a0eae
Compare
Choose a tag to compare

What's Changed

  • Adding chi-square distance method for categorical variables by @bevhanno in #444
  • [WIP] Row Level Results by @mentekid in #451
  • [Experimental] Addition of dataset comparison utilities by @rdsharma26 in #449

New Contributors

Full Changelog: 2.0.2...2.0.3

2.0.2

30 Jan 20:02
2dc9c0a
Compare
Choose a tag to compare

Adds Spark 3.3 compatibility.

What's Changed

New Contributors

Full Changelog: 2.0.1...2.0.2

2.0.1

16 Feb 08:49
Compare
Choose a tag to compare

Adds Spark 3.2 compatibility.

2.0.0

05 Aug 16:09
Compare
Choose a tag to compare

Add Spark 3.1 compatibility.

Note: this version is no longer compatible with Spark <=3.0. Use previous versions and branch legacy-spark-3.0 instead.

Fix build setup to make artefact importable with maven/sbt

03 May 07:36
7bbb7fa
Compare
Choose a tag to compare

This release updates the build setup (i.e. the pom.xml and the publishing process) so that the artefacts published to maven can now be imported using maven or sbt. There are four branches associated with this new release:

1.1.0

17 Dec 15:58
Compare
Choose a tag to compare

Changes to the build setup to support Spark 2.2.x to 2.4.x and 3.0.x. There now is one maven release available per Spark version:

  • spark-3.0-scala-2.12
  • spark-2.4-scala-2.11
  • spark-2.3-scala-2.11
  • spark-2.2-scala-2.11