Support Spark 2.3

[hangelwen]

Hi Deequ team,

This is a pretty useful tool. However, it only supports Spark 2.2. Is there a plan to support Spark 2.3 version?

[sscdotopen]

Hi,

We haven't done extensive testing, but deequ should also work with Spark 2.3, as Spark is guaranteeing stability of its non-experimental APIs for all 2.X releases..

[hangelwen]

@sscdotopen thanks for the quick reply.

I tried and it does not work on Spark 2.3. One issue I found it that some variables defined for HyperloglogPlus was moved from HyperLogLogPlusPlus to HyperLogLogPlusPlusHelper in this PR

https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/analyzers/catalyst/StatefulHyperloglogPlus.scala#L195
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/HyperLogLogPlusPlusHelper.scala#L280

[sscdotopen]

Thank you for the hint, I did not think about the fact that we also extended some internal functions. I'll file this issue as a bug on our side, and we will have a look into that.

[sscdotopen]

@hangelwen We think we have fixed the incompatibility, we will release a new RC later today.