Spark 3.0

[pokpetr]

Would it be possible to add support for Spark 3.0 ? We upgrading our project to Spark 3.0.1 and Atum fails with the following error on disableControlMeasuresTracking even though enableControlMeasuresTracking was called before.

Control framework tracking is not initialized.
java.lang.IllegalStateException: Control framework tracking is not initialized.
    at za.co.absa.atum.core.Atum$.preventNotInitialized(Atum.scala:229)
    at za.co.absa.atum.core.Atum$.preventNotInitialized(Atum.scala:233)
    at za.co.absa.atum.core.Atum$.dispose(Atum.scala:163)
    at za.co.absa.atum.AtumImplicits$SparkSessionWrapper.disableControlMeasuresTracking(AtumImplicits.scala:144)

[Zejnilovic]

Hello @pokpetr, the plan is to support spark3 in the near future. I cannot give you a precise date now but will keep you posted.

As for this issue.

• Can we get more info about the code or maybe an example?
• Is it something that worked with 2.4 and not is not working?

[pokpetr]

Sorry, I was wrong. The original issue is not spark3 incompatibility but Scala 2.12 incompatibility.

SparkSessionWrapper.enableControlMeasuresTracking throws the following error

Caused by: java.lang.NoClassDefFoundError: scala/Product$class

The "Control framework tracking is not initialized" error is caused by calling disableControlMeasuresTracking on the spark session where the tracking isn't initialized because of the error above.

Anyway I assume that the upgrade to Scala 2.12 is not possible without upgrade to spark3.

[Zejnilovic]

Are you using Atum from Maven central directly? Currently, Atum on maven is built only with/for scala 2.11 (2.11 doesn't work with spark3). You can build it yourself with 2.12 and see if it works. Otherwise please wait for the new release.

[Zejnilovic]

Hello @pokpetr, please try the newly released Atum

<dependency>
  <groupId>za.co.absa</groupId>
  <artifactId>atum_2.12</artifactId>
  <version>3.2.0</version>
</dependency>