Adding a Postprocessing filter (starting from scratch)

[diegoeche]

Background [Optional]

I'm working on a POC for spline where we want to add metadata using a Postprocessing filter.

• I'm using Docker.
• I have knowledge of the Scala language

This is probably a silly question out of the fact I haven't used java in ages, and I lost touch with the ecosystem :P

Question

I have a Scala source code for a Postprocessing filter CustomFilter.scala.

I'm unsure what are the steps to be able to add something like:

pyspark \
  --packages za.co.absa.spline.agent.spark:spark-2.4-spline-agent-bundle_2.12:0.6.2 \
  --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" \
  --conf "spark.spline.lineageDispatcher.http.producer.url=http://localhost:8080/producer"
  --conf "spline.postProcessingFilter=my-filter" \
  --conf "spline.postProcessingFilter.my-filter.className=CustomFilter"

I just want to be able to iterate fast, edit my filter, run the pyspark shell, and see if it's doing the right thing.

[diegoeche]

Probably I should have posted in the spline-agent repository and not on this one. Any pointers would be really appreciated though.

[wajda]

You need to add CustomFilter.class and all its dependencies to the Spark driver classpath (excluding ones that are already in the Spark driver or the Spline agent).

Check --driver-class-path or --jars options of the pyspark

  --jars JARS                 Comma-separated list of jars to include on the driver
                              and executor classpaths.

  --driver-class-path         Extra class path entries to pass to the driver. Note that
                              jars added with --jars are automatically included in the
                              classpath.

[diegoeche]

Awesome. Thanks for the reply.

I was also having issues compiling and knowing which dependencies I need for running the given filter.

FWIW:
This little build.sbt did the trick:

// The simplest possible sbt build file is just one line:

scalaVersion := "2.12.14"
// That is, to create a valid sbt build, all you've got to do is define the
// version of Scala you'd like your project to use.

// ============================================================================

// Lines like the above defining `scalaVersion` are called "settings". Settings
// are key/value pairs. In the case of `scalaVersion`, the key is "scalaVersion"
// and the value is "2.13.3"

// It's possible to define many kinds of settings, such as:

name := "dataopsfilter"
organization := "com.riotgames"
version := "1.0"


// Want to use a published library in your project?
// You can define other libraries as dependencies in your build like this:

libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"

libraryDependencies += "za.co.absa.spline.agent.spark" %% "spark-2.4-spline-agent-bundle" % "0.6.2"

[wajda]

A little suggestion - don't use spark-2.4-spline-agent-bundle as a dependency (it's a fat-jar and isn't intended for being referred from other projects). Use agent-core instead.

[diegoeche]

This worked like a charm. Thanks so much @wajda !

I'm adding the updated snippet here in case anybody finds it useful.

[diegoeche]

scalaVersion := "2.12.14"

name := "dataopsfilter"
organization := "your.package"
version := "1.0"

// You can define other libraries as dependencies in your build like this:
libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
libraryDependencies += "za.co.absa.spline.agent.spark" %% "agent-core" % "0.6.2"