Find file History
Failed to load latest commit information.
src/main SDC-4561. Add destroy method for Spark Transformer API Nov 22, 2016
pom.xml Update version in master to for spark-api Feb 9, 2017

Streamsets Spark Transformer

This API can be used in combination with the Spark Evaluator in Streamsets Data Collector to apply arbitrary transformations to a batch of Records using Spark.

An implementation of the com.streamsets.pipeline.spark.api.SparkTransformer should be inserted as an additional library in the SDC's classpath using the method described here The Spark Processor is available in several stage libraries, and this jar should be in the correct library for the specific stage library being used.

Once you have installed the library, start the SDC with the Spark Processor and specify the Fully Qualified Class Name of the class that implements SparkTransformer class.