JPMML-Evaluator-Spark

PMML evaluator library for Apache Spark.

Features

This library provides an Apache Spark ML-compatible wrapper for the JPMML-Evaluator library.

Compatibility matrix

Active development branches:

JPMML-Evaluator-Spark branch	Apache Spark version	Scala version
2.0.X	3.0.X through 3.5.X	2.12.X
master	4.0.X and 4.1.X	2.13.X

Archived development branches:

JPMML-Evaluator-Spark branch	Apache Spark version	Scala version
1.4.X	3.0.X through 3.5.X	2.12.X

Prerequisites

• Apache Spark 3.0.X through 3.5.X, 4.0.X or 4.1.X.

Installation

The JPMML-Evaluator-Spark is distributed in two variants:

• Library JAR file. Suitable for integrating JPMML-Evaluator into a full-scale Apache Spark application.
• Runtime uber-JAR file. Suitable for ad hoc tasks, such as adding JPMML-Evaluator to Apache Toree-powered Jupyter notebook.

Binary distribution

Library

The JPMML-Evaluator-Spark library JAR file (together with accompanying Java source and Javadocs JAR files) is released via Maven Central Repository.

The current version is 2.1.2 (8 February, 2026).

<dependency>
	<groupId>org.jpmml</groupId>
	<artifactId>jpmml-evaluator-spark</artifactId>
	<version>2.1.2</version>
</dependency>

Runtime uber-JAR file

The JPMML-Evaluator-Spark runtime uber-JAR file is released via the GitHub releases page.

Source distribution

Enter the project root directory and build using Apache Maven:

mvn clean install

The build produces two JAR files:

• target/jpmml-evaluator-spark-2.1-SNAPSHOT.jar - Library JAR file.
• target/jpmml-evaluator-spark-runtime-2.1-SNAPSHOT.jar - Runtime uber-JAR file.

Usage

Workflow

Build a org.jpmml.evaluator.Evaluator object that will do the actual evaluation work:

import java.io.InputStream
import org.jpmml.evaluator.{Evaluator, LoadingModelEvaluatorBuilder}

val pmmlIs: InputStream = ???

val evaluator: Evaluator = try {
	new LoadingModelEvaluatorBuilder()
		.load(pmmlIs)
		.build()
} finally {
	pmmlIs.close()
}

// Perform self-check
evaluator.verify()

Wrap this Evaluator object into an org.jpmml.evaluator.spark.PMMLTransformer object to make it look and behave like a regular Apache Spark ML transformer.

There are two built-in implementation classes available:

• org.jpmml.evaluator.spark.FlatPMMLTransformer
• org.jpmml.evaluator.spark.NestedPMMLTransformer

They are functionally identical evaluation-wise. The only difference is about how the result columns are structured (flat layout vs. nested layout).

It is possible to toggle column groups (ie. inputs, targets and outputs) on and off to keep the results maximally relevant.

import org.jpmml.evaluator.spark.{FlatPMMLTransformer, NestedPMMLTransformer, PMMLTransformer}

val pmmlTransformer = new FlatPMMLTransformer(evaluator)
//val pmmlTransformer = new NestedPMMLTransformer(evaluator)

A newly-constructed PMMLTransformer object is instantly ready for transformation work (ie. it does not exhibit any fitting behaviour).

val df = ???

val transformedDf = pmmlTransformer.transform(df)

Evaluation exceptions, if any, are caught and stored in a dedicated exceptions column.

Public API

PMMLTransformer

Abstract base class that provides common parameters and functionality.

Parameters:

• inputs: BooleanParam = true. Copy all columns from the input dataset to the transformed dataset?
• targets: BooleanParam = true. Produce columns for PMML target fields?
• outputs: BooleanParam = true. Produce columns for PMML output fields?
• exceptionCol: Param[String] = "pmmlException". The name of the exceptions column.
• syntheticTargetName: Param[String] = "_target". The substitute name for a synthetic target field column (the default name for a synthetic target field is null, which is not a valid Apache Spark column name).

FlatPMMLTransformer

Concrete implementation class, which maps all PMML result fields to top-level columns.

Parameters: N/A

Transformed schema for the example DecisionTreeIris model:

• Four input columns.
• One PMML target field column.
• Three PMML output field columns.
• One exceptions column.

root
 |-- Sepal.Length: double (nullable = true)
 |-- Sepal.Width: double (nullable = true)
 |-- Petal.Length: double (nullable = true)
 |-- Petal.Width: double (nullable = true)
 |-- Species: string (nullable = true)
 |-- probability(setosa): double (nullable = true)
 |-- probability(versicolor): double (nullable = true)
 |-- probability(virginica): double (nullable = true)
 |-- pmmlException: string (nullable = true)

If the evaluation fails for some row, then all PMML result fields columns contain null for that row.

NestedPMMLTransformer

Concrete implementation class, which maps all PMML target and output fields to a single top-level nested results column.

Parameters:

• resultsCol: Param[String] = "pmmlResults". The name of the nested results column.

Transformed schema for the example DecisionTreeIris model:

• Four input columns.
• One nested results column, containing one PMML target field and three output fields.
• One exceptions column.

root
 |-- Sepal.Length: double (nullable = true)
 |-- Sepal.Width: double (nullable = true)
 |-- Petal.Length: double (nullable = true)
 |-- Petal.Width: double (nullable = true)
 |-- pmmlResults: struct (nullable = true)
 |    |-- Species: string (nullable = true)
 |    |-- probability(setosa): double (nullable = true)
 |    |-- probability(versicolor): double (nullable = true)
 |    |-- probability(virginica): double (nullable = true)
 |-- pmmlException: string (nullable = true)

If the evaluation fails for some row, then the results column contains null for that row.

Use the dot notation to access individual fields afterwards:

transformedDf.select("pmmlResults.Species").show()

License

JPMML-Evaluator-Spark is licensed under the terms and conditions of the GNU Affero General Public License, Version 3.0. For a quick summary of your rights ("Can") and obligations ("Cannot" and "Must") under AGPLv3, please refer to TLDRLegal.

If you would like to use JPMML-Evaluator-Spark in a proprietary software project, then it is possible to enter into a licensing agreement which makes it available under the terms and conditions of the BSD 3-Clause License instead.

Additional information

JPMML-Evaluator-Spark is developed and maintained by Openscoring Ltd, Estonia.

Interested in using JPMML software in your software? Please contact info@openscoring.io