Skip to content

autodeployai/pypmml-spark

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

PyPMML-Spark

PyPMML-Spark is a Python PMML scoring library for PySpark as SparkML Transformer, it really is the Python API for PMML4S-Spark.

Prerequisites

  • Java >= 1.8
  • Python 2.7 or >= 3.5

Dependencies

Module PySpark
pypmml-spark PySpark >= 3.0.0
pypmml-spark2 PySpark >= 2.4.0, < 3.0.0

Installation

pip install pypmml-spark

Or install the latest version from github:

pip install --upgrade git+https://github.com/autodeployai/pypmml-spark.git

After that, you need to do more to use it in Spark that must know those jars in the package pypmml_spark.jars. There are several ways to do that:

  1. The easiest way is to run the script link_pmml4s_jars_into_spark.py that is delivered with pypmml-spark:

    link_pmml4s_jars_into_spark.py
  2. Use those config options to specify dependent jars properly. e.g. --jars, or spark.executor.extraClassPath and spark.executor.extraClassPath. See Spark for details about those parameters.

Usage

  1. Load model from various sources, e.g. filename, string, or array of bytes.

    from pypmml_spark import ScoreModel
    
    # The model is from http://dmg.org/pmml/pmml_examples/KNIME_PMML_4.1_Examples/single_iris_dectree.xml
    model = ScoreModel.fromFile('single_iris_dectree.xml')
  2. Call transform(dataset) to run a batch score against an input dataset.

    # The data is from http://dmg.org/pmml/pmml_examples/Iris.csv
    df = spark.read.csv('Iris.csv', header='true')
    score_df = model.transform(df)

Use PMML in Scala or Java

See the PMML4S project. PMML4S is a PMML scoring library for Scala. It provides both Scala and Java Evaluator API for PMML.

Use PMML in Python

See the PyPMML project. PyPMML is a Python PMML scoring library, it really is the Python API for PMML4S.

Use PMML in Spark

See the PMML4S-Spark project. PMML4S-Spark is a PMML scoring library for Spark as SparkML Transformer.

Deploy PMML as REST API

See the AI-Serving project. AI-Serving is serving AI/ML models in the open standard formats PMML and ONNX with both HTTP (REST API) and gRPC endpoints.

Deploy and Manage AI/ML models at scale

See the DaaS system that deploys AI/ML models in production at scale on Kubernetes.

Support

If you have any questions about the PyPMML-Spark library, please open issues on this repository.

Feedback and contributions to the project, no matter what kind, are always very welcome.

License

PyPMML-Spark is licensed under APL 2.0.

About

Python PMML scoring library for PySpark as SparkML Transformer

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages