Improve application classpath diagnostics

[yashwanthmadaka24]

df = spark.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load(input_dir+'stroke_100K_1.csv')
classifier = DecisionTreeClassifier(labelCol="label", featuresCol="features", maxDepth=25, minInstancesPerNode=30, impurity="gini")
pipeline = Pipeline(stages = [classifier])
pipelineModel = pipeline.fit(trainingData)


from pyspark2pmml import PMMLBuilder
pmmlBuilder = PMMLBuilder(sc, trainingData, pipelineModel) \
	.putOption(classifier, "compact", True)
pmmlBuilder.buildFile("DecisionTree.pmml")``

the following code is throwing an error which says:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-66-a6260012e861> in <module>()
      1 from pyspark2pmml import PMMLBuilder
      2 
----> 3 pmmlBuilder = PMMLBuilder(spark, trainingData, pipelineModel)   .putOption(classifier, "compact", True)
      4 
      5 pmmlBuilder.buildFile("DecisionTreeIris.pmml")

~/.local/lib/python3.6/site-packages/pyspark2pmml/__init__.py in __init__(self, sc, df, pipelineModel)
     10                 javaSchema = javaDf.schema.__call__()
     11                 javaPipelineModel = pipelineModel._to_java()
---> 12                 javaPmmlBuilder = sc._jvm.org.jpmml.sparkml.PMMLBuilder(javaSchema, javaPipelineModel)
     13                 if(not isinstance(javaPmmlBuilder, JavaObject)):
     14                         raise RuntimeError("JPMML-SparkML not found on classpath")

TypeError: 'JavaPackage' object is not callable

[vruusmann]

You probably don't have the JPMML-SparkML library on your Apache Spark application's classpath - see the README file.

What is your PySpark/Apache Spark version? It's strange that the execution doesn't reach the next line, which would throw a nice "JPMML-SparkML not found on classpath" exception.

[yashwanthmadaka24]

@vruusmann My pyspark version is 2.4.0

[yashwanthmadaka24]

I am having difficulties adding jar file into my sparks application classpath. What steps should I follow in order for it to reflect them to my jupyter notebook.

[yashwanthmadaka24]

This blog was helpful to solve my problem.
https://blog.sicara.com/get-started-pyspark-jupyter-guide-tutorial-ae2fe84f594f

[wiwengweng]

if you're using spark and pyspark, I recommend that you add the jar to spark jars directory