[BUG] java.lang.NoSuchMethodError when using ComputeModelStatistics.transform()

[kyoungrok0517]

SynapseML version

1.0.4

System information

• Language version: python 3.11, scala 2.12
• Spark Version: 3.5.1
• Spark Platform: Custom

Describe the problem

I see the following error when I call ComputeModelStatistics.transform()

Py4JJavaError: An error occurred while calling o196.transform.
: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/catalyst/encoders/ExpressionEncoder;
	at com.microsoft.azure.synapse.ml.train.ComputeModelStatistics.addConfusionMatrixToResult(ComputeModelStatistics.scala:255)
	at com.microsoft.azure.synapse.ml.train.ComputeModelStatistics.$anonfun$transform$1(ComputeModelStatistics.scala:124)
	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb(BasicLogging.scala:62)
	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb$(BasicLogging.scala:59)
	at com.microsoft.azure.synapse.ml.train.ComputeModelStatistics.logVerb(ComputeModelStatistics.scala:58)
	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTransform(BasicLogging.scala:52)
	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTransform$(BasicLogging.scala:51)
	at com.microsoft.azure.synapse.ml.train.ComputeModelStatistics.logTransform(ComputeModelStatistics.scala:58)
	at com.microsoft.azure.synapse.ml.train.ComputeModelStatistics.transform(ComputeModelStatistics.scala:76)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:745)

Code to reproduce issue

model_dir = ...

# read
predictions = ...
# predictions.head()

metrics = ComputeModelStatistics(
    evaluationMetric="classification",
    labelCol="label",
    scoredLabelsCol="prediction",
)
metrics.transform(predictions)

Other info / logs

Py4JJavaError: An error occurred while calling o196.transform.
: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.encoders.RowEncoder$.apply(Lorg/apache/spark/sql/types/StructType;)Lorg/apache/spark/sql/catalyst/encoders/ExpressionEncoder;
	at com.microsoft.azure.synapse.ml.train.ComputeModelStatistics.addConfusionMatrixToResult(ComputeModelStatistics.scala:255)
	at com.microsoft.azure.synapse.ml.train.ComputeModelStatistics.$anonfun$transform$1(ComputeModelStatistics.scala:124)
	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb(BasicLogging.scala:62)
	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb$(BasicLogging.scala:59)
	at com.microsoft.azure.synapse.ml.train.ComputeModelStatistics.logVerb(ComputeModelStatistics.scala:58)
	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTransform(BasicLogging.scala:52)
	at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTransform$(BasicLogging.scala:51)
	at com.microsoft.azure.synapse.ml.train.ComputeModelStatistics.logTransform(ComputeModelStatistics.scala:58)
	at com.microsoft.azure.synapse.ml.train.ComputeModelStatistics.transform(ComputeModelStatistics.scala:76)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.lang.Thread.run(Thread.java:745)

What component(s) does this bug affect?

• area/cognitive: Cognitive project
• area/core: Core project
• area/deep-learning: DeepLearning project
• area/lightgbm: Lightgbm project
• area/opencv: Opencv project
• area/vw: VW project
• area/website: Website
• area/build: Project build system
• area/notebooks: Samples under notebooks folder
• area/docker: Docker usage
• area/models: models related issue

What language(s) does this bug affect?

• language/scala: Scala source code
• language/python: Pyspark APIs
• language/r: R APIs
• language/csharp: .NET APIs
• language/new: Proposals for new client languages

What integration(s) does this bug affect?

• integrations/synapse: Azure Synapse integrations
• integrations/azureml: Azure ML integrations
• integrations/databricks: Databricks integrations