I am getting below exception while reading csv file and writing it to parquet file

[pallav1991]

Today After Upgrade when I ran my Program I am getting below exception.

{Py4JJavaError}An error occurred while calling o125.parquet.
: java.lang.NoSuchMethodError: org.apache.spark.sql.internal.SQLConf$.LEGACY_PASS_PARTITION_BY_AS_OPTIONS()Lorg/apache/spark/internal/config/ConfigEntry;
	at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:277)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
	at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:566)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.comm...

Can you tell me why is this exception occurring.

[issue-label-bot]

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.59. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[argenisleon]

Can you put here the exactly code you are using?

[pallav1991]

 if self.requestSubType == "createCSVToParquet":
                df = self.manager.optimus.load.csv(self.filePath)
            elif self.requestSubType == "createExcelToParquet":
                df = self.manager.optimus.load.excel(self.filePath, self.sheetName)
            if df is not None:
                df = convertColumnParquetFormat(df)
                df.write.mode("overwrite").parquet(self.haddopPath + self.fileName + '.parquet')
                return "success"
            return "failed"

[argenisleon]

It seems like some kind of bug
Can you save the data successfully to another format like csv?
Can you try:

self.manager.optimus.save.parquet(self.haddopPath + self.fileName + '.parquet')

It almost the same as the native spark save plus a couple of checks

[argenisleon]

Hi @pallav1991, this works for you?

[rajib76]

is this bug fixed? i am also getting this

[argenisleon]

Hi @rajib76, are you getting the same error? the code @pallav1991 provide use
df.write.mode("overwrite").parquet(self.haddopPath + self.fileName + '.parquet') which is pyspark code not optimus code.

Did you try:
optimus.save.parquet(self.haddopPath + self.fileName + '.parquet')

If that not work can you show us your code?

[pallav1991]

optimus.save

sorry @argenisleon for the late reply.
I have a question before i execute the above request.
in the above code where am i specifying the dataframe to write on the parquet file.

[argenisleon]

My bad,

The correct code is:
df.save.parquet(self.haddopPath + self.fileName + '.parquet')