[Bug] Error when writing to a Paimon table in Spark with the Paimon catalog in Hive

[Aiden-Dong]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

0.8.1

Compute Engine

spark 3.3.1

Minimal reproduce step

val session:SparkSession = SparkSession.builder()
.conf("spark.sql.extensions", "org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions")
.enableHiveSupport()
.getOrCreate()

// parquet table wirte to paimon table
session.sql(
"""
|insert into sparksql_test.tb_yarn_job_summary_off_on_v2_paimon_dongtao partition(pdate)
|select
| app_name ,
| start_time ,
| mem ,
| cpu ,
| run_time ,
| run_user ,
| gid ,
| group_name ,
| onwer ,
| cluster ,
| queue ,
| app_type ,
| queue_type ,
| tid ,
| tenant_name ,
| memcost ,
| cpucost ,
| cost ,
| rk ,
| pdate
|from
| sparksql_test.tb_yarn_job_summary_off_on_v2
|""".stripMargin
)

What doesn't meet your expectations?

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:274)
at org.apache.spark.sql.hive.execution.HiveOutputWriter.(HiveFileFormat.scala:148)
at org.apache.spark.sql.hive.execution.HiveFileFormat$$anon$1.newInstance(HiveFileFormat.scala:106)
at org.apache.spark.sql.execution.datasources.BaseDynamicPartitionDataWriter.renewCurrentWriter(FileFormatDataWriter.scala:298)
at org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:365)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:85)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:335)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1550)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:342)
... 9 more
Caused by: java.lang.NullPointerException
at org.apache.paimon.hive.mapred.TezUtil$TaskAttemptWrapper.(TezUtil.java:111)
at org.apache.paimon.hive.mapred.TezUtil.taskAttemptWrapper(TezUtil.java:84)
at org.apache.paimon.hive.mapred.PaimonOutputFormat.writer(PaimonOutputFormat.java:76)
at org.apache.paimon.hive.mapred.PaimonOutputFormat.getHiveRecordWriter(PaimonOutputFormat.java:72)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:286)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:271)

Anything else?

none

Are you willing to submit a PR?

• I'm willing to submit a PR!

[imzhouhao]

hi，i also met this question .how do you fix it?

[Aiden-Dong]

hi，i also met this question .how do you fix it?

#5933

[imzhouhao]

5933 seems to linked with #5932, not this