Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time

[HariprasadAllaka1612]

I am trying to write data to s3 as Hudi file.

Code:

 inputDF
      .write.format("org.apache.hudi")
      .option(HoodieWriteConfig.TABLE_NAME, tablename)
      .option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "GameId")
      .option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY,"OperatorShortName")
      .option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "HandledTimestamp")
      .option(DataSourceWriteOptions.OPERATION_OPT_KEY, DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL)
      .option(DataSourceWriteOptions.STORAGE_TYPE_OPT_KEY, DataSourceWriteOptions.MOR_STORAGE_TYPE_OPT_VAL)
      .mode(SaveMode.Append)
      .save("s3a://" + s3RawDataLakeBucket + "/Games2" )

coming across below exception

Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:66)
Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20190917110809
	at org.apache.hudi.HoodieWriteClient.upsert(HoodieWriteClient.java:177)
	at org.apache.hudi.DataSourceUtils.doWriteOperation(DataSourceUtils.java:181)
	at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:143)
	at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:91)
	at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
	at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
	at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
	at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
	at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
	at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)
	at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:276)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:270)
	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:228)
	at com.playngoplatform.scala.dao.DataAccessS3.writeDataToRefinedS3(DataAccessS3.scala:30)
	at com.playngoplatform.scala.controller.GameAndProviderDataTransform.processData(GameAndProviderDataTransform.scala:27)
	at com.playngoplatform.scala.action.GameAndProviderData$.main(GameAndProviderData.scala:10)
	at com.playngoplatform.scala.action.GameAndProviderData.main(GameAndProviderData.scala)
	... 5 more
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.fs.FSDataOutputStream: method <init>(Ljava/io/OutputStream;)V not found
	at org.apache.hudi.common.io.storage.SizeAwareFSDataOutputStream.<init>(SizeAwareFSDataOutputStream.java:46)
	at org.apache.hudi.common.io.storage.HoodieWrapperFileSystem.wrapOutputStream(HoodieWrapperFileSystem.java:160)
	at org.apache.hudi.common.io.storage.HoodieWrapperFileSystem.create(HoodieWrapperFileSystem.java:168)
	at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createFileInPath(HoodieActiveTimeline.java:386)
	at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createFileInMetaPath(HoodieActiveTimeline.java:371)
	at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.saveToInflight(HoodieActiveTimeline.java:359)
	at org.apache.hudi.HoodieWriteClient.saveWorkloadProfileMetadataToInflight(HoodieWriteClient.java:417)
	at org.apache.hudi.HoodieWriteClient.upsertRecordsInternal(HoodieWriteClient.java:440)
	at org.apache.hudi.HoodieWriteClient.upsert(HoodieWriteClient.java:172)
	... 33 more

[vinothchandar]

@HariprasadAllaka1612 what version of Hadoop and Hudi are you using? We have not tested with Hadoop 3.0 for e.g if thats what you are using

Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.fs.FSDataOutputStream: method <init>(Ljava/io/OutputStream;)V not found

[HariprasadAllaka1612]

I will change the version of hadoop to 2.8.4 and check if it works.

[pratyakshsharma]

I am using Hadoop 3.1.0 with Hoodie-0.4.7, actually the constructor has changed in hadoop 3.x versions for FSDataOutputStream class. This is a possible duplicate of - https://www.mail-archive.com/dev@hudi.apache.org/msg00286.html

[vinothchandar]

accmulating hadoop 3 issues here https://issues.apache.org/jira/browse/HUDI-259

[BdLearnerr]

@HariprasadAllaka1612 i am facing same issue , what is the solution , i am using spark2.4.5 and hudi 0.6.0 version