[SUPPORT]Caused by: org.apache.hudi.exception.HoodieIOException: Exception create input stream from file: HoodieLogFile{pathStr='hdfs://nameservice1/xxx/.00000056-15ec-459f-bb67-5f8c2b319203_20241228152030542.log.1_1-5-1', fileLen=-1} #12554
Description
操作步骤:
1.Flink 批处理将Hive历史数据迁移到Hudi表中；
2.Flink 接入Kafka数据写入HDFS；
3.Hudi 采用的是离线合并；
4.Spark每小时读取Hudi全量数据。
出现的问题
Caused by: org.apache.hudi.exception.HoodieIOException: Exception create input stream from file: HoodieLogFile{pathStr='hdfs://nameservice1/xxx/.00000056-15ec-459f-bb67-5f8c2b319203_20241228152030542.log.1_1-5-1', fileLen=-1}
运行环境
-
Hudi version : 0.14.1
-
Spark version : 3.3.2
-
Hive version : 11.0-cdh5.13.2
-
Hadoop version : 2.6.0-cdh5.13.2
-
Storage (HDFS/S3/GCS..) : HDFS
-
Running on Docker? (yes/no) : no
具体的报错
2024-12-28 16:30:28,806 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Failed to trigger checkpoint for job dbb88271f16bc32329acd07b5da75321 because Some tasks of the job have already finished and checkpointing with finished tasks is not enabled. Failure reason: Not all required tasks are currently running.
2024-12-28 16:30:37,983 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - compact_task (38/60) (8d6d4c0f31ae7ce0af767c5dd919b407) switched from RUNNING to FAILED on container_1724338424009_17149855_01_011191 @ bdc-dn3077.bigdata.jxtelecom.cn (dataPort=38642).
org.apache.hudi.exception.HoodieException: Exception when reading log file
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternalV1(AbstractHoodieLogRecordReader.java:414) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:220) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:201) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.(HoodieMergedLogRecordScanner.java:117) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.(HoodieMergedLogRecordScanner.java:76) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner$Builder.build(HoodieMergedLogRecordScanner.java:466) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:210) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:147) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.sink.compact.CompactOperator.doCompaction(CompactOperator.java:142) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.sink.compact.CompactOperator.processElement(CompactOperator.java:130) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) ~[flink-dist_2.11-1.14.2.jar:1.14.2]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]
Caused by: org.apache.hudi.exception.HoodieIOException: Exception create input stream from file: HoodieLogFile{pathStr='hdfs://bdccluster-1/telecom_data/CORE_MODEL/IP/BSS/DAY/INF_SS_OFFER_INST_D_HUDI_ALL_FLINK2/.00000056-15ec-459f-bb67-5f8c2b319203_20241228152030542.log.1_1-5-1', fileLen=-1}
at org.apache.hudi.common.table.log.HoodieLogFileReader.getFSDataInputStream(HoodieLogFileReader.java:486) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieLogFileReader.(HoodieLogFileReader.java:121) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieLogFormatReader.(HoodieLogFormatReader.java:64) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternalV1(AbstractHoodieLogRecordReader.java:245) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
... 22 more
Caused by: org.apache.hadoop.ipc.RemoteException: java.lang.ArrayIndexOutOfBoundsException
at org.apache.hadoop.ipc.Client.call(Client.java:1476) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.ipc.Client.call(Client.java:1413) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at com.sun.proxy.$Proxy35.getBlockLocations(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_111]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_111]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at com.sun.proxy.$Proxy36.getBlockLocations(Unknown Source) ~[?:?]
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1235) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1222) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1210) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:306) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:272) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:264) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1535) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:312) ~[flink-shaded-hadoop-2-uber-2.7.5-7.0.jar:2.7.5-7.0]
at org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:203) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieLogFileReader.getFSDataInputStream(HoodieLogFileReader.java:484) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieLogFileReader.<init>(HoodieLogFileReader.java:121) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.HoodieLogFormatReader.<init>(HoodieLogFormatReader.java:64) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternalV1(AbstractHoodieLogRecordReader.java:245) ~[hudi-flink1.14-bundle-0.14.1.jar:0.14.1]
... 22 more
2024-12-28 16:30:37,984 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job dbb88271f16bc32329acd07b5da75321: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=59}]
2024-12-28 16:30:37,984 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task 0d75f8675b58c9bef24c60fb5f5d0a16_37.
2024-12-28 16:30:37,984 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 62 tasks should be restarted to recover the failed task 0d75f8675b58c9bef24c60fb5f5d0a16_37.