[SUPPORT] Recreate deleted metadata table

[szingerpeter]

Hy,

I'm using Hudi CLI version 1.0; hudi version 0.11.0; Spark version 3.2.1-amzn-0 and Hive version 3.1.3-amzn-0.
After rolling back a table I was facing the issue described in #4747

Caused by: java.io.FileNotFoundException: No such file or directory 's3://...

Thereafter, following the recommendation on #4747, I deleted manually the metadata folder under s3://<table_path>/.hoodie/metadata, which solved the problem.

After upserting into the table, the metadata s3://<table_path>/.hoodie/metadata gets recreated. However, after querying the data via spark and beeline, it only returns the entries, which have been upserted in the last operation (~40M rows) and not any previous data (~2B rows). If i delete s3://<table_path>/.hoodie/metadata again, then both spark and beeline returns all the historical data and the newly inserted data.

I tried using hudi cli's metadata create command, but it fails with:

29990 [Spring Shell] ERROR org.apache.spark.SparkContext  - Error initializing SparkContext.
java.lang.ClassCastException: org.apache.hadoop.yarn.proto.YarnServiceProtos$GetClusterMetricsRequestProto cannot be cast to com.google.protobuf.Message
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
        at com.sun.proxy.$Proxy46.getClusterMetrics(Unknown Source)
        at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:271)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
        at com.sun.proxy.$Proxy47.getClusterMetrics(Unknown Source)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:631)
        at org.apache.spark.deploy.yarn.Client.$anonfun$submitApplication$1(Client.scala:181)
        at org.apache.spark.internal.Logging.logInfo(Logging.scala:57)
        at org.apache.spark.internal.Logging.logInfo$(Logging.scala:56)
        at org.apache.spark.deploy.yarn.Client.logInfo(Client.scala:65)
        at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:181)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:220)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:582)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
        at org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:117)
        at org.apache.hudi.cli.commands.MetadataCommand.initJavaSparkContext(MetadataCommand.java:367)
        at org.apache.hudi.cli.commands.MetadataCommand.create(MetadataCommand.java:128)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
        at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
        at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
        at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
        at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
        at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
        at java.lang.Thread.run(Thread.java:750)

is there a way of recreating the metadata table of an existing hudi table such that it will reference historical data as well?

[yihua]

@szingerpeter Thanks for reporting the issue. On Hudi 0.11.0 release, the metadata table is enabled by default. After you deleted the s3://<table_path>/.hoodie/metadata, the next write operation doing upserts should reinitialize the metadata table automatically, with all files in the table, not just the ones from the last upsert operation.

To help triage the issue, could you share the write configs for the upsert job, and the active timeline of the metadata table (under s3://<table_path>/.hoodie/metadata/.hoodie)?

Based on the information, it looks like you're using EMR 6.7.0 release. Could you try EMR 6.8.0 release which has Hudi 0.11.1 release and see if you hit the same problem?

[szingerpeter]

@yihua , thank you for your quick reply.

Unfortunately, the environment is fixed and upgrading EMR is not possible right now.

i sent the requested files via slack

[nsivabalan]

feel free to close the issue if you get it resolved and not looking for any more AIs from us.

[szingerpeter]

i'm still in touch with @yihua

[xushiyan]

@szingerpeter @yihua what is the latest state of this issue?

[leobiscassi]

I'm experiencing a similar situation. I did upgrade my tables to EMR 6.9 with hudi 0.12, my pipelines broke, so I downgraded the tables back to EMR 6.5 and hudi 0.9. After that I'm seeing that even with the metadata table config enabled I'm not able to see them on s3. I've tried to do the following:

• Started EMR 6.5 cluster
• Executed hudi cli with the command sudo /usr/lib/hudi/cli/bin/hudi-cli.sh
• Connected to my table using connect --path <S3-PATH>
• Executed the command metadata create

The command fails but it seems to create an empty metadata table, this is the stack trace

2023-01-10 18:47:36,061 INFO scheduler.DAGScheduler: ResultStage 0 (collect at HoodieSparkEngineContext.java:73) failed in 0.607 s due to Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3) (ip-172-31-2-164.us-west-2.compute.internal executor 1): java.lang.IllegalStateException: unread block data
        at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2934)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1704)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:457)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

Driver stacktrace:
2023-01-10 18:47:36,064 INFO scheduler.DAGScheduler: Job 0 failed: collect at HoodieSparkEngineContext.java:73, took 0.652691 s
2023-01-10 18:47:36,065 ERROR core.SimpleExecutionStrategy: Command failed java.lang.reflect.UndeclaredThrowableException
2023-01-10 18:47:36,066 WARN JLineShellComponent.exceptions: 
java.lang.reflect.UndeclaredThrowableException
        at org.springframework.util.ReflectionUtils.rethrowRuntimeException(ReflectionUtils.java:315)
        at org.springframework.util.ReflectionUtils.handleInvocationTargetException(ReflectionUtils.java:295)
        at org.springframework.util.ReflectionUtils.handleReflectionException(ReflectionUtils.java:279)
        at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:219)
        at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
        at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
        at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
        at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
        at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3) (ip-172-31-2-164.us-west-2.compute.internal executor 1): java.lang.IllegalStateException: unread block data
        at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2934)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1704)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:457)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2470)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2419)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2418)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2418)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1125)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1125)
        at scala.Option.foreach(Option.scala:407)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1125)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2684)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2626)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2615)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:914)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2241)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2262)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2281)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2306)
        at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
        at org.apache.spark.rdd.RDD.collect(RDD.scala:1029)
        at org.apache.spark.api.java.JavaRDDLike.collect(JavaRDDLike.scala:362)
        at org.apache.spark.api.java.JavaRDDLike.collect$(JavaRDDLike.scala:361)
        at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
        at org.apache.hudi.client.common.HoodieSparkEngineContext.map(HoodieSparkEngineContext.java:73)
        at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.getPartitionsToFilesMapping(HoodieBackedTableMetadataWriter.java:365)
        at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.bootstrapFromFilesystem(HoodieBackedTableMetadataWriter.java:313)
        at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.bootstrapIfNeeded(HoodieBackedTableMetadataWriter.java:272)
        at org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.initialize(SparkHoodieBackedTableMetadataWriter.java:91)
        at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.<init>(HoodieBackedTableMetadataWriter.java:114)
        at org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.<init>(SparkHoodieBackedTableMetadataWriter.java:62)
        at org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.create(SparkHoodieBackedTableMetadataWriter.java:58)
        at org.apache.hudi.cli.commands.MetadataCommand.create(MetadataCommand.java:104)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
        ... 6 more
Caused by: java.lang.IllegalStateException: unread block data
        at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2934)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1704)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2431)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2355)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2213)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1669)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:457)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        ... 1 more

Does anyone have a clue on why this is happening?

[nsivabalan]

Recommended way to delete metadata table for hudi versions > 0.11.0, disable metadata via write configs(hoodie.metadata.enable=false) in next write to hudi and hudi will programmatically take care of deleting the metadata table and remove all references (like hoodie.properties).

After disabling, if you wish to re-instantiate metadata table, after few commits, you can re-enable it back via write configs (hoodie.metadata.enable=true) and hudi will take care of populating the metadata table from scratch for you.

If you prefer async way of building metadata table so as not to block your regular writers, you can try our async indexer as well. More info can be found here.

Having said all this, curious to know what issues you folks encountered that demanded deleting metadata table. Would you help going over the details.

[nsivabalan]

During downgrade, depending on versions, hudi will automatically delete the metadata table. For eg, if you downgrade from 0.12.0 to 0.9.0, hudi will delete the metadata table since the way we populate metadata table differs across these versions and hence.