[MINOR] Fix npe for get internal schema #9984
Conversation
watermelon12138
force-pushed
the
FixNpeForGetInternalSchema
branch
from
7872833
to
e77abbf
Compare
danny0405
added
the
writer-core
|
@hudi-bot run azure |
| ? (StringUtils.isNullOrEmpty(avroSchema) | ||
| ? InternalSchema.getEmptyInternalSchema() | ||
| : AvroInternalSchemaConverter.convert(HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(avroSchema)))) | ||
| : fileSchema; |
There was a problem hiding this comment.
thanks for your fix.
why avro schema is null here ?
There was a problem hiding this comment.
you can read the exception stack of this problem:
Caused by: org.apache.avro.SchemaParseException: Cannot parse schema
at org.apache.avro.Schema.parse(Schema.java:1633)
at org.apache.avro.Schema$Parser.parse(Schema.java:1430)
at org.apache.avro.Schema$Parser.parse(Schema.java:1418)
at org.apache.hudi.common.util.InternalSchemaCache.getInternalSchemaByVersionId(InternalSchemaCache.java:220)
at org.apache.hudi.common.util.InternalSchemaCache.getInternalSchemaByVersionId(InternalSchemaCache.java:226)
at org.apache.hudi.table.action.commit.HoodieMergeHelper.composeSchemaEvolutionTransformer(HoodieMergeHelper.java:177)
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:94)
at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.handleUpdateInternal(HoodieSparkCopyOnWriteTable.java:252)
at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.handleUpdate(HoodieSparkCopyOnWriteTable.java:235)
at org.apache.hudi.table.action.compact.CompactionExecutionHelper.writeFileAndGetWriteStats(CompactionExecutionHelper.java:64)
at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:237)
at org.apache.hudi.table.action.compact.HoodieCompactor.lambda$compact$988df80a$1(HoodieCompactor.java:132)
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:223)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:352)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1498)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1408)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1472)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1295)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:133)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1474)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
There was a problem hiding this comment.
@xiarixiaoyao Hi, This problem is difficult to reproduce. I think that we can try to prevent it from the code perspective.
watermelon12138
force-pushed
the
FixNpeForGetInternalSchema
branch
2 times, most recently
from
63a77e3
to
23eb3d5
Compare
watermelon12138
force-pushed
the
FixNpeForGetInternalSchema
branch
from
23eb3d5
to
2fb3eb5
Compare
| return fileSchema.isEmptySchema() ? AvroInternalSchemaConverter.convert(HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(avroSchema))) : fileSchema; | ||
| return fileSchema.isEmptySchema() | ||
| ? StringUtils.isNullOrEmpty(avroSchema) | ||
| ? InternalSchema.getEmptyInternalSchema() |
There was a problem hiding this comment.
Is it because the version upgrade or something? Is the null avro schema coming from an old version Hudi table?
There was a problem hiding this comment.
@danny0405 Yes, Some users find this problem in the upgrade scenario(0.12.3 -> 0.14).
|
@hudi-bot run azure |
1 similar comment
|
@hudi-bot run azure |
|
@hudi-bot run azure |
|
@danny0405 👍🏻 |
|
@zyclove 1.0.0-beta is already under release process. |
Change Logs
related issue: #9902
get internal schema maybe meet npe when parse avroSchema. So, we need to return InternalSchema.getEmptyInternalSchema() when avroSchema is null or empty.
Impact
none
Risk level (write none, low medium or high below)
none
Contributor's checklist