-
Notifications
You must be signed in to change notification settings - Fork 582
Closed
Closed
Copy link
Labels
Description
Backend
VL (Velox)
Bug description
NPE occurs after uniffle shuffle writer fallback to row based.
sql:
select count(1) from (select /*+ repartition(100) */ * from web_page) t
fallback message:
error:
24/10/14 17:18:38 ERROR TaskResources: Task 0 failed by error:
java.lang.NullPointerException
at org.apache.spark.shuffle.writer.WriteBufferManager.addRecord(WriteBufferManager.java:300)
at org.apache.spark.shuffle.writer.RssShuffleWriter.writeImpl(RssShuffleWriter.java:357)
at org.apache.spark.shuffle.writer.RssShuffleWriter.write(RssShuffleWriter.java:280)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
We should reset rss.row.based after shuffle writer fallback.
Spark version
None
Spark configurations
No response
System information
No response
Relevant logs
No response
Reactions are currently unavailable
