Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Databricks 13.3 executor side broadcast failure #10165

Closed
tgravescs opened this issue Jan 8, 2024 · 0 comments · Fixed by #10230
Closed

[BUG] Databricks 13.3 executor side broadcast failure #10165

tgravescs opened this issue Jan 8, 2024 · 0 comments · Fixed by #10230
Assignees
Labels
bug Something isn't working

Comments

@tgravescs
Copy link
Collaborator

Describe the bug
Running 24.02 snapshot on Databricks 13.3 I ran into the following error on a custom query:

Caused by: org.apache.spark.SparkException: Unexpected build plan for Executor Side Broadcast Join: 
ColumnarToRow
+- AQEShuffleRead ebj
   +- ShuffleQueryStage 132, Statistics(sizeInBytes=7.8 MiB, rowCount=2.74E+5, ColumnStat: N/A, isRuntime=true)
      +- GpuColumnarExchange gpuhashpartitioning(cast(user_id#83458 as int), 200), ENSURE_REQUIREMENTS, [plan_id=394739]
         +- GpuProject [gpucoalesce(user_id#84619, cast(user_id#84639 as string)) AS user_id#83458, if ((gpucoalesce(is_deleted#84640, true) AND NOT (cast(user_id#84619 as int) = 117563))) gdpr-deleted else gpucoalesce(email_address#84631, email_address#84646) AS email_address#83462]
            +- GpuRowToColumnar targetsize(268435456)
               +- *(92) BroadcastHashJoin [org_name#84630], [org_name#84647], LeftOuter, BuildRight, false
                  :- GpuColumnarToRow false
                  :  +- GpuFilter gpuisnotnull(gpucoalesce(user_id#84619, cast(user_id#84639 as string)))
                  :     +- GpuShuffledHashJoin [user_id#84619], [cast(user_id#84639 as string)], FullOuter, GpuBuildRight, false
                  :        :- GpuProject [user_id#84619, org_name#84630, email_address#84631]
                  :        :  +- GpuCoalesceBatches targetsize(268435456)

First part of stack trace:

 at org.apache.spark.sql.execution.joins.ExecutorBroadcast$.getShuffleIdFromPlan(ExecutorBroadcast.scala:189)
        at org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.executorBroadcast$lzycompute(BroadcastHashJoinExec.scala:73)
        at org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.executorBroadcast(BroadcastHashJoinExec.scala:71)
        at org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.prepareBroadcast(BroadcastHashJoinExec.scala:284)
        at org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.prepareRelation(BroadcastHashJoinExec.scala:304)
        at org.apache.spark.sql.execution.joins.HashJoin.codegenOuter(HashJoin.scala:451)
        at org.apache.spark.sql.execution.joins.HashJoin.codegenOuter$(HashJoin.scala:450)
        at org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.codegenOuter(BroadcastHashJoinExec.scala:53)
        at org.apache.spark.sql.execution.joins.HashJoin.doConsume(HashJoin.scala:359)
        at org.apache.spark.sql.execution.joins.HashJoin.doConsume$(HashJoin.scala:356)
        at org.apache.spark.sql.execution.joins.BroadcastHashJoinExec.doConsume(BroadcastHashJoinExec.scala:53)
        at org.apache.spark.sql.execution.CodegenSupport.consume(WholeStageCodegenExec.scala:199)
        at org.apache.spark.sql.execution.CodegenSupport.consume$(WholeStageCodegenExec.scala:154)
        at org.apache.spark.sql.execution.InputAdapter.consume(WholeStageCodegenExec.scala:503)
        at org.apache.spark.sql.execution.InputRDDCodegen.doProduce(WholeStageCodegenExec.scala:490)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants