Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Spark 4.0] Account for CommandUtils.uncacheTableOrView signature change. #10863

Merged
merged 4 commits into from
May 30, 2024
Merged

[Spark 4.0] Account for CommandUtils.uncacheTableOrView signature change. #10863

merged 4 commits into from
May 30, 2024

Conversation

mythrocks
Copy link
Collaborator

Fixes #10710.

This commit accounts for the changes in the signature of CommandUtils.uncacheTableOrView in Apache Spark 4.0. (See SPARK-47191.)

@mythrocks
Account for CommandUtils.uncacheTableOrView signature change.
41df548 
Fixes NVIDIA#10710.

This commit accounts for the changes in the signature of `CommandUtils.uncacheTableOrView`
in Apache Spark 4.0. (See [SPARK-47191](apache/spark#45289).)

Signed-off-by: MithunR <mithunr@nvidia.com>
@mythrocks mythrocks self-assigned this May 22, 2024
@mythrocks
Copy link
Collaborator Author

Build

@sameerz sameerz added the Spark 4.0+ Spark 4.0+ issues label May 22, 2024
@mythrocks mythrocks changed the base branch from branch-24.06 to branch-24.08 May 28, 2024 19:33
@mythrocks
Copy link
Collaborator Author

Build

@mythrocks mythrocks requested a review from razajafri May 28, 2024 19:41
@mythrocks
Copy link
Collaborator Author

mythrocks commented May 29, 2024 •
edited
Loading

The CI run failed with the following error. (It looks unrelated to my current change.)

024-05-28T20:46:22.5133881Z [2024-05-28T20:45:49.822Z] [2024-05-28T20:45:38.458Z] [ERROR] /home/ubuntu/spark-rapids/sql-plugin/src/main/spark330db/scala/com/nvidia/spark/rapids/shims/Spark321PlusDBShims.scala:44: Unused import
2024-05-28T20:46:22.5134484Z [2024-05-28T20:45:49.822Z] [2024-05-28T20:45:38.458Z] [ERROR] import org.apache.spark.sql.rapids.execution._
2024-05-28T20:46:22.5135036Z [2024-05-28T20:45:49.822Z] [2024-05-28T20:45:38.458Z] [ERROR]                                              ^
2024-05-28T20:46:22.5136444Z [2024-05-28T20:45:49.822Z] [2024-05-28T20:45:38.458Z] [ERROR] /home/ubuntu/spark-rapids/sql-plugin/src/main/spark330db/scala/org/apache/spark/rapids/execution/shims/GpuSubqueryBroadcastMeta.scala:97: type mismatch;
2024-05-28T20:46:22.5136788Z [2024-05-28T20:45:49.822Z] [2024-05-28T20:45:38.458Z]  found   : Int
2024-05-28T20:46:22.5137168Z [2024-05-28T20:45:49.822Z] [2024-05-28T20:45:38.458Z]  required: Seq[Int]
2024-05-28T20:46:22.5137918Z [2024-05-28T20:45:49.822Z] [2024-05-28T20:45:38.458Z] [ERROR]     GpuSubqueryBroadcastExec(s.name, s.index, s.buildKeys, broadcastBuilder())(
2024-05-28T20:46:22.5138437Z [2024-05-28T20:45:49.822Z] [2024-05-28T20:45:38.458Z] [ERROR]                                        ^
2024-05-28T20:46:22.5140243Z [2024-05-28T20:45:49.822Z] [2024-05-28T20:45:38.458Z] [ERROR] /home/ubuntu/spark-rapids/sql-plugin/src/main/spark330db/scala/org/apache/spark/rapids/execution/shims/GpuSubqueryBroadcastMeta.scala:40: private var broadcastBuilder in class GpuSubqueryBroadcastMeta is never used
2024-05-28T20:46:22.5140860Z [2024-05-28T20:45:49.822Z] [2024-05-28T20:45:38.459Z] [ERROR]   private var broadcastBuilder: () => SparkPlan = _
2024-05-28T20:46:22.5141247Z [2024-05-28T20:45:49.822Z] [2024-05-28T20:45:38.459Z] [ERROR]               ^
2024-05-28T20:46:22.5143138Z [2024-05-28T20:45:49.822Z] [2024-05-28T20:45:38.459Z] [ERROR] /home/ubuntu/spark-rapids/sql-plugin/src/main/spark330db/scala/org/apache/spark/rapids/execution/shims/GpuSubqueryBroadcastMeta.scala:102: private method getBroadcastModeKeyExprs in class GpuSubqueryBroadcastMeta is never used
2024-05-28T20:46:22.5143839Z [2024-05-28T20:45:49.822Z] [2024-05-28T20:45:38.459Z] [ERROR]   private def getBroadcastModeKeyExprs: Option[Seq[Expression]] = {

I'm fairly certain #10933 fixes this.

@mythrocks
Copy link
Collaborator Author

I've merged up for #10933. I'll kick off the CI run shortly.

@mythrocks
Copy link
Collaborator Author

Build

@mythrocks
Copy link
Collaborator Author

This PR still fails on an unrelated error:

2024-05-29T17:47:06.1091539Z [2024-05-29T17:46:28.999Z] [2024-05-29T17:46:20.351Z] [ERROR] /home/ubuntu/spark-rapids/sql-plugin/src/main/spark330db/scala/org/apache/spark/rapids/execution/GpuSubqueryBroadcastMeta.scala:33: GpuSubqueryBroadcastMeta is already defined as class GpuSubqueryBroadcastMeta
2024-05-29T17:47:06.1092047Z [2024-05-29T17:46:28.999Z] [2024-05-29T17:46:20.352Z] [ERROR] class GpuSubqueryBroadcastMeta(
2024-05-29T17:47:06.1092396Z [2024-05-29T17:46:28.999Z] [2024-05-29T17:46:20.352Z] [ERROR]       ^
2024-05-29T17:47:06.1092787Z [2024-05-29T17:46:28.999Z] [2024-05-29T17:46:20.352Z] [ERROR] one error found
2024-05-29T17:47:06.1093385Z [2024-05-29T17:46:28.999Z] [2024-05-29T17:46:20.352Z] [INFO] ------------------------------------------------------------------------

#10945 should resolve it.

@razajafri
Copy link
Collaborator

build

1 similar comment
@razajafri
Copy link
Collaborator

build

@razajafri razajafri merged commit 499a45b into NVIDIA:branch-24.08 May 30, 2024
42 of 44 checks passed
@mythrocks mythrocks mentioned this pull request May 31, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@razajafri razajafri razajafri approved these changes

Assignees

@mythrocks mythrocks

Labels
Spark 4.0+ Spark 4.0+ issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[AUDIT] uncacheTableOrView changed in CommandUtils
3 participants
@mythrocks @razajafri @sameerz