Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-34087][FOLLOW-UP][SQL] Manage ExecutionListenerBus register inside itself #31919

Closed
wants to merge 2 commits into from
Closed

[SPARK-34087][FOLLOW-UP][SQL] Manage ExecutionListenerBus register inside itself #31919

wants to merge 2 commits into from

Conversation

Ngone51
Copy link
Member

@Ngone51 Ngone51 commented Mar 22, 2021

What changes were proposed in this pull request?

Move ExecutionListenerBus register (both ListenerBus and ContextCleaner register) into itself.

Also with a minor change that put registerSparkListenerForCleanup to a better place.

Why are the changes needed?

improve code

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass existing tests.

@Ngone51
Copy link
Member Author

Ngone51 commented Mar 22, 2021

cc @mridulm @HyukjinKwon

dongjoon-hyun
dongjoon-hyun reviewed Mar 22, 2021
View reviewed changes
core/src/main/scala/org/apache/spark/ContextCleaner.scala
private def registerForCleanup(objectForCleanup: AnyRef, task: CleanupTask): Unit = {
referenceBuffer.add(new CleanupTaskWeakReference(task, objectForCleanup, referenceQueue))
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @Ngone51 . Could you revert the change in this file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @dongjoon-hyun, this change intends to let the newly added registerSparkListenerForCleanup() follow the same placement with other register...ForCleanup methods. This would make the ContextCleaner looks neater.

This is a style mistake made in the original PR so I fixed it in this follow-up PR. Does it sound ok to you?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it.

@HyukjinKwon
Copy link
Member

cc @mridulm

@Ngone51
Copy link
Member Author

Ngone51 commented Mar 22, 2021

This PR addressed the comment from https://github.com/apache/spark/pull/31881/files#r597370114

HyukjinKwon
HyukjinKwon approved these changes Mar 22, 2021
View reviewed changes
Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks fine to me

@SparkQA
Copy link

SparkQA commented Mar 22, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40907/

@SparkQA
Copy link

SparkQA commented Mar 22, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40907/

@SparkQA
Copy link

SparkQA commented Mar 22, 2021

Test build #136325 has finished for PR 31919 at commit 53be3fa.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Mar 22, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40917/

@SparkQA
Copy link

SparkQA commented Mar 22, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40917/

@SparkQA
Copy link

SparkQA commented Mar 22, 2021

Test build #136333 has finished for PR 31919 at commit dfd7d38.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Mar 23, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40966/

@SparkQA
Copy link

SparkQA commented Mar 23, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40966/

@SparkQA
Copy link

SparkQA commented Mar 23, 2021

Test build #136382 has finished for PR 31919 at commit ae5d5d6.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

GA passed, merging to master, thanks!

@cloud-fan cloud-fan closed this in e00afd3 Mar 23, 2021
@Ngone51
Copy link
Member Author

Ngone51 commented Mar 23, 2021

Thanks all!

@Ngone51 Ngone51 deleted the SPARK-34087-followup branch March 23, 2021 08:27
@mridulm
Copy link
Contributor

mridulm commented Mar 25, 2021

Late LGTM, thanks for fixing this @Ngone51 !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dongjoon-hyun dongjoon-hyun dongjoon-hyun left review comments

@cloud-fan cloud-fan cloud-fan approved these changes

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

Assignees
No one assigned
Labels
CORE SQL
Projects
None yet
Milestone
No milestone
6 participants
@Ngone51 @HyukjinKwon @SparkQA @cloud-fan @mridulm @dongjoon-hyun