Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-26264][CORE]It is better to add @transient to field 'locs' for class ResultTask. #23216

Closed
wants to merge 1 commit into from

Conversation

10110346
Copy link
Contributor

@10110346 10110346 commented Dec 4, 2018

What changes were proposed in this pull request?

The field 'locs' is only used in driver side for class ResultTask, so it is not needed to serialize when sending the ResultTask to executor.
Although it's not very big, it's very frequent, so we can add transient for it like ShuffleMapTask.

How was this patch tested?

Existed unit tests

@SparkQA
Copy link

SparkQA commented Dec 4, 2018

Test build #99660 has finished for PR 23216 at commit b3ede8b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member

wangyum commented Dec 4, 2018

retest this please

@SparkQA
Copy link

SparkQA commented Dec 4, 2018

Test build #99674 has finished for PR 23216 at commit b3ede8b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member

srowen commented Dec 4, 2018

Are you sure it's even a field in the class? it looks like it's only used to define this:

  @transient private[this] val preferredLocs: Seq[TaskLocation] = {
    if (locs == null) Nil else locs.toSet.toSeq
  }

I'd expect Scala would not generate a field. Indeed the thing it is used to make is transient.

@@ -56,7 +56,7 @@ private[spark] class ResultTask[T, U](
stageAttemptId: Int,
taskBinary: Broadcast[Array[Byte]],
partition: Partition,
locs: Seq[TaskLocation],
@transient private var locs: Seq[TaskLocation],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is it var BTW?

@10110346
Copy link
Contributor Author

10110346 commented Dec 5, 2018

Are you sure it's even a field in the class? it looks like it's only used to define this:

  @transient private[this] val preferredLocs: Seq[TaskLocation] = {
    if (locs == null) Nil else locs.toSet.toSeq
  }

I'd expect Scala would not generate a field. Indeed the thing it is used to make is transient.

Yeah, it would not generate a field, thanks @srowen
By the way, is it better to remove transient for ShuffleMapTask?

@srowen
Copy link
Member

srowen commented Dec 5, 2018

I think just leave it. The @transient in ShuffleMapTasks's locs is just superfluous here, not sure it's worth changing.

@10110346
Copy link
Contributor Author

10110346 commented Dec 5, 2018

Ok, I will close this PR, thank you very much

@10110346 10110346 closed this Dec 5, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants