-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-13279] Remove unnecessary duplicate check in addPendingTask fu… #11175
Conversation
Jenkins this is ok to test |
Jenkins, test this please |
if (!list.contains(index)) { | ||
list += index | ||
} | ||
list += index | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It strikes me that addTo
is now only making a negative contribution toward understanding the code -- mostly because the "...if it's not already there" comment is now wrong. I don't see why the handful of uses of addTo
shouldn't just be replaced with, e.g.:
pendingTasksForExecutor.getOrElseUpdate(e.executorId, new ArrayBuffer) += index
3fe1af8
to
a0c3512
Compare
a0c3512
to
26b4370
Compare
Jenkins, test this please |
Test build #51266 has finished for PR 11175 at commit
|
@kayousterhout I'll let you push the button on this one when you're sure but it looks OK to me. |
@kayousterhout did you get some time to look into this? |
This commit removes an unnecessary duplicate check in addPendingTask that meant that scheduling a task set took time proportional to (# tasks)^2. Author: Sital Kedia <skedia@fb.com> Closes #11175 from sitalkedia/fix_stuck_driver. (cherry picked from commit 1e1e31e) Signed-off-by: Kay Ousterhout <kayousterhout@gmail.com>
…nction