[WIP]SPARK-1712: TaskDescription instance is too big causes Spark to hang#677
[WIP]SPARK-1712: TaskDescription instance is too big causes Spark to hang#677witgo wants to merge 4 commits intoapache:masterfrom
Conversation
|
Can one of the admins verify this patch? |
There was a problem hiding this comment.
The reference Executor.scala#L235.may not suitable for here
There was a problem hiding this comment.
@kayousterhout - you added this. Any reason you left the 1024 of wiggle room?
There was a problem hiding this comment.
real example:
serialized task with dependencies |
serialized LaunchTask |
|---|---|
| 4356 | 4797 |
| 2690585 | 2704141 |
| 5238584 | 5264580 |
The difference has been increasing
There was a problem hiding this comment.
ah I see - it's to leave room for the other contents of the message. In the second case here, do you know what is taking all the extra room? 2704141 - 2690585 = ~13KB which is very large for a few string/int fields!
There was a problem hiding this comment.
Yes, the problem is the value of uncertainty
There was a problem hiding this comment.
It would be nicer to use string interpolation here to make it easier to parse.
There was a problem hiding this comment.
The reason for this is to keep the same style of the code.
|
Thanks for tracking down this fix. It looks good - added some small comments. |
witgo
changed the title
witgo
changed the title
witgo
changed the title
|
@pwendell |
3 participants
No description provided.