Fixed resources used/wasted computation for spark jobs - (Depends on Custom SHS - Requires peakJvmUsedMemory metric) #287
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shankar37
reviewed
Sep 12, 2017
| val totalExecutorTaskTimeMillis = totalExecutorTaskTimeMillisOf(data) | ||
|
|
||
| val resourcesAllocatedForUse = | ||
| aggregateresourcesAllocatedForUse(executorInstances, executorMemoryBytes, applicationDurationMillis) |
There was a problem hiding this comment.
Allocated resources should take care of the dynamic allocation.
| var sumResourceUsage: BigInt = 0 | ||
| executorSummaries.foreach( | ||
| executorSummary => { | ||
| var memUsed: Long = executorSummary.peakJvmUsedMemory.getOrElse(JVM_USED_MEMORY, 0) //+ MemoryFormatUtils.stringToBytes("300M") |
There was a problem hiding this comment.
We might want to add a buffer on top of the peak memory.
There was a problem hiding this comment.
The variable "resourcesActuallyUsedWithBuffer" (line no: 64) is the resource usage with buffer and for calculating resources wasted, this value is being considered.
skakker
force-pushed
the
ResourceUsage
branch
from
7f7717f
to
fc9b866
Compare
|
I think it would good if you can provide a brief description of this PR. It helps in the review. |
|
@shkhrgpt: Done |
akshayrai
approved these changes
Jan 5, 2018
akshayrai
changed the title
skakker
force-pushed
the
ResourceUsage
branch
from
b81a2b0
to
9dfcc1e
Compare
akshayrai
pushed a commit
that referenced
this pull request
Feb 21, 2018
…Custom SHS - Requires peakJvmUsedMemory metric) (#287)
akshayrai
pushed a commit
that referenced
this pull request
Feb 27, 2018
…Custom SHS - Requires peakJvmUsedMemory metric) (#287)
akshayrai
pushed a commit
that referenced
this pull request
Mar 6, 2018
…Custom SHS - Requires peakJvmUsedMemory metric) (#287)
arpang
pushed a commit
to arpang/dr-elephant
that referenced
this pull request
Mar 14, 2018
…Custom SHS - Requires peakJvmUsedMemory metric) (linkedin#287)
akshayrai
pushed a commit
that referenced
this pull request
Mar 19, 2018
…Custom SHS - Requires peakJvmUsedMemory metric) (#287)
akshayrai
pushed a commit
that referenced
this pull request
Mar 19, 2018
…Custom SHS - Requires peakJvmUsedMemory metric) (#287)
akshayrai
pushed a commit
that referenced
this pull request
Mar 30, 2018
…Custom SHS - Requires peakJvmUsedMemory metric) (#287)
akshayrai
pushed a commit
that referenced
this pull request
Apr 6, 2018
…Custom SHS - Requires peakJvmUsedMemory metric) (#287)
akshayrai
pushed a commit
that referenced
this pull request
May 21, 2018
…Custom SHS - Requires peakJvmUsedMemory metric) (#287)
pralabhkumar
pushed a commit
to pralabhkumar/dr-elephant
that referenced
this pull request
Aug 31, 2018
…Custom SHS - Requires peakJvmUsedMemory metric) (linkedin#287)
varunsaxena
pushed a commit
that referenced
this pull request
Oct 16, 2018
…Custom SHS - Requires peakJvmUsedMemory metric) (#287)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixed the calculation of resources used/wastage, now the resources used is calculated by summing up the resources used by each executor as opposed to before where the executors were assumed to use the entire memory allocated for them. The resources allocated for use is also calculated by multiplying the time spent by each executor with the total memory allocated for an executor, and then summing them up.