HADOOP-16854. ABFS: Fix for the OutOfMemoryException in AbfsOutputStream #2014
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
bilaharith
force-pushed
the
HADOOP-16854-oomfix
branch
from
a134eaa
to
cb5fad0
Compare
bilaharith
force-pushed
the
HADOOP-16854-oomfix
branch
from
cb5fad0
to
f71c48b
Compare
|
💔 -1 overall
This message was automatically generated. |
|
Driver test results using accounts in Central India Account with HNS Support Account without HNS support |
steveloughran
requested changes
Jun 12, 2020
| @@ -172,6 +172,12 @@ | |||
| <groupId>com.google.guava</groupId> | |||
| <artifactId>guava</artifactId> | |||
| </dependency> | |||
| <dependency> | |||
There was a problem hiding this comment.
- can you add to hadoop project pom and then refer here. its how we guarantee consistent versions.
- do we really need to add a new JAR into production just for annotations? if that is all it is for, maybe we could somehow avoid doing that
which annotations is it actually for? as VisibleForTesting is in guava
| import static org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.MIN_VALUE_MAX_AZURE_WRITE_MEM_USAGE_PERCENTAGE; | ||
|
|
||
| /** | ||
| * Pool for byte[] |
There was a problem hiding this comment.
. at ehd for javadoc in java 8
| } | ||
|
|
||
| private synchronized boolean isPossibleToIssueNewBuffer() { | ||
| Runtime rt = Runtime.getRuntime(); |
There was a problem hiding this comment.
could this be done outside a sync block? it probably does a JNI call.
| private synchronized boolean isPossibleToIssueNewBuffer() { | ||
| Runtime rt = Runtime.getRuntime(); | ||
| int bufferCountByMaxFreeBuffers = | ||
| maxBuffersToPool + rt.availableProcessors(); |
There was a problem hiding this comment.
store availableProcessors in constructor.
|
|
||
| initWriteBufferPool(abfsOutputStreamContext); | ||
|
|
||
| ThreadFactory daemonThreadFactory = new ThreadFactory() { |
There was a problem hiding this comment.
there's some hadoop thread factory you should be able to lift. Also: name the threads
| return null; | ||
| })); | ||
| } | ||
| for (Future<Void> futureTask : futureTasks) { |
There was a problem hiding this comment.
see if you can use org.apache.hadoop.fs.impl.FutureIOSupport here. And somewhere there's a method to block waiting for futures to complete without doing it sequentally; I believe it is faster
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently in environments where memory is restricted, It is observed at times a large number of buffers needed for the execution and the same and are kept within the bufferpool for ever this lead to out Of Memory exceptions.
This change addresses certain improvemnts to AbfsOutputStream which help fix the issue.
Driver test results using accounts in Central India
mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Account with HNS Support
[INFO] Tests run: 67, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 426, Failures: 0, Errors: 0, Skipped: 66
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24
Account without HNS support
[INFO] Tests run: 67, Failures: 0, Errors: 0, Skipped: 0
[WARNING] Tests run: 426, Failures: 0, Errors: 0, Skipped: 240
[WARNING] Tests run: 206, Failures: 0, Errors: 0, Skipped: 24