[FLINK-9969][batch] Dispose InMemorySorters created by the UnilateralSortMerger #6479
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tillrohrmann
force-pushed
the
fixYarnBatchMemory
branch
from
6a36633
to
a1a6ca0
Compare
zentol
reviewed
Aug 2, 2018
| if (initialSerializer == null) { | ||
| typeSerializer = typeSerializerFactory.getSerializer(); | ||
| } else { | ||
| typeSerializer = initialSerializer; |
There was a problem hiding this comment.
couldn't we simplify this by always discarding the serializer created in the constructor? We could then remove the initialSerializer field and always create a new serializer here.
There was a problem hiding this comment.
Yes, the initialSerializer is an optimization to decrease the number of created TypeSerializer instances at the cost of a slightly more complicated create method. I guess both ways should work given that the UnilateralSortMerger does not create thousands of InMemorySorters.
zentol
reviewed
Aug 2, 2018
| new DefaultInMemorySorterFactory<>(serializerFactory, comparator, THRESHOLD_FOR_IN_PLACE_SORTING)); | ||
| } | ||
|
|
||
| protected UnilateralSortMerger( |
There was a problem hiding this comment.
why? we might config which InMemorySorterFactory to use if needed. semantically it's no need to be @VisibleForTesting
There was a problem hiding this comment.
I agree with @tisonkun
tisonkun
reviewed
Aug 3, 2018
| import java.util.ArrayList; | ||
| import java.util.Hashtable; | ||
| import java.util.Iterator; | ||
| import java.util.NoSuchElementException; | ||
|
|
There was a problem hiding this comment.
This change can be suppressed if merged.
There was a problem hiding this comment.
You're right it's unrelated. I will remove it from the commit.
tillrohrmann
force-pushed
the
fixYarnBatchMemory
branch
from
a1a6ca0
to
685293d
Compare
tillrohrmann
added a commit
to tillrohrmann/flink
that referenced
this pull request
Aug 3, 2018
…SortMerger This commit changes the behaviour of the UnilateralSortMerger to keep references of the created InMemorySorters in order to explicitly dispse them when the sort merger is closed. This prevents that InMemorySorters leak and block the garbage collection of MemorySegments to which they keep references. This closes apache#6479.
To guard against memory leaks, the Task releases the reference to its AbstractInvokable when it shuts down or cancels. This closes apache#6480.
…SortMerger This commit changes the behaviour of the UnilateralSortMerger to keep references of the created InMemorySorters in order to explicitly dispse them when the sort merger is closed. This prevents that InMemorySorters leak and block the garbage collection of MemorySegments to which they keep references. This closes apache#6479.
tillrohrmann
force-pushed
the
fixYarnBatchMemory
branch
from
685293d
to
0278395
Compare
tillrohrmann
added a commit
to tillrohrmann/flink
that referenced
this pull request
Aug 3, 2018
…SortMerger This commit changes the behaviour of the UnilateralSortMerger to keep references of the created InMemorySorters in order to explicitly dispse them when the sort merger is closed. This prevents that InMemorySorters leak and block the garbage collection of MemorySegments to which they keep references. This closes apache#6479.
tillrohrmann
added a commit
to tillrohrmann/flink
that referenced
this pull request
Aug 3, 2018
…SortMerger This commit changes the behaviour of the UnilateralSortMerger to keep references of the created InMemorySorters in order to explicitly dispse them when the sort merger is closed. This prevents that InMemorySorters leak and block the garbage collection of MemorySegments to which they keep references. This closes apache#6479.
asfgit
pushed a commit
that referenced
this pull request
Aug 3, 2018
…SortMerger This commit changes the behaviour of the UnilateralSortMerger to keep references of the created InMemorySorters in order to explicitly dispse them when the sort merger is closed. This prevents that InMemorySorters leak and block the garbage collection of MemorySegments to which they keep references. This closes #6479.
asfgit
pushed a commit
that referenced
this pull request
Aug 3, 2018
…SortMerger This commit changes the behaviour of the UnilateralSortMerger to keep references of the created InMemorySorters in order to explicitly dispse them when the sort merger is closed. This prevents that InMemorySorters leak and block the garbage collection of MemorySegments to which they keep references. This closes #6479.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
This commit changes the behaviour of the UnilateralSortMerger to keep references of the created
InMemorySorters in order to explicitly dispse them when the sort merger is closed. This prevents
that InMemorySorters leak and block the garbage collection of MemorySegments to which they keep
references.
cc @StephanEwen
Brief change log
InMemorySorterFactoryto makeUnilateralSortMergertestableInMemorySortersinUnilateralSortMergerInMemorySortersbefore releasing memory toMemoryManagerVerifying this change
UnilateralSortMergerTest#testInMemorySorterDisposalDoes this pull request potentially affect one of the following parts:
@Public(Evolving): (no)Documentation