Enhance the CachingMemoryManager to possibly handle high memory pressure #188
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
WilliamTambellini
force-pushed
the
CMM
branch
from
c555931
to
ee66782
Compare
jacobkahn
requested changes
Oct 14, 2020
WilliamTambellini
force-pushed
the
CMM
branch
from
ee66782
to
8cb8a74
Compare
WilliamTambellini
force-pushed
the
CMM
branch
2 times, most recently
from
bed2c60
to
112cab8
Compare
WilliamTambellini
force-pushed
the
CMM
branch
from
112cab8
to
6a4dd1c
Compare
There was a problem hiding this comment.
@jacobkahn has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
jacobkahn
approved these changes
Oct 24, 2020
|
@WilliamTambellini - thank again for the contribution - looks good to me - do me a favor and rebase on matter then we'll be ready to merge. |
WilliamTambellini
force-pushed
the
CMM
branch
from
6a4dd1c
to
e55bb48
Compare
|
@WilliamTambellini has updated the pull request. You must reimport the pull request before landing. |
done |
WilliamTambellini
force-pushed
the
CMM
branch
from
e55bb48
to
ed35c16
Compare
|
@WilliamTambellini has updated the pull request. You must reimport the pull request before landing. |
| target_link_libraries(Train flashlight-app-asr) | ||
| target_link_libraries(Test flashlight-app-asr) | ||
| target_link_libraries(Decoder flashlight-app-asr) | ||
| target_link_libraries(Train flashlight-app-asr ${CMAKE_DL_LIBS}) |
There was a problem hiding this comment.
@jacobkahn This fixes circleCI link errors:
[ 8%] Linking CXX executable Train
/usr/bin/ld: CMakeFiles/Train.dir/flashlight/common/Plugin.cpp.o: undefined reference to symbol 'dlclose@@GLIBC_2.2.5'
//lib/x86_64-linux-gnu/libdl.so.2: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
Enhance the CachingMemoryManager to possibly handle high memory pressure by adding 2 options. Note: the default behavior is unchanged.
WilliamTambellini
force-pushed
the
CMM
branch
from
ed35c16
to
3943b2a
Compare
|
@WilliamTambellini has updated the pull request. You must reimport the pull request before landing. |
There was a problem hiding this comment.
@jacobkahn has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
@jacobkahn has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@jacobkahn merged this pull request in 1b5796e. |
This was referenced Oct 26, 2020
Closed
avidov
pushed a commit
to avidov/flashlight
that referenced
this pull request
Jan 14, 2021
Add flags to set configurable limits of the CachingMemoryManager. These flags allow customization for different workloads. This diff changes the default recycling size limit for training to 256MB. Following investigation of OOM on dynamic batching nd based on past allocation analyses we found that this limit works well. Big thanks for WilliamTambellini@ for his work on improving the memory manager capability in handling flexible workload: flashlight#188
avidov
pushed a commit
to avidov/flashlight
that referenced
this pull request
Jan 14, 2021
Add flags to set configurable limits of the CachingMemoryManager. These flags allow customization for different workloads. This diff changes the default recycling size limit for training to 256MB. Following investigation of OOM on dynamic batching nd based on past allocation analyses we found that this limit works well. Big thanks for WilliamTambellini@ for his work on improving the memory manager capability in handling flexible workload: flashlight#188
avidov
pushed a commit
to avidov/flashlight
that referenced
this pull request
Jan 14, 2021
Add flags to set configurable limits of the CachingMemoryManager. These flags allow customization for different workloads. This diff changes the default recycling size limit for training to 256MB. Following investigation of OOM on dynamic batching nd based on past allocation analyses we found that this limit works well. Big thanks for WilliamTambellini@ for his work on improving the memory manager capability in handling flexible workload: flashlight#188
avidov
pushed a commit
to avidov/flashlight
that referenced
this pull request
Jan 20, 2021
Add flags to set configurable limits of the CachingMemoryManager. These flags allow customization for different workloads. This diff changes the default recycling size limit for training to 256MB. Following investigation of OOM on dynamic batching nd based on past allocation analyses we found that this limit works well. Big thanks for WilliamTambellini@ for his work on improving the memory manager capability in handling flexible workload: flashlight#188
jacobkahn
pushed a commit
to flashlight/text
that referenced
this pull request
Mar 25, 2022
…ure (#188) Summary: Enhance the CachingMemoryManager to handle high memory pressure by adding 2 options. Note: the default behavior is unchanged. Add a new API to MemoryManagerAdapter to setOption(). Side: remove some glog includes failing compilation if glog-dev not installed locally **Original Issue**: flashlight/flashlight#180 closes flashlight/flashlight#180 The CachingMemoryManager being greedy in memory. This PR adds 2 options to mitigate this. ### Test Plan (required) Implemented a new unitest in order to show OOM with the default CachingMM and no OOM if the right option is set. Pull Request resolved: flashlight/flashlight#188 Reviewed By: vineelpratap Differential Revision: D24435873 Pulled By: jacobkahn fbshipit-source-id: 99186439c1e306ad987079cc326ab68fe1f028fc
jacobkahn
pushed a commit
to flashlight/text
that referenced
this pull request
Mar 25, 2022
…ure (#188) Summary: Enhance the CachingMemoryManager to handle high memory pressure by adding 2 options. Note: the default behavior is unchanged. Add a new API to MemoryManagerAdapter to setOption(). Side: remove some glog includes failing compilation if glog-dev not installed locally **Original Issue**: flashlight/flashlight#180 closes flashlight/flashlight#180 The CachingMemoryManager being greedy in memory. This PR adds 2 options to mitigate this. ### Test Plan (required) Implemented a new unitest in order to show OOM with the default CachingMM and no OOM if the right option is set. Pull Request resolved: flashlight/flashlight#188 Reviewed By: vineelpratap Differential Revision: D24435873 Pulled By: jacobkahn fbshipit-source-id: 99186439c1e306ad987079cc326ab68fe1f028fc
jacobkahn
pushed a commit
to flashlight/text
that referenced
this pull request
Mar 25, 2022
…ure (#188) Summary: Enhance the CachingMemoryManager to handle high memory pressure by adding 2 options. Note: the default behavior is unchanged. Add a new API to MemoryManagerAdapter to setOption(). Side: remove some glog includes failing compilation if glog-dev not installed locally **Original Issue**: flashlight/flashlight#180 closes flashlight/flashlight#180 The CachingMemoryManager being greedy in memory. This PR adds 2 options to mitigate this. ### Test Plan (required) Implemented a new unitest in order to show OOM with the default CachingMM and no OOM if the right option is set. Pull Request resolved: flashlight/flashlight#188 Reviewed By: vineelpratap Differential Revision: D24435873 Pulled By: jacobkahn fbshipit-source-id: 99186439c1e306ad987079cc326ab68fe1f028fc
jacobkahn
pushed a commit
to flashlight/text
that referenced
this pull request
Mar 25, 2022
…ure (#188) Summary: Enhance the CachingMemoryManager to handle high memory pressure by adding 2 options. Note: the default behavior is unchanged. Add a new API to MemoryManagerAdapter to setOption(). Side: remove some glog includes failing compilation if glog-dev not installed locally **Original Issue**: flashlight/flashlight#180 closes flashlight/flashlight#180 The CachingMemoryManager being greedy in memory. This PR adds 2 options to mitigate this. ### Test Plan (required) Implemented a new unitest in order to show OOM with the default CachingMM and no OOM if the right option is set. Pull Request resolved: flashlight/flashlight#188 Reviewed By: vineelpratap Differential Revision: D24435873 Pulled By: jacobkahn fbshipit-source-id: 99186439c1e306ad987079cc326ab68fe1f028fc
jacobkahn
pushed a commit
to flashlight/text
that referenced
this pull request
Mar 25, 2022
…ure (#188) Summary: Enhance the CachingMemoryManager to handle high memory pressure by adding 2 options. Note: the default behavior is unchanged. Add a new API to MemoryManagerAdapter to setOption(). Side: remove some glog includes failing compilation if glog-dev not installed locally **Original Issue**: flashlight/flashlight#180 closes flashlight/flashlight#180 The CachingMemoryManager being greedy in memory. This PR adds 2 options to mitigate this. ### Test Plan (required) Implemented a new unitest in order to show OOM with the default CachingMM and no OOM if the right option is set. Pull Request resolved: flashlight/flashlight#188 Reviewed By: vineelpratap Differential Revision: D24435873 Pulled By: jacobkahn fbshipit-source-id: 99186439c1e306ad987079cc326ab68fe1f028fc
jacobkahn
pushed a commit
to flashlight/sequence
that referenced
this pull request
Oct 13, 2022
…ure (#188) Summary: Enhance the CachingMemoryManager to handle high memory pressure by adding 2 options. Note: the default behavior is unchanged. Add a new API to MemoryManagerAdapter to setOption(). Side: remove some glog includes failing compilation if glog-dev not installed locally **Original Issue**: flashlight/flashlight#180 closes flashlight/flashlight#180 The CachingMemoryManager being greedy in memory. This PR adds 2 options to mitigate this. ### Test Plan (required) Implemented a new unitest in order to show OOM with the default CachingMM and no OOM if the right option is set. Pull Request resolved: flashlight/flashlight#188 Reviewed By: vineelpratap Differential Revision: D24435873 Pulled By: jacobkahn fbshipit-source-id: 99186439c1e306ad987079cc326ab68fe1f028fc
jacobkahn
pushed a commit
to flashlight/sequence
that referenced
this pull request
Oct 13, 2022
…ure (#188) Summary: Enhance the CachingMemoryManager to handle high memory pressure by adding 2 options. Note: the default behavior is unchanged. Add a new API to MemoryManagerAdapter to setOption(). Side: remove some glog includes failing compilation if glog-dev not installed locally **Original Issue**: flashlight/flashlight#180 closes flashlight/flashlight#180 The CachingMemoryManager being greedy in memory. This PR adds 2 options to mitigate this. ### Test Plan (required) Implemented a new unitest in order to show OOM with the default CachingMM and no OOM if the right option is set. Pull Request resolved: flashlight/flashlight#188 Reviewed By: vineelpratap Differential Revision: D24435873 Pulled By: jacobkahn fbshipit-source-id: 99186439c1e306ad987079cc326ab68fe1f028fc
jacobkahn
pushed a commit
to flashlight/sequence
that referenced
this pull request
Oct 13, 2022
…ure (#188) Summary: Enhance the CachingMemoryManager to handle high memory pressure by adding 2 options. Note: the default behavior is unchanged. Add a new API to MemoryManagerAdapter to setOption(). Side: remove some glog includes failing compilation if glog-dev not installed locally **Original Issue**: flashlight/flashlight#180 closes flashlight/flashlight#180 The CachingMemoryManager being greedy in memory. This PR adds 2 options to mitigate this. ### Test Plan (required) Implemented a new unitest in order to show OOM with the default CachingMM and no OOM if the right option is set. Pull Request resolved: flashlight/flashlight#188 Reviewed By: vineelpratap Differential Revision: D24435873 Pulled By: jacobkahn fbshipit-source-id: 99186439c1e306ad987079cc326ab68fe1f028fc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Enhance the CachingMemoryManager to handle high memory pressure by adding 2 options.
Note: the default behavior is unchanged.
Add a new API to MemoryManagerAdapter to setOption().
Side: remove some glog includes failing compilation if glog-dev not installed locally
Original Issue:
#180
closes #180
Summary
The CachingMemoryManager being greedy in memory. This PR adds 2 options to mitigate this.
Test Plan (required)
Implemented a new unitest in order to show OOM with the default CachingMM and no OOM if the right option is set.