Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
ARROW-530: [C++/Python] Provide subpools for better memory allocation … #2057
Currently we can only track the amount of bytes allocated by the main memory pool or the alternative jemalloc implementation. To better understand certain situation, we should provide a MemoryPool proxy implementation that tracks only the amount of memory that was made through its direct calls but delegates the actual allocation to an underlying pool.
import pyarrow as pa
def report(pool, proxy_pool):
pool = pa.default_memory_pool()
xhochy left a comment
Implementation-wise this looks good. Also for me it seems as this pool will already tracks only the allocations made for it. To complete this PR, please add some C++ unit tests that check the implementation.
Also have a look at Travis which reports some linting errors.
@@ Coverage Diff @@ ## master #2057 +/- ## ========================================== - Coverage 87.44% 86.35% -1.09% ========================================== Files 189 230 +41 Lines 29376 40513 +11137 ========================================== + Hits 25687 34987 +9300 - Misses 3689 5526 +1837
changed the title
ARROW-530: C++/Python: Provide subpools for better memory allocation …
May 20, 2018
Looks good except for a small formatting issue: https://travis-ci.org/apache/arrow/jobs/387368172#L1227
Would merge on a green build.
It seems tests in Travis failed due to timeouts: https://travis-ci.org/apache/arrow/jobs/388699104#L4333, https://travis-ci.org/apache/arrow/jobs/388699106#L523.