feat(profiling): new memory profiler #1673
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jd
force-pushed
the
profiling/memalloc
branch
8 times, most recently
from
5b1dde8
to
bb119be
Compare
jd
force-pushed
the
profiling/memalloc
branch
from
bb119be
to
4c17fa8
Compare
This is a new memory allocation profiler that directly leverages the CPython memory API. It tracks memory allocation events.
This adds a environment variable to opt-int to the new memory profiler.
jd
force-pushed
the
profiling/memalloc
branch
from
4c17fa8
to
7148e04
Compare
majorgreys
reviewed
Sep 25, 2020
| # _memalloc buffer to our Recorder. This is fine for now, but we might want to store the nanoseconds | ||
| # timestamp in C and then return it via iter_events. | ||
| return ( | ||
| tuple( |
There was a problem hiding this comment.
Any particular reason here to use tuple?
There was a problem hiding this comment.
They're smaller and we don't need to modify the result, therefore tuple is good enough :)
| ) | ||
|
|
||
| nevents = len(events) | ||
| sampling_ratio_avg = sum(event.capture_pct for event in events) / nevents / 100.0 |
There was a problem hiding this comment.
We could shave off some effort by collapsing these three iterations over events into one.
There was a problem hiding this comment.
You'd think so…
l = list(range(10_000_000))
def v1():
return sum(l), sum(l), sum(l)
def v2():
x = y = z = 0
for v in l:
x += v
y += v
z += v
return x, y, z
assert v1() == v2()
import timeit
print(timeit.timeit("v1()", number=10, globals=globals()))
print(timeit.timeit("v2()", number=10, globals=globals()))v2 is 6 times slower:
$ python3 t.py
2.568303665
14.879553143
So I'm not sure it'd be a win even here 🤔
There was a problem hiding this comment.
Should have actually tested out my suggestion before making it! 🔬
Funny but my initial thought was to use functools.reduce but that's even slower:
def v3():
return functools.reduce(lambda x,y: (x[0]+y, x[1]+y, x[2]+y), l, (0,0,0))
Which takes ~21s.
majorgreys
approved these changes
Sep 29, 2020
Kyle-Verhoog
pushed a commit
that referenced
this pull request
Oct 1, 2020
* feat(profiling): introduce memalloc collector This is a new memory allocation profiler that directly leverages the CPython memory API. It tracks memory allocation events. * feat(profiling): export MemoryAllocSampleEvent to pprof * feat(profiling): allow to use memalloc collector This adds a environment variable to opt-int to the new memory profiler. Co-authored-by: Tahir H. Butt <tahir.butt@datadoghq.com>
Kyle-Verhoog
pushed a commit
that referenced
this pull request
Oct 1, 2020
* feat(profiling): introduce memalloc collector This is a new memory allocation profiler that directly leverages the CPython memory API. It tracks memory allocation events. * feat(profiling): export MemoryAllocSampleEvent to pprof * feat(profiling): allow to use memalloc collector This adds a environment variable to opt-int to the new memory profiler. Co-authored-by: Tahir H. Butt <tahir.butt@datadoghq.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
feat(profiling): introduce memalloc collector
This is a new memory allocation profiler that directly leverages the CPython
memory API. It tracks memory allocation events.
feat(profiling): export MemoryAllocSampleEvent to pprof
feat(profiling): allow to use memalloc collector