Memory profiling on python

[arunthamma]

Hi,

I have started using pyroscope for profiling cloud applications developed using python and it's fantastic. Currently I see that we are only profiling CPU. Can I get some information on integrating memory//io profiling with pyroscope.

Regards.

[petethepig]

Hi @arunthamma Thanks for submitting this! We're planning to look into it at some point. There's an issue for it where we're going to track progress #316 but we don't have a timeline at the moment. Any pointers on how this can be implemented (maybe similar projects that do this) are very much appreciated!

[Rperry2174]

Was talking to someone who suggested we check out https://github.com/mgedmin/objgraph. Looking at there docs it seems like we could definitely turn the output into a flamegraph.

The biggest concern is the amount of overhead that this would add to our Python integration

[jkgenser]

Is it possible to add a feature matrix somewhere in the documentation about which profiling type are supported for which runtimes? Since the headline advertising says includes memory profiling and it's wasn't explicit that memory profiling wasn't supported for pyroscope until digging into github issues here

[petethepig]

@jkgenser That is a great idea, we'll add that to the website soon, I made an issue #474

[nixjdm]

Hey, Python developer here. I've just been pointed to check out this project, looks great! I know of another tool I'd suggest checking out.

I used py-spy (directly) for some profiling needs a while back, but it couldn't provide all that I wanted. Eventually I found https://github.com/plasma-umass/scalene and have been using it since. It can profile memory usage. Check out the profiler comparison table they have and sample output in their README.

It doesn't auto-generate a flame graph (but it's a "good first issue" plasma-umass/scalene#33). Like py-spy, you'd run it via CLI, which I'd think is how you're using py-spy currently. Hopefully they'll open up a Python API not long from now too.

[Rperry2174]

Adding https://github.com/bloomberg/memray as a potential solution here to be able to support memory profiling

[dakshvar22]

Hi, any update on this? Is this feature on the roadmap for the project?

[ducminhle]

Hi, any update? Are you waiting for Python 3.12 and develop this feature for Python 3.12 and newer versions?

[Rperry2174]

@ducminhle still looking for suitable memory profilers out there to support ingesting data from:
bloomberg/memray#445

[mrchi]

Looking forward to using this feature as soon as possible.

[yaroslavkasatikov]

dear team,

Do you have any updates here?

I found the article about phlare https://martinheinz.dev/blog/89 (which seems to be ancestor of Pyroscope). They discussed about python memory and shared this screenshot -

)

And that's that I could find related to python memory continues profiling.

May be anyone knows any other solution? Will be appreciative for any advice :(

[korniltsev]

Have you tried pypprof ? I've never tried it, but according to the project description it should work.

[yaroslavkasatikov]

Hey @korniltsev

Yes, I have tried it and it works fine with one detail: you should use Python 3.11 for it.
According the article I have shared, I can tell you that it works with Pyroscope too.

So it can be great if you will try to take pypprof profiler for memory profiling in mainstream.
There is the sad new here.. Pypprof last commit was done several ages ago:)