New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"idle" detalization #41

Closed
fillest opened this Issue Oct 30, 2016 · 2 comments

Comments

Projects
None yet
2 participants
@fillest
Copy link

fillest commented Oct 30, 2016

Do you have any plans to support sampling "idle" call stacks too?

pyflame's design looks the best for me, making it potentially the best sampling python profiler, but lack of "idle" detalization is currently a show-stopper, as I'm working on web software that performs various IO (e.g. networking to several servers, locks) and I need to know which calls caused the extra latency and so on ("off-cpu").
Another story is threading support but as I see from #13 it is already planned, isn't it?

I had to make a custom profiler for running it continuously in production with a very low overhead, it works good and proved itself useful but I'm struggling with releasing it to the public because it needs a nice documentation (a complication: English is not my native) and quite more polishing and so on, and these are the boring demotivating tasks for me alone. So when I read the blog post about pyflame, I thought I could abandon my project for a clearly potentially superior one (it's sad but relieving enough). So now I need to deside :) Probably I could try to contribute, though I'm new to Python internals (and to ptrace-related things too). All my experience with Python internals is reducing the GIL impact for my profiler sampling loop - it runs in a thread, and I reduced the GIL-related calls to the minimum (using Cython tricks) as they turned out to be very expensive (I profiled it with vtune).

@eklitzke

This comment has been minimized.

Copy link
Collaborator

eklitzke commented Oct 31, 2016

Hi @fillest ,

Thanks for the feedback. This is what I intend to implement in #13. For that reason I'm going to close this issue, as I consider it a duplicate. But let me give you some more details.

Currently Pyflame just finds the currently running thread, and gets its stack. It does this by using a global variable in the Python interpreter called _PyThreadState_Current which points to the active thread. As you've pointed out, a lot of applications spend most of their time being idle, and it's often useful to know why they're idle.

The task in #13 is to change the Pyflame internals such that Pyflame can walk all of the threads without using _PyThreadState_Current. With this code change, reporting details on idle time would be pretty easy.

This is probably the next major feature I'll implement, after merging #42 which fixes #34.

@fillest

This comment has been minimized.

Copy link

fillest commented Oct 31, 2016

Good news, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment