Skip to content
This repository has been archived by the owner. It is now read-only.

"idle" detalization #41

Closed
fillest opened this issue Oct 30, 2016 · 2 comments
Closed

"idle" detalization #41

fillest opened this issue Oct 30, 2016 · 2 comments

Comments

@fillest
Copy link

@fillest fillest commented Oct 30, 2016

Do you have any plans to support sampling "idle" call stacks too?

pyflame's design looks the best for me, making it potentially the best sampling python profiler, but lack of "idle" detalization is currently a show-stopper, as I'm working on web software that performs various IO (e.g. networking to several servers, locks) and I need to know which calls caused the extra latency and so on ("off-cpu").
Another story is threading support but as I see from #13 it is already planned, isn't it?

I had to make a custom profiler for running it continuously in production with a very low overhead, it works good and proved itself useful but I'm struggling with releasing it to the public because it needs a nice documentation (a complication: English is not my native) and quite more polishing and so on, and these are the boring demotivating tasks for me alone. So when I read the blog post about pyflame, I thought I could abandon my project for a clearly potentially superior one (it's sad but relieving enough). So now I need to deside :) Probably I could try to contribute, though I'm new to Python internals (and to ptrace-related things too). All my experience with Python internals is reducing the GIL impact for my profiler sampling loop - it runs in a thread, and I reduced the GIL-related calls to the minimum (using Cython tricks) as they turned out to be very expensive (I profiled it with vtune).

@eklitzke
Copy link
Collaborator

@eklitzke eklitzke commented Oct 31, 2016

Hi @fillest ,

Thanks for the feedback. This is what I intend to implement in #13. For that reason I'm going to close this issue, as I consider it a duplicate. But let me give you some more details.

Currently Pyflame just finds the currently running thread, and gets its stack. It does this by using a global variable in the Python interpreter called _PyThreadState_Current which points to the active thread. As you've pointed out, a lot of applications spend most of their time being idle, and it's often useful to know why they're idle.

The task in #13 is to change the Pyflame internals such that Pyflame can walk all of the threads without using _PyThreadState_Current. With this code change, reporting details on idle time would be pretty easy.

This is probably the next major feature I'll implement, after merging #42 which fixes #34.

Loading

@fillest
Copy link
Author

@fillest fillest commented Oct 31, 2016

Good news, thanks!

Loading

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants