Support pyinstaller binaries

[DavidVentura]

At the moment if you try to run py-spy on a binary created by py-installer you get

Error: Failed to find python version from target process

A trivial fix would be to allow the user to specify the version in case it cannot be detected.
Not entirely sure if the rest would work..

[benfred]

What operating system are you on? I managed to replicate this on OSX, but want to make sure I'm targeting the right platform. Also can you enable logging and paste the output here (export RUST_LOG=info on osx/linux or set RUST_LOG=info on windows cmd.exe then run the command again).

I don't think the rest would work even if we passed the version in - py-spy is failing to find the right libpython binary, so even with the right version it would fail to find the python interpreter.

On OSX pyinstaller is bundling the libpython to something like

 INFO 2018-09-21T19:26:51Z: py_spy::python_spy: map: 0000000100c1c000-0000000100d11000 r-x /Users/ben/code/py-spy/test_programs/dist/test/Python

and we are expecting something like 'libpython' or 'Python.Framework/.../Python'. I'm not sure what this is like on other systems =(.

[DavidVentura]

I'm on debian/centos. Would you mind making a release that allows you to specify the version and the offset for the PyInterpreterState (or something that is useful)? then I could play with this trying to figure out how to get this address out of the static binary in a nice way

[benfred]

I tested py-spy with pyinstaller on centos7, and found two different minor issues.

1. python runs in a subprocess of the pyinstaller executable

It looks like the pyinstaller process forks, and loads up python in the subprocess (while the main process waits for the subprocess to finish). This means that you can't go 'py-spy -- path/to/executable' and have to instead go 'py-spy --pid <pid_of_subprocess>'. Eventually I'd like for py-spy to inspect subprocesses too (to help with multiprocessing etc), but for now you'll have to specify the pid

Note: pyinstaller on OSX doesn't seem to fork like this 🤷‍♂️

2. py-spy isn't finding the libpython binary

This is what's preventing py-spy from getting the version string for you =(.

The problem is that libpython is something like /home/vagrant/dist/test/libpython2.7.so.1.0 or /tmp/_MEIOqzg01/libpython2.7.so.1.0 (depending on whether you're creating a single file or single directory pyinstaller).

we are expecting libpython to be in lib/ or lib64, matching like :

py-spy/src/python_spy.rs

Lines 409 to 410 in 9823477

	let is_python_lib = |pathname: &str| pathname.contains("lib/libpython") ||
	pathname.contains("lib64/libpython");

which isn't the case for pyinstaller here =(

The good news is that is pretty easy to fix, and I'll have a fix for this in the next version (either later today or on sunday).

[benfred]

This commit should fix the failing to find libpython problem with pyinstaller: 064b8e7

This will be in the next release. Note that you still will need to pass the pid of the python subprocess for now.

[benfred]

fixed in v0.1.8

[DavidVentura]

Working great! Thank you

[DavidVentura]

When doing this I see 3 processes functions taking 100% of the time - it doesn't really make any sense to count these in and they make the flamegraph weird

Can you add them to the blacklist ? Should I open another issue?

[benfred]

hmm - marking these as idle seems unideal too though (would show process not being busy, when it's busy in subprocessess). I'm going to add an issue for a config file where you can specify custom functions to exclude.

Also I think eventually it would be good to also sample from subprocesses

[DavidVentura]

In this case, looking at real cpu usage only 1 process is busy. The subprocesses only send a message via a queue at the very end of the processing, so there's also no communication.

…

On September 28, 2018 11:10:05 PM GMT+02:00, Ben Frederickson ***@***.***> wrote: hmm - marking these as idle seems unideal too though (would show process not being busy, when it's busy in subprocessess). I'm going to add an issue for a config file where you can specify custom functions to exclude. Also I think eventually it would be good to also sample from subprocesses -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #42 (comment)