How to profile only certain part of Python process?

[EricCousineau-TRI]

Not sure if this has already been asked, but was curious if there's a best way to only profile part of process - I'd also be willing to make code modifications if need be.

Motivation is two-fold:

• I would like to ignore import times for certain things.
• I am using Bazel w/ additional wrappers. Even using --subprocesses, I found it hard to see the timings I expected (multiple hops with execv, maybe?).

I found this works 'aight, albeit somewhat sloppy:

from contextlib import contextmanager
import os
import signal
import subprocess


@contextmanager
def use_py_spy(output_file, *, sudo=False):
    """Use py-spy in specific context."""
    args = ["py-spy", "record", "-o", output_file, "--pid", str(os.getpid())]
    if sudo:
        args = ["sudo"] + args
    p = subprocess.Popen(args)
    # TODO(eric.cousineau): Startup time of py-spy may lag behind other
    # instructions. However, maybe can assume this isn't critical for profiling
    # purposes?
    yield
    p.send_signal(signal.SIGINT)
    p.wait()


def main():
    with use_py_spy("/tmp/crazy_stuff.svg"):
       do_crazy_stuff()

...

See usecase for CPython API extension stuff:

• https://github.com/EricCousineau-TRI/repro/tree/2e3865a7aefe8adc19a6ff69e84025def03da7fd/drake_stuff/python_profiling (relates https://drake.mit.edu/ )

Thanks for the awesome tool!

[olejorgenb]

Maybe https://github.com/joerick/pyinstrument is more suited for this? It is designed to be called from within python.