Systemtap and DTrace support

[bkabrda]

BPO	21590
Nosy	@jcea, @davidmalcolm, @masklinn, @cburroughs, @ambv, @berkerpeksag, @koobs, @stratakis
Files	21590_2.diff create-Include-dir-to-properly-generate-pydtrace_probes.h-file.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ambv'
closed_at = <Date 2016-11-07.22:33:56.533>
created_at = <Date 2014-05-27.13:06:35.744>
labels = ['interpreter-core', 'type-feature', '3.7']
title = 'Systemtap and DTrace support'
updated_at = <Date 2016-11-21.11:48:12.608>
user = 'https://bugs.python.org/bkabrda'

bugs.python.org fields:

activity = <Date 2016-11-21.11:48:12.608>
actor = 'cstratak'
assignee = 'lukasz.langa'
closed = True
closed_date = <Date 2016-11-07.22:33:56.533>
closer = 'lukasz.langa'
components = ['Interpreter Core']
creation = <Date 2014-05-27.13:06:35.744>
creator = 'bkabrda'
dependencies = []
files = ['45264', '45393']
hgrepos = []
issue_num = 21590
keywords = ['patch']
message_count = 14.0
messages = ['219219', '219222', '219342', '233126', '275543', '275623', '279670', '279672', '279932', '280162', '280163', '280254', '280318', '281350']
nosy_count = 13.0
nosy_names = ['jcea', 'durin42', 'SilentGhost', 'dmalcolm', 'xmorel', 'cburroughs', 'lukasz.langa', 'python-dev', 'berker.peksag', 'bkabrda', 'koobs', 'benesch', 'cstratak']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21590'
versions = ['Python 3.6', 'Python 3.7']

[bkabrda]

This is a tracking bug for development of combined systemtap and dtrace patch for Python. The separate patches were submitted at [1] (systemtap) and [2] (dtrace).
Since it was agreed that it'd be best to merge the two patches to reuse as much code as possible between them, I'm opening this issue to track the progress and discuss possible improvements and suggestions.

[1] http://bugs.python.org/issue14776
[2] http://bugs.python.org/issue13405

[jcea]

Bohuslav, do you have Jabber/XMPP?. Mine is "jcea@jabber.org". This thing will require real time communication.

I rather prefer XMPP/Jabber, but I could accept IRC :-)

[bkabrda]

I'd really prefer IRC, if at all possible for you :)
When online, I'm usually on #fedora-python, but I can really come any channel of your choice.

Sorry for the delays in communication, I've got a lot on my plate right now and I'll probably not be very quick to respond

[benesch]

Since it looks like development on this has stalled, I took the liberty of compiling all prior patches

https://github.com/benesch/python-dtrace-tracker

and am actively working on merging them into one superpatch. I am neither a Python nor a DTrace expert, so I'm happy to step aside should one of you more experienced folks have something ready.

A few thoughts so far:

* SPARC support? Can we drop this? It seems to add a bunch of intrusive
  inlined NOPs—that I'm not sure are even necessary.

* @jcea, is this the original DTrace patch? https://github.com/benesch/python-dtrace-tracker/blob/master/dtrace-solaris-oracle-24.patch

  I'm trying to determine where those NOPs came from.

Cheers!

[ambv]

Landed during the 3.6 beta1 sprint:
https://hg.python.org/cpython/rev/d622dbd71f2b

Thanks all for the preliminary tasks! I rebuilt the patch from first principles, reusing large parts of Jesús' work, as well as work by Dave and Nikhil.

Things deliberately left out for simplicity:

• ustack helpers, I have no way of testing them at this point since
  they are Solaris-specific, don't work on macOS or Linux
• PyFrameObject * in function__entry/function__return, this is
  SystemTap-specific and won't compile with DTrace
• SPARC support
• dynamic tracing
• sys module dtrace facility introspection (as described in the documentation, there's facilities in both Linux and macOS to do the same thing)

All of those might be added later.

I also left out instance new/delete probes for now since I had problems making them stable. We might still be able to squeeze them in if we're quick about it.

Dave, if you have an idea how to integrate your FrameObject pointer in a way that compiles for DTrace, too, this is the time to speak up :)

Jesús, if you'd like to add the ustack helper back in a separate patch and add to the documentation how this can be tested, that's great, too. I just didn't feel comfortable committing code myself that I have no way of testing, especially that it required more code (the offsets, including the generated .h here and there, etc.).

[koobs]

@Łukasz in case you're not aware, all koobs-freebsd* bots are DTrace enabled, and can be tested with the custom builder. I'm on IRC (python-dev) if you need anything from me to help progress this

[SilentGhost]

Building the documentation on Ubuntu 16.10, I started getting the following warnings:

/Doc/howto/instrumentation.rst:58: WARNING: Could not lex literal_block as "python3". Highlighting skipped.
/Doc/howto/instrumentation.rst:139: WARNING: Could not lex literal_block as "c". Highlighting skipped.
/Doc/howto/instrumentation.rst:211: WARNING: Could not lex literal_block as "c". Highlighting skipped.
/Doc/howto/instrumentation.rst:328: WARNING: Could not lex literal_block as "c". Highlighting skipped.

I guess the trick of using c for systemtap doesn't work anymore, not entirely sure what's the problem with the python3 highlighting.

[SilentGhost]

Here is the patch that fixes warning in /Doc/howto/instrumentation.rst as well as cleans up a few nits here and there. I've removed attempt at highlighting using c since that doesn't work in practice.

[stratakis]

Fedora so far has been using the systemtap patch downstream from dmalcolm [0].

So for 3.6 by removing the systemtap patch (it cannot be applied anymore cleanly) and by enabling the --with-dtrace configure flag I get this error [1]:
<snip>
make: *** [Makefile:884: Include/pydtrace_probes.h] Error 1
<snip>

If however I touch the pydtrace_probes.h file I get various undefined references [2] and also:
<snip>
make: *** [Makefile:724: Programs/_freeze_importlib] Error 1
<snip>

[0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/00055-systemtap.patch
[1] https://kojipkgs.fedoraproject.org//work/tasks/9656/16279656/build.log
[2] https://kojipkgs.fedoraproject.org//work/tasks/9670/16279670/build.log

[python-dev]

New changeset d05d312161f2 by Berker Peksag in branch '3.6':
Issue bpo-21590: Silence Sphinx warnings in instrumentation.rst
https://hg.python.org/cpython/rev/d05d312161f2

New changeset 442453fa3370 by Berker Peksag in branch 'default':
Issue bpo-21590: Merge from 3.6
https://hg.python.org/cpython/rev/442453fa3370

[berkerpeksag]

Thanks. SilentGhost. I went with ".. highlight:: shell-session" to simplify the patch a bit.

Łukasz, can we close this one and create new issues for further improvements now?

[ambv]

Yes, let's close this.

@Charalampos, the error in your build comes from the /usr/bin/dtrace wrapper:
Traceback (most recent call last):
  File "/usr/bin/dtrace", line 440, in <module>
    sys.exit(main())
  File "/usr/bin/dtrace", line 385, in main
    providers.probe_write(s_filename, filename + suffix)
  File "/usr/bin/dtrace", line 181, in probe_write
    hdr = open(header, mode='w')
FileNotFoundError: [Errno 2] No such file or directory: 'Include/pydtrace_probes.h'

For some reason the path Include/pydtrace_probes.h is not writable. Maybe this is RPM-specific?

[stratakis]

@Łukasz

Dug a bit more to it.

Yes it is RPM specific for that case, in the sense that we create different subfolders for the debug and the normal(optimized) builds under the build/ dir, where the Include directory does not exist. the dtrace wrapper assumes that the directory already exists. The issue of course can be reproduced with any packaging system (or ways to compile python) that follow this approach, although I am not aware currently if this can happen or might be happening anywhere else.

By creating the directory (if it doesn't exist) at the Makefile rule for generating pydtrace_probes.h, this can be circumvented.

Attaching a patch that fixes this issue for consideration.

[stratakis]

@Łukasz

Would it be possible to review the patch?

Or is it preferable to open a new issue?