Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Systemtap and DTrace support #65789

Closed
bkabrda mannequin opened this issue May 27, 2014 · 14 comments
Closed

Systemtap and DTrace support #65789

bkabrda mannequin opened this issue May 27, 2014 · 14 comments
Assignees
Labels
3.7 interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement

Comments

@bkabrda
Copy link
Mannequin

bkabrda mannequin commented May 27, 2014

BPO 21590
Nosy @jcea, @davidmalcolm, @masklinn, @cburroughs, @ambv, @berkerpeksag, @koobs, @stratakis
Files
  • 21590_2.diff
  • create-Include-dir-to-properly-generate-pydtrace_probes.h-file.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ambv'
    closed_at = <Date 2016-11-07.22:33:56.533>
    created_at = <Date 2014-05-27.13:06:35.744>
    labels = ['interpreter-core', 'type-feature', '3.7']
    title = 'Systemtap and DTrace support'
    updated_at = <Date 2016-11-21.11:48:12.608>
    user = 'https://bugs.python.org/bkabrda'

    bugs.python.org fields:

    activity = <Date 2016-11-21.11:48:12.608>
    actor = 'cstratak'
    assignee = 'lukasz.langa'
    closed = True
    closed_date = <Date 2016-11-07.22:33:56.533>
    closer = 'lukasz.langa'
    components = ['Interpreter Core']
    creation = <Date 2014-05-27.13:06:35.744>
    creator = 'bkabrda'
    dependencies = []
    files = ['45264', '45393']
    hgrepos = []
    issue_num = 21590
    keywords = ['patch']
    message_count = 14.0
    messages = ['219219', '219222', '219342', '233126', '275543', '275623', '279670', '279672', '279932', '280162', '280163', '280254', '280318', '281350']
    nosy_count = 13.0
    nosy_names = ['jcea', 'durin42', 'SilentGhost', 'dmalcolm', 'xmorel', 'cburroughs', 'lukasz.langa', 'python-dev', 'berker.peksag', 'bkabrda', 'koobs', 'benesch', 'cstratak']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue21590'
    versions = ['Python 3.6', 'Python 3.7']

    @bkabrda
    Copy link
    Mannequin Author

    bkabrda mannequin commented May 27, 2014

    This is a tracking bug for development of combined systemtap and dtrace patch for Python. The separate patches were submitted at [1] (systemtap) and [2] (dtrace).
    Since it was agreed that it'd be best to merge the two patches to reuse as much code as possible between them, I'm opening this issue to track the progress and discuss possible improvements and suggestions.

    [1] http://bugs.python.org/issue14776
    [2] http://bugs.python.org/issue13405

    @bkabrda bkabrda mannequin added interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement labels May 27, 2014
    @jcea
    Copy link
    Member

    jcea commented May 27, 2014

    Bohuslav, do you have Jabber/XMPP?. Mine is "jcea@jabber.org". This thing will require real time communication.

    I rather prefer XMPP/Jabber, but I could accept IRC :-)

    @bkabrda
    Copy link
    Mannequin Author

    bkabrda mannequin commented May 29, 2014

    I'd really prefer IRC, if at all possible for you :)
    When online, I'm usually on #fedora-python, but I can really come any channel of your choice.

    Sorry for the delays in communication, I've got a lot on my plate right now and I'll probably not be very quick to respond

    @benesch
    Copy link
    Mannequin

    benesch mannequin commented Dec 27, 2014

    Since it looks like development on this has stalled, I took the liberty of compiling all prior patches

    https://github.com/benesch/python-dtrace-tracker
    

    and am actively working on merging them into one superpatch. I am neither a Python nor a DTrace expert, so I'm happy to step aside should one of you more experienced folks have something ready.

    A few thoughts so far:

    * SPARC support? Can we drop this? It seems to add a bunch of intrusive
      inlined NOPs—that I'm not sure are even necessary.
    
    * @jcea, is this the original DTrace patch? https://github.com/benesch/python-dtrace-tracker/blob/master/dtrace-solaris-oracle-24.patch
    
      I'm trying to determine where those NOPs came from.
    

    Cheers!

    @ambv
    Copy link
    Contributor

    ambv commented Sep 10, 2016

    Landed during the 3.6 beta1 sprint:
    https://hg.python.org/cpython/rev/d622dbd71f2b

    Thanks all for the preliminary tasks! I rebuilt the patch from first principles, reusing large parts of Jesús' work, as well as work by Dave and Nikhil.

    Things deliberately left out for simplicity:

    • ustack helpers, I have no way of testing them at this point since
      they are Solaris-specific, don't work on macOS or Linux
    • PyFrameObject * in function__entry/function__return, this is
      SystemTap-specific and won't compile with DTrace
    • SPARC support
    • dynamic tracing
    • sys module dtrace facility introspection (as described in the documentation, there's facilities in both Linux and macOS to do the same thing)

    All of those might be added later.

    I also left out instance new/delete probes for now since I had problems making them stable. We might still be able to squeeze them in if we're quick about it.

    Dave, if you have an idea how to integrate your FrameObject pointer in a way that compiles for DTrace, too, this is the time to speak up :)

    Jesús, if you'd like to add the ustack helper back in a separate patch and add to the documentation how this can be tested, that's great, too. I just didn't feel comfortable committing code myself that I have no way of testing, especially that it required more code (the offsets, including the generated .h here and there, etc.).

    @ambv ambv self-assigned this Sep 10, 2016
    @koobs
    Copy link

    koobs commented Sep 10, 2016

    @Łukasz in case you're not aware, all koobs-freebsd* bots are DTrace enabled, and can be tested with the custom builder. I'm on IRC (python-dev) if you need anything from me to help progress this

    @SilentGhost
    Copy link
    Mannequin

    SilentGhost mannequin commented Oct 29, 2016

    Building the documentation on Ubuntu 16.10, I started getting the following warnings:

    /Doc/howto/instrumentation.rst:58: WARNING: Could not lex literal_block as "python3". Highlighting skipped.
    /Doc/howto/instrumentation.rst:139: WARNING: Could not lex literal_block as "c". Highlighting skipped.
    /Doc/howto/instrumentation.rst:211: WARNING: Could not lex literal_block as "c". Highlighting skipped.
    /Doc/howto/instrumentation.rst:328: WARNING: Could not lex literal_block as "c". Highlighting skipped.

    I guess the trick of using c for systemtap doesn't work anymore, not entirely sure what's the problem with the python3 highlighting.

    @SilentGhost
    Copy link
    Mannequin

    SilentGhost mannequin commented Oct 29, 2016

    Here is the patch that fixes warning in /Doc/howto/instrumentation.rst as well as cleans up a few nits here and there. I've removed attempt at highlighting using c since that doesn't work in practice.

    @SilentGhost SilentGhost mannequin added the 3.7 label Oct 29, 2016
    @stratakis
    Copy link
    Mannequin

    stratakis mannequin commented Nov 2, 2016

    Fedora so far has been using the systemtap patch downstream from dmalcolm [0].

    So for 3.6 by removing the systemtap patch (it cannot be applied anymore cleanly) and by enabling the --with-dtrace configure flag I get this error [1]:
    <snip>
    make: *** [Makefile:884: Include/pydtrace_probes.h] Error 1
    <snip>

    If however I touch the pydtrace_probes.h file I get various undefined references [2] and also:
    <snip>
    make: *** [Makefile:724: Programs/_freeze_importlib] Error 1
    <snip>

    [0] http://pkgs.fedoraproject.org/cgit/rpms/python3.git/tree/00055-systemtap.patch
    [1] https://kojipkgs.fedoraproject.org//work/tasks/9656/16279656/build.log
    [2] https://kojipkgs.fedoraproject.org//work/tasks/9670/16279670/build.log

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Nov 6, 2016

    New changeset d05d312161f2 by Berker Peksag in branch '3.6':
    Issue bpo-21590: Silence Sphinx warnings in instrumentation.rst
    https://hg.python.org/cpython/rev/d05d312161f2

    New changeset 442453fa3370 by Berker Peksag in branch 'default':
    Issue bpo-21590: Merge from 3.6
    https://hg.python.org/cpython/rev/442453fa3370

    @berkerpeksag
    Copy link
    Member

    berkerpeksag commented Nov 6, 2016

    Thanks. SilentGhost. I went with ".. highlight:: shell-session" to simplify the patch a bit.

    Łukasz, can we close this one and create new issues for further improvements now?

    @ambv
    Copy link
    Contributor

    ambv commented Nov 7, 2016

    Yes, let's close this.

    @Charalampos, the error in your build comes from the /usr/bin/dtrace wrapper:
    Traceback (most recent call last):
      File "/usr/bin/dtrace", line 440, in <module>
        sys.exit(main())
      File "/usr/bin/dtrace", line 385, in main
        providers.probe_write(s_filename, filename + suffix)
      File "/usr/bin/dtrace", line 181, in probe_write
        hdr = open(header, mode='w')
    FileNotFoundError: [Errno 2] No such file or directory: 'Include/pydtrace_probes.h'

    For some reason the path Include/pydtrace_probes.h is not writable. Maybe this is RPM-specific?

    @ambv ambv closed this as completed Nov 7, 2016
    @stratakis
    Copy link
    Mannequin

    stratakis mannequin commented Nov 8, 2016

    @Łukasz

    Dug a bit more to it.

    Yes it is RPM specific for that case, in the sense that we create different subfolders for the debug and the normal(optimized) builds under the build/ dir, where the Include directory does not exist. the dtrace wrapper assumes that the directory already exists. The issue of course can be reproduced with any packaging system (or ways to compile python) that follow this approach, although I am not aware currently if this can happen or might be happening anywhere else.

    By creating the directory (if it doesn't exist) at the Makefile rule for generating pydtrace_probes.h, this can be circumvented.

    Attaching a patch that fixes this issue for consideration.

    @stratakis
    Copy link
    Mannequin

    stratakis mannequin commented Nov 21, 2016

    @Łukasz

    Would it be possible to review the patch?

    Or is it preferable to open a new issue?

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants