Skip to content

gh-125914: Clean up the dtrace code and documentation for the hook is not using #126246

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

gh-125914: Clean up the dtrace code and documentation for the hook is not using #126246

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3ac40ed
gh-125914: Clean up the dtrace code and documentation for the hook is…
Zheaoli Oct 31, 2024
3b2a83c
Update news
Zheaoli Oct 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
345 changes: 44 additions & 301 deletions Doc/howto/instrumentation.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,221 +89,59 @@ Sufficiently modern readelf can print the metadata::

$ readelf -n ./python

Displaying notes found at file offset 0x00000254 with length 0x00000020:
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 2.6.32

Displaying notes found at file offset 0x00000274 with length 0x00000024:
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: df924a2b08a7e89f6e11251d4602022977af2670

Displaying notes found at file offset 0x002d6c30 with length 0x00000144:
Owner Data size Description
stapsdt 0x00000031 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: gc__start
Location: 0x00000000004371c3, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf6
Arguments: -4@%ebx
stapsdt 0x00000030 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: gc__done
Location: 0x00000000004374e1, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bf8
Arguments: -8@%rax
stapsdt 0x00000045 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: function__entry
Location: 0x000000000053db6c, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6be8
Arguments: 8@%rbp 8@%r12 -4@%eax
stapsdt 0x00000046 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: function__return
Location: 0x000000000053dba8, Base: 0x0000000000630ce2, Semaphore: 0x00000000008d6bea
Arguments: 8@%rbp 8@%r12 -4@%eax
Displaying notes found in: .note.gnu.property
Owner Data size Description
GNU 0x00000030 NT_GNU_PROPERTY_TYPE_0
Properties: x86 ISA needed: x86-64-baseline
x86 feature used: x86, XMM
x86 ISA used: x86-64-baseline

Displaying notes found in: .note.gnu.build-id
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)
Build ID: 880aff8660aefcd70df49bad72e8695c3bc65ea9

Displaying notes found in: .note.ABI-tag
Owner Data size Description
GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 4.4.0

Displaying notes found in: .note.stapsdt
Owner Data size Description
stapsdt 0x00000032 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: gc__start
Location: 0x0000000000287b01, Base: 0x00000000004a7595, Semaphore: 0x00000000005ff818
Arguments: -4@%r13d
stapsdt 0x00000030 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: gc__done
Location: 0x0000000000287b44, Base: 0x00000000004a7595, Semaphore: 0x00000000005ff81a
Arguments: -8@%rax
stapsdt 0x00000040 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: import__find__load__start
Location: 0x0000000000297c25, Base: 0x00000000004a7595, Semaphore: 0x00000000005ff81c
Arguments: 8@%rax
stapsdt 0x00000047 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: import__find__load__done
Location: 0x0000000000297c3d, Base: 0x00000000004a7595, Semaphore: 0x00000000005ff81e
Arguments: 8@%rax -4@%edx
stapsdt 0x00000033 NT_STAPSDT (SystemTap probe descriptors)
Provider: python
Name: audit
Location: 0x00000000002d68c4, Base: 0x00000000004a7595, Semaphore: 0x00000000005ff820
Arguments: 8@%r12 8@%r13

The above metadata contains information for SystemTap describing how it
can patch strategically placed machine code instructions to enable the
tracing hooks used by a SystemTap script.


Static DTrace probes
--------------------

The following example DTrace script can be used to show the call/return
hierarchy of a Python script, only tracing within the invocation of
a function called "start". In other words, import-time function
invocations are not going to be listed:

.. code-block:: none

self int indent;

python$target:::function-entry
/copyinstr(arg1) == "start"/
{
self->trace = 1;
}

python$target:::function-entry
/self->trace/
{
printf("%d\t%*s:", timestamp, 15, probename);
printf("%*s", self->indent, "");
printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2);
self->indent++;
}

python$target:::function-return
/self->trace/
{
self->indent--;
printf("%d\t%*s:", timestamp, 15, probename);
printf("%*s", self->indent, "");
printf("%s:%s:%d\n", basename(copyinstr(arg0)), copyinstr(arg1), arg2);
}

python$target:::function-return
/copyinstr(arg1) == "start"/
{
self->trace = 0;
}

It can be invoked like this::

$ sudo dtrace -q -s call_stack.d -c "python3.6 script.py"

The output looks like this:

.. code-block:: none

156641360502280 function-entry:call_stack.py:start:23
156641360518804 function-entry: call_stack.py:function_1:1
156641360532797 function-entry: call_stack.py:function_3:9
156641360546807 function-return: call_stack.py:function_3:10
156641360563367 function-return: call_stack.py:function_1:2
156641360578365 function-entry: call_stack.py:function_2:5
156641360591757 function-entry: call_stack.py:function_1:1
156641360605556 function-entry: call_stack.py:function_3:9
156641360617482 function-return: call_stack.py:function_3:10
156641360629814 function-return: call_stack.py:function_1:2
156641360642285 function-return: call_stack.py:function_2:6
156641360656770 function-entry: call_stack.py:function_3:9
156641360669707 function-return: call_stack.py:function_3:10
156641360687853 function-entry: call_stack.py:function_4:13
156641360700719 function-return: call_stack.py:function_4:14
156641360719640 function-entry: call_stack.py:function_5:18
156641360732567 function-return: call_stack.py:function_5:21
156641360747370 function-return:call_stack.py:start:28


Static SystemTap markers
------------------------

The low-level way to use the SystemTap integration is to use the static
markers directly. This requires you to explicitly state the binary file
containing them.

For example, this SystemTap script can be used to show the call/return
hierarchy of a Python script:

.. code-block:: none

probe process("python").mark("function__entry") {
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;

printf("%s => %s in %s:%d\\n",
thread_indent(1), funcname, filename, lineno);
}

probe process("python").mark("function__return") {
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;

printf("%s <= %s in %s:%d\\n",
thread_indent(-1), funcname, filename, lineno);
}

It can be invoked like this::

$ stap \
show-call-hierarchy.stp \
-c "./python test.py"

The output looks like this:

.. code-block:: none

11408 python(8274): => __contains__ in Lib/_abcoll.py:362
11414 python(8274): => __getitem__ in Lib/os.py:425
11418 python(8274): => encode in Lib/os.py:490
11424 python(8274): <= encode in Lib/os.py:493
11428 python(8274): <= __getitem__ in Lib/os.py:426
11433 python(8274): <= __contains__ in Lib/_abcoll.py:366

where the columns are:

- time in microseconds since start of script
- name of executable
- PID of process

and the remainder indicates the call/return hierarchy as the script executes.

For a :option:`--enable-shared` build of CPython, the markers are contained within the
libpython shared library, and the probe's dotted path needs to reflect this. For
example, this line from the above example:

.. code-block:: none

probe process("python").mark("function__entry") {

should instead read:

.. code-block:: none

probe process("python").library("libpython3.6dm.so.1.0").mark("function__entry") {

(assuming a :ref:`debug build <debug-build>` of CPython 3.6)


Available static markers
------------------------

.. object:: function__entry(str filename, str funcname, int lineno)

This marker indicates that execution of a Python function has begun.
It is only triggered for pure-Python (bytecode) functions.

The filename, function name, and line number are provided back to the
tracing script as positional arguments, which must be accessed using
``$arg1``, ``$arg2``, ``$arg3``:

* ``$arg1`` : ``(const char *)`` filename, accessible using ``user_string($arg1)``

* ``$arg2`` : ``(const char *)`` function name, accessible using
``user_string($arg2)``

* ``$arg3`` : ``int`` line number

.. object:: function__return(str filename, str funcname, int lineno)

This marker is the converse of :c:func:`!function__entry`, and indicates that
execution of a Python function has ended (either via ``return``, or via an
exception). It is only triggered for pure-Python (bytecode) functions.

The arguments are the same as for :c:func:`!function__entry`

.. object:: line(str filename, str funcname, int lineno)

This marker indicates a Python line is about to be executed. It is
the equivalent of line-by-line tracing with a Python profiler. It is
not triggered within C functions.

The arguments are the same as for :c:func:`!function__entry`.

.. object:: gc__start(int generation)

Fires when the Python interpreter starts a garbage collection cycle.
Expand Down Expand Up @@ -337,98 +175,3 @@ Available static markers
pointer to a tuple object.

.. versionadded:: 3.8


SystemTap Tapsets
-----------------

The higher-level way to use the SystemTap integration is to use a "tapset":
SystemTap's equivalent of a library, which hides some of the lower-level
details of the static markers.

Here is a tapset file, based on a non-shared build of CPython:

.. code-block:: none

/*
Provide a higher-level wrapping around the function__entry and
function__return markers:
\*/
probe python.function.entry = process("python").mark("function__entry")
{
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
frameptr = $arg4
}
probe python.function.return = process("python").mark("function__return")
{
filename = user_string($arg1);
funcname = user_string($arg2);
lineno = $arg3;
frameptr = $arg4
}

If this file is installed in SystemTap's tapset directory (e.g.
``/usr/share/systemtap/tapset``), then these additional probepoints become
available:

.. object:: python.function.entry(str filename, str funcname, int lineno, frameptr)

This probe point indicates that execution of a Python function has begun.
It is only triggered for pure-Python (bytecode) functions.

.. object:: python.function.return(str filename, str funcname, int lineno, frameptr)

This probe point is the converse of ``python.function.return``, and
indicates that execution of a Python function has ended (either via
``return``, or via an exception). It is only triggered for pure-Python
(bytecode) functions.


Examples
--------
This SystemTap script uses the tapset above to more cleanly implement the
example given above of tracing the Python function-call hierarchy, without
needing to directly name the static markers:

.. code-block:: none

probe python.function.entry
{
printf("%s => %s in %s:%d\n",
thread_indent(1), funcname, filename, lineno);
}

probe python.function.return
{
printf("%s <= %s in %s:%d\n",
thread_indent(-1), funcname, filename, lineno);
}


The following script uses the tapset above to provide a top-like view of all
running CPython code, showing the top 20 most frequently entered bytecode
frames, each second, across the whole system:

.. code-block:: none

global fn_calls;

probe python.function.entry
{
fn_calls[pid(), filename, funcname, lineno] += 1;
}

probe timer.ms(1000) {
printf("\033[2J\033[1;1H") /* clear screen \*/
printf("%6s %80s %6s %30s %6s\n",
"PID", "FILENAME", "LINE", "FUNCTION", "CALLS")
foreach ([pid, filename, funcname, lineno] in fn_calls- limit 20) {
printf("%6d %80s %6d %30s %6d\n",
pid, filename, lineno, funcname,
fn_calls[pid, filename, funcname, lineno]);
}
delete fn_calls;
}

7 changes: 0 additions & 7 deletions Include/pydtrace.d
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,6 @@
/* Python DTrace provider */

provider python {
probe function__entry(const char *, const char *, int);
probe function__return(const char *, const char *, int);
probe instance__new__start(const char *, const char *);
probe instance__new__done(const char *, const char *);
probe instance__delete__start(const char *, const char *);
probe instance__delete__done(const char *, const char *);
probe line(const char *, const char *, int);
probe gc__start(int);
probe gc__done(long);
probe import__find__load__start(const char *);
Expand Down
Loading
Loading