gh-106597: Add debugging struct with offsets for out-of-process tools

[pablogsal]

• Issue: Add a collection of offsets to facilitate the work of out-of-process debuggers #106597

[pablogsal]

@markshannon Can you take another look?

[markshannon]

I actually meant using pointers, so that the fields don't move. But this is fine.

I'm not the one using this. If it suits pystack, pyspy and austin, them I'm cool with whatever you want.

[pablogsal]

I actually meant using pointers, so that the fields don't move.

Ah, that would make retrieving these a bit more annoying because it would imply having to copy an extra pointer per structure group and also it makes initializing the runtime state more verbose so I would prefer to leave it as is.

[carljm]

What are the backward-compat guarantees for this struct? Should we document them? I assume for this to be useful we should only ever add anything to the end of this struct, so that profilers don't have to have per-Python-version maps of the struct in order to be able to use the values in it correctly? (Also should we commit explicitly that this struct will always occur first in PyRuntimeState?)

And does that mean we commit to the future existence of all the struct fields with offsets listed here? Or if the struct field would be removed in a future Python, we'd set the offset to some kind of sentinel value without removing the entry from the debug-offsets struct?

EDIT: ok, I see now in the issue that there is no backwards-compatibility guarantee at all between minor versions. That also seems worth documenting, at least in a comment? There is already a comment saying the debug struct should stay first.

[carljm]

FWIW, here's a list of offsets that we currently export from Cinder for use by our in-house out-of-process profiler (Strobelight), that are not currently exported in this PR:

offsetof(PyObject, ob_type)
offsetof(PyTypeObject, tp_name)
offsetof(PyCodeObject, co_flags)
offsetof(PyTupleObject, ob_item)
offsetof(_PyRuntimeState, gilstate.autoTSSkey._key)

Also, if we give an offset for PyCodeObject.co_localsplusnames, should we also give one for PyCodeObject.co_localspluskinds? Currently we use co_varnames which no longer exists as a field, to reconstruct it you'd need both names and kinds.

And if we expose an offset for PyThreadState.cframe, should we also explicitly give offsets for _PyCFrame.current_frame and _PyCFrame.previous? Otherwise you can't safely do much with the cframe value.

[pablogsal]

Also, if we give an offset for PyCodeObject.co_localsplusnames, should we also give one for PyCodeObject.co_localspluskinds? Currently we use co_varnames which no longer exists as a field, to reconstruct it you'd need both names and kinds.

And if we expose an offset for PyThreadState.cframe, should we also explicitly give offsets for _PyCFrame.current_frame and _PyCFrame.previous? Otherwise you can't safely do much with the cframe value.

I will add these to this PR 👍

[pablogsal]

That also seems worth documenting, at least in a comment? There is already a comment saying the debug struct should stay first.

Good point! Will update the PR tomorrow to include that comment.

[godlygeek]

PyStack needs a few more fields than we have here. We also need:

• native_thread_id from the thread state, for matching a C backtrace up to the Python thread it's for
• locked and holder from the GIL runtime state, for checking whether a thread holds the GIL

[pablogsal]

FWIW, here's a list of offsets that we currently export from Cinder for use by our in-house out-of-process profiler (Strobelight), that are not currently exported in this PR:

I added all the offsets you mentioned for Strobelight except offsetof(_PyRuntimeState, gilstate.autoTSSkey._key) which doesn't exist anymore

[pablogsal]

@carljm I added the fields you mentioned and modified the comment. Could you please take a look?

[carljm]

Looks great! Thank you @pablogsal