[subinterpreters] Explicitly track object ownership (and allocator).

[ericsnowcurrently]

BPO	33607
Nosy	@ncoghlan, @pitrou, @vstinner, @ericsnowcurrently, @emilyemorehouse

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-05-22.19:19:38.720>
labels = ['expert-subinterpreters', '3.8']
title = '[subinterpreters] Explicitly track object ownership (and allocator).'
updated_at = <Date 2020-05-15.01:51:49.021>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2020-05-15.01:51:49.021>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Subinterpreters']
creation = <Date 2018-05-22.19:19:38.720>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 33607
keywords = []
message_count = 6.0
messages = ['317330', '317339', '317403', '318037', '318091', '368913']
nosy_count = 5.0
nosy_names = ['ncoghlan', 'pitrou', 'vstinner', 'eric.snow', 'emilyemorehouse']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue33607'
versions = ['Python 3.8']

[ericsnowcurrently]

When an object is created it happens relative to the current
thread (ergo interpreter) and the current allocator (part of
global state). We do not track either of these details for
the object. It may make sense to start doing so (reasons next).

Regarding tracking the interpreter, that originating interpreter
can be thought of as the owner. Any lifecycle operations should
happen relative to that interpreter. Furthermore, the object
should be used in C-API calls only in that interpreter (i.e.
when the current thread's Py_ThreadState belongs to that
interpreter). This hasn't been an issue since currently all
interpreters in the process share the GIL, as well as the fact
that subinterpreters haven't been heavily used historically.
However, the possibility of no longer sharing the GIL suggests
that tracking the owning interpreter (and perhaps even other
"sharing" interpreters) would be important. Furthermore,
in the last few years subinterpreters have seen increasing usage
(see Openstack Ceph), and knowing the originating interpreter
for an object can be useful there. Regardless, even in the
single interpreter case knowing the owning interpreter is
important during runtime finalization (which is currently
slightly broken), which impacts CPython embedders.

Regarding the allocator, there used to be just a single global
one that the runtime used from start to finish. Now the C-API
offers a way to switch the allocator, so there's no guarantee
that the right allocator is used in PyMem_Free(). This has
already had a negative impact on efforts to clean up CPython's
runtime initialization. It also results in problems during
finalization. Additionally, we are looking into moving the
allocator from the global runtime state to the per-interpreter
(or even per-thread or per-context) state value. In that world
it would be essential to know which allocator was used when
creating the object. There are other possible applications
based on knowing an object's allocator, but I'll stop there.

To sort all this out we would need to track per-object:

• originating allocator (pointer or id)
• owning interpreter (pointer or id)
• (possibly) "sharing" interpreters (linked list?)

Either we'd add 2 pointer-size fields to PyObject or we would
keep a separate hash table (or two) pointing from each object
to the info (similar to how we've considered doing for
refcounts). To alleviate impact on the common case (not
embedded, single interpreter, same allocator), we could default
to not tracking interpreter/allocator and take a lookup failure
to mean "main interpreter, default allocator".

[vstinner]

"Either we'd add 2 pointer-size fields to PyObject or we would keep a separate hash table (or two) pointing from each object to the info (...)"

The expect a huge impact on the memory footprint. I dislike the idea.

Currently, the smallest Python object is:

>>> sys.getsizeof(object())
16

It's just two pointers. Adding two additional pointers would simply double the size of the object.

"Now the C-API offers a way to switch the allocator, so there's no guarantee
that the right allocator is used in PyMem_Free()."

I would expect that either all interpreters use the same memory allocator, or that each interpreter uses its own allocator. If you use one allocator per interpreter, calling PyMem_Free() from the wrong interpreter would just crash. As you get a crash when you call free() on an object allocated by PyMem_Free(). You can extend PYTHONMALLOC=debug to detect bugs. This builtin debugger is already able to catch misuses of allocators. Adding extra pointers to this debugger is acceptable since it doesn't modify the footprint of the default mode.

[ncoghlan]

Rather than tracking this per object, you could potentially track it per arena at the memory allocator level instead. Then if you really need the info (e.g. when running the debug allocator), you can check it in a reliable way, but in the normal case, you assume the associations are being managed correctly and avoid any significant bookkeeping overhead.

[ericsnowcurrently]

Note that I wouldn't call this issue absolutely specific to subinterpreters. The "ownership" part is, but tracking the allocator has practical application under a single interpreter. I suppose I could split this issue apart. I lumped the two together because I expected the solution would be the same for both. However, that's not necessarily the case. Would it help to open a separate issue for tracking the allocator?

[pitrou]

I agree with Victor, we shouldn't add PyObject fields that only have use in certain (minority) situations.

The idea of tracking per arena will be non-trivial to implement, as only small objects (smaller than 512 bytes) use our own allocator; larger objects go to the system allocator.

Can I ask why you're considering this? I thought you didn't want to transfer ownership between interpreters.

[vstinner]

I see two options:

• Add a field to PyObject, but only in a special debug mode. Maybe not even in Py_DEBUG (since I managed to make Py_DEBUG ABI-compatible with the release mode!)

• Add an hash table mapping an object to its interpreter. The hash table would only be used in debug mode. It may even be turned on at runtime depending on a command line option or something else.

See also bpo-40514: [subinterpreters] Add --experimental-isolated-subinterpreters build option.

Antoine: "Can I ask why you're considering this? I thought you didn't want to transfer ownership between interpreters."

I guess that the purpose is to ensure that: detect when an object is shared between two interpreters.

Currently, tons of objects are still shared between interpreters. Starting with static types: see bpo-40601 "[C API] Hide static types from the limited C API".