Enable cross-interpreter sharing of tuples

[tonybaloney]

Feature or enhancement

Proposal:

This issue is for adding support of sharing tuples (and tuples of tuples) through the crossinterp API and the interpreters module when that arrives.

The data structure will have to have a variable length.

struct _shared_tuple_data {
    Py_ssize_t len;
    _PyCrossInterpreterData **data;
};

It will use the xid registry to lookup the type values in the tuple, then encode them using the crossinterpdatafunc callbacks. This should work recursively if the tuple contains a tuple.

PR to follow.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs

• gh-111623 Serialize tuples with sub interpreters #111628

[a-reich]

Quick question @tonybaloney , is is_shareable intended to be reliable for containers? It now returns True for tuples even if they contain non-shareable types.

[ericsnowcurrently]

Excellent question! The underlying machinery for shareable objects isn't very sophisticated currently. is_shareable() basically just checks if the object's type supports sharing. To make it more accurate, as you've implied, we would need either to try to actually share the object (like how the builtin hasattr() works) or to add a hook that each type may implement to indicate if an object is shareable.

FWIW, the relevant code for the (currently internal-only) shareable-type protocol is in Include/internal/pycore_crossinterp.h and Python/crossinterp.c. For the is_shareable() implementation look in Modules/_xxsubinterpretersmodule.c. Any changes would happen in those files.

Keep in mind that, right now, the machinery for shareable objects is implemented with a global registry of types but I plan on writing a PEP in the near future targeting 3.13 to add a dedicated field to PyTypeObject to replace the registry. Also note that is_shareable() is not part of PEP 734, though we may add it to the module (not the PEP) later.

[a-reich]

Thank you for the explanation on the current design! Yeah, it makes sense that is_shareable is not very complex right now, and that it's not part of the official spec in PEP 734. (I just saw it in test.support.interpreters and was wondering how to use it when experimenting with the stuff there). The idea of replacing the registry with a type field/protocol sounds interesting for sure.