gh-94673: Add Per-Interpreter tp_subclasses for Static Builtin Types

[ericsnowcurrently]

• Issue: [subinterpreters] Static types incorrectly share some objects between interpreters #94673

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit 2fa815e 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[kumaraditya303]

LGTM, I have left some questions.

FWIW, we could also get rid of the new type flag and use tp_subclasses as a tagged pointer and steal one bit to check if the tp_subclasses is per interpreter or local, I'll probably do that but for now this is fine.

[encukou]

I'm afraid this isn't backwards compatible. The tp_subclasses docs do say it's internal only, but do currently guarantee that it's a list, so IMO it's valid for API users to treat it as a PyObject*, and expect exceptions (rather than crashes).

Consider:

• Seeking a SC exception from Python's backcompat policy
• Renaming the field to _tp_subclasses, so anyone currently using it gets a compiler error
• Adding the change to What's New so people know what's up
• Removing the field from docs
• Getting a PEP accepted for changes like this, to find unintended consequences
• (edit) Changing the type to void* since it's not PyObject* any more

[ericsnowcurrently]

FWIW, tp_subclasses is already a dict and probably has been for quite a while.

[kumaraditya303]

but do currently guarantee that it's a list,

The current implementation uses a dictionary to store subclasses as a mapping of PyLongObject * to subclass. On the Python level it is a list but not at C level.

so IMO it's valid for API users to treat it as a PyObject*, and expect exceptions (rather than crashes).

Why do you expect it to raise an exception when you dereference a pointer which is clearly documented to be a internal implementation detail?

[encukou]

I'm not saying this is a bad change. I'm asking to be mindful of the possible effects on code other than CPython that might look at this field.

FWIW, tp_subclasses is already a dict and probably has been for quite a while.

Yes, the docs are wrong. I don't think that's a free pass make this an invalid PyObject* pointer, though.
Bad docs just mean that people will experiment, or read the source, to see what things do.

Why do you expect it to raise an exception when you dereference a pointer which is clearly documented to be a internal implementation detail?

Well, I do expect to be able to inspect internal details, e.g. for debugging.

[ericsnowcurrently]

Hey, the feedback is super helpful! Thanks!

I'll go ahead and add a whatsnew entry about the special-casing of tp_subclasses for static builtin types.

[ericsnowcurrently]

Making tp_sublcasses void * is a good idea.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @ericsnowcurrently for commit dace9c4 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[encukou]

Thanks!
I left some more suggestions on the docs in ericsnowcurrently#31