How can we make Python's C-API more friendly for Java bindings?

[encukou]

Hello,
I'm figuring out a strategy for the future of Python's C-API, and as one of the goals I'd like to make it friendly for non-C languages. But since I mostly deal with C & Python, I'd like to ask you for any suggestions, pain points or comments on how do better to that.
Would you like to help?

I started a Google doc to collect notes, please feel free to comment there, or here, or suggest a better place.
Thank you for your time!

[Thrameos]

@encukou I have a number of issues with the Python C API and would be happy to add any of them to the document if you feel that it is appropriate.

Here is a brief run down of the main issues that I have encountered. If you see an issue that make sense to push to the google doc, just follow up on this discussion and I will try to elaborate. (I have posted these in the past to the Python dev list, but unfortunately my employer refuses to sign the Python contribution list and thus I simply can't fit them myself.)

1. Concrete strings - In theory Java and Python have compatible definitions of strings. Both are immutable and thus one should just be able to wrap a Java string as a Python string and be done. However, Strings are not a protocol but rather concrete objects, thus I can't just implement an interface for the C API of Python string and make them compatible. This failing forces either immediate conversion (which itself is problematic if a string is very large and the user does not intend to work with in Python extensively), or use a conversion. This problem does not just affect Java wrappers, but Qt wrappers and many other language bindings where immutable strings are available.

2. StackFrames has a very similar situation. Java and Python both have stack frames and when calling an exception I aim to return both the Java, C and Python stack information. However, just like strings Python stack frames are all concrete, and there is no API for me to add a marker into the Python stack frame system so that I know where I left from Python to Java and when I returned so that I can integrate my stack frames into existing Python ones. As a result I have to write some extraordinarily dangerous code which manually creates stack frame object for exception handling. Because there is no protocol this must be done aggressively even if no stack frame is every used. The code for this monstrosity is located in...
   

   jpype/native/common/jp_exception.cpp

   Line 492 in f38a066

   PyTracebackObject *tb_create(

3. Dedicated language slot for wrappers - The next largest hack was that I needed to use true multiple inheritance. I have a huge number of Java types (JObject, JString, List, JInt, JChar, JDouble, JException) which needed to derive from different Python base classes but all needed to inherit a common element which was a language slot. The solution that I was forced to go to as Python does not support multiple true multiple inheritance was to borrow the same system as used by weak_ref system. I created a custom allocator which added an extra set of bytes to the end of the class and then inserted my slot space after the regular space. Obviously this only supports one extra slot but as objects from a language binding only need one language (you won't have a type which is both Java and C# wrapper at the same time), this isn't a big restriction. Having such a mechanism being formalized would help a great deal. The code for this abomination is located in...
   https://github.com/jpype-project/jpype/blob/master/native/python/pyjp_value.cpp

As you can see this makes an O(1) lookup for language binding slots. If there were a slot in the class with an offset (usually negative to reference from them end I would not need such code.

4. Next is is the poor naming conventions for C interfaces. The Python code has a mix of borrow, stolen, and new references, but nothing in the Python naming convention tells you which it which. Most of the time I can locate it in the documentation, but if I click on the header there is simply no documentation. Some people propose always giving a hard reference which would simplify the system but is pretty prohibitable. I designed a system using macros and anonymous unions to emulate the same system that Java used, but it has the downside that it is macro based (so not very good for other languages) and forces the base types for the interface to be PyObject** as you need to have references to the local frame table. This is fine for the language I have been developing, but it has overhead and thus would meet a lot of resistance (even if it makes it so that Python C code looks much cleaner as the module code would never need to deal with references directly). Since that approach is no good for the C Python API, the next best thing is (1) Add the requirements for a call to the headers so that when you jump to the header the calling requirements (new vs borrow returns) and arguments (steal or borrow) are clear without having to search the documentation; (2) If possible use a notation like Pyb_ or ``Pyn_` on calls to the API so that the code will be clear without having to navigate to documentation. For reference on what we have done while working with the C API please refer to
   https://github.com/jpype-project/jpype/blob/master/native/python/jp_pythontypes.cpp

Also please see it in action...

jpype/native/common/jp_classhints.cpp

Line 155 in f38a066

JPPyObject args = JPPyObject::call(PyTuple_Pack(2,

5. Next up is exposing useful methods for subclassing of types. In order to introduce new types as a meta class I need to be able to replicate the process. As none of the methods that are used in Python I was forced to replicate the whole process myself which is horrible to maintain. It is much to much to explain here, but the nightmare can be found at

jpype/native/python/pyjp_class.cpp

Line 134 in f38a066

PyObject* PyJPClass_FromSpecWithBases(PyType_Spec *spec, PyObject *bases)

I likely have another 3 more issues after this, but I have to head to my work meeting for that is all for now.

[encukou]

Thank you for sharing these!
In this timezone the weekend is starting, so I'll be brief for now:

1. This is more of an interpreter design issue than API design I'm focusing on. That said, @methane is working to make PyUnicode a bit more generic, but I don't think allowing custom string representations is planned.
2. Cython's struggling with this as well. I'll add it to the doc (along with the other issues).
3. Would this proposal help? https://mail.python.org/archives/list/capi-sig@python.org/ (combined with item 5, for using custom metaclasses)
4. The plan here is to tighten up the limited API to be consistent (always return strong refs, etc.), and then, if exceptions are needed, expose obviously-named functions. Cleaning up the existing API would be too much of a compatibility break.
5. This is getting implemented, see gh-60074: add new stable API function PyType_FromMetaclass python/cpython#93012 (comment)

[Thrameos]

I am not sure how the proposal for number 3 would work (also not sure which thread I should be looking at). Lets consider the problem and two possible solutions.

1. A language binding needs to add memory to hold language dependent data to multiple concrete base classes. (The same a heap type added dictionary or weakref list).
2. There needs to be O(1) operation to check it if belongs to the that particular language binding class because we can't integrate the object to figure out which memory layout it is working with.
3. There needs to be O(1) operation to retrieve the binding data.

JPype Solution

• Place a new memory allocator on all of the types that are for the language binding using the meta class.
• The allocator adds the required space to the end of the requested memory structure for every language wrapper. (with padding so the allocating address is aligned)
• To check to see if the slot belongs to the language wrapper it checks to see if the allocator slot == the language binding allocator. O(1) (after all if you have a different allocator it doesn't have the memory attached)
• To access the slot, it checks to see if the slot exists and if it does compute the end location for a standard Python structure computes if there is padding, then returns the end address as the language binding slot.

Standard solution

(this is modeled off of heap added dict used in heap types)

• When declaring the meta class , it registers this as a language extension with the desired space for the language extension.
• When the heap type using the meta class is declared, it adds the language extension space to the desired memory size and fills out a "extension type slot" pointing to the language descriptor and the extension offset to where this extra data will be. (it may be before or after the end referenced data like heap dict and weakref).
• When the language binding is looking to see if object belongs to language, it checks to see if the language descriptor points to one for the language (C#, Rust, Java, etc). If yes then the language slot will belong to the this language.
• To retrieve the slot instance, it checks the class description to get the language slot offset and adds that to the PyObject address and casts it to the language struct.
• When a python class is constructed with multiple base classes, if there are two classes which differing language slots, then give a TypeError for incompatible meta classes. (no C# + Java objects)

The JPype method is a kludge as it is abusing slots and hiding memory. The good news it didn't steal a bitfield, nor have to troll through the meta tuple to get the language info. Having a defined slot mechanism is just much better.

[encukou]

The proposed solution is to add a metaclass which reserves the extra memory, and an O(1) function to get that memory for a given class. That should cover 1 and 3.
This leaves the O(1) check. I'll watch for a similar request from the other language/framework bindings. If it'd be just for JPype -- I don't think the solution with a custom allocator is that bad. Even if the custom allocator would just call the standard one.

[Thrameos]

The severe downside of the full JPype solution (using the allocator to add extra memory) is that it appears to break PyPy. Having an actual method for the memory as part of the Python API will resolve that portion of my issue.

Using a slot like allocator to decide the meta class with O(1) should be safe in most cases, though I have occasionally had silly conflicts when something that looked like an allocator was accidentally declared. Of course, my meta class guards against the possibility so not a big issue. I suppose that if there was an O(1) method to just check to see if it is the correct meta class I could drop the kluge.

[encukou]

That would be Py_TYPE(PyTYPE(obj)), assuming you don't allow subclassing that metaclass. (If you do, maybe you can relax the requirement to O(N) in the length of the metaclass inheritance chain, and use isinstance.)

[Thrameos]

Yeah, I only have one meta class so Py_TYPE(PyTYPE(obj)) would be a good alternative once I don't need to do the allocator hack.

[Thrameos]

@encukou In addition to these issues I have had at times had to "augment" the C API to perform tasks for which there were not relevant calls. You may want to review them to see which ones are relevant to the API.

jpype/native/python/pyjp_module.cpp

Line 144 in f38a066

void PyJP_SetStringWithCause(PyObject *exception,

jpype/native/python/pyjp_module.cpp

Line 166 in f38a066

PyObject* PyJP_GetAttrDescriptor(PyTypeObject *type, PyObject *attr_name)

jpype/native/python/pyjp_module.cpp

Line 199 in f38a066

int PyJP_IsSubClassSingle(PyTypeObject* type, PyTypeObject* obj)

jpype/native/python/pyjp_module.cpp

Line 211 in f38a066

int PyJP_IsInstanceSingle(PyObject* obj, PyTypeObject* type)

[Thrameos]

Regarding 3 & 4: Although my implementations are very specific, there is some value that maybe worth considering. If the Python type system can track which classes are a child class has an MRO order such that the parent position is know to be at a fixed position for each of its parents, then it allows O(1) IsInstance checks. Simply verifying the order during class creation and tagging it with "ProperSingleInheritance" would then allow a bypass like these to be added to PyType_IsSubtype which would make some inheritance checks much faster (and eliminate my horrible hacks).

(No need for my methods to be on the public API, but the concept of using O(1) checks to speed up issubtype is something not currently in the Python core and as it presented a speed bottleneck I just worked around it.)

int
PyType_IsSubtype(PyTypeObject *a, PyTypeObject *b)
{
    PyObject *mro;
    mro = a->tp_mro;

    /** New bypass psuedo code**/
    int a_proper_order = a->tp_properorder;  // Get the number of types in the mro before we hit multiple inheritance.
    int b_proper_order = b->tp_properorder; // Get the number of types in the mro where b is single inherited.
    if (b_proper_order == n)
    {
         // No need to search the MRO as we know the position in the mro where the type must be
         if (a_proper_order >=n && PyTuple_GET_ITEM(mro, n-1) == (PyObject *)b)
                return 1;
    }
    /** End bypass **/
   

    if (mro != NULL) {
        /* Deal with multiple inheritance without recursion
           by walking the MRO tuple */
        Py_ssize_t i, n;
        assert(PyTuple_Check(mro));
        n = PyTuple_GET_SIZE(mro);
        for (i = 0; i < n; i++) {
            if (PyTuple_GET_ITEM(mro, i) == (PyObject *)b)
                return 1;
        }
        return 0;
    }
    else
        /* a is not completely initialized yet; follow tp_base */
        return type_is_subtype_base_chain(a, b);
}

[encukou]

I am somewhat surprised this was a performance bottleneck. How deep do your inheritance chains tend to get?

[Thrameos]

Because they shadow the entire Java type hierarchy and I need typically 4 python classes before I hit the Java base classes I typically am in the 10s or more.

The reason that I hit a performance bottleneck is method resolution. When you need to decide which overload will be called prior to converting every one of the type elements, you must match each argument in the incoming Python tuple to each potential argument list. As Java sometimes has overloads for its base classes with 10 or more overloads on methods with sometimes multiple arguments, and I need to check every possible conversion (meaning each argument in the match list may check up to 10 different isinstance), that meant that each call to a Java call for something silly like using a put method on IOStream was pegging hundreds isinstance checks. If someone was then running in a tight loop calling the method a thousand times it was just painful. So just like Python is cheating by assign a bunch of bit fields to speed up checking for basic types like int, I needed to make O(1) isinstance and issubtype checks.

I implemented a huge piles of caching so that if you hit the same overload many times it would bypass the overload resolution, but you could still hit the bottleneck if you happened to write cursed code that bounced between two overloads.

I am a bit surprised that some shortcut is not there in Python already. Verifying the MRO is a proper base class (all items between its bases match order) is trivial and if you have something 8 levels deep like I typically do searching the mro to find it at the 8th spot would seem like a big slowdown.

[encukou]

I see. Such long inheritance chains are pretty rare in Python, so I guess no one saw the need to optimize this so far.
Anyway, this veered away from API to optimization, and I'll need to keep focused to get anything done, so I'll need to leave the thread with another “PRs welcome” :(

[Thrameos]

😢 Another one where I would love to make a PR but can't (unless someone can wave the signed requirement the contributor agreement).

I should note, that you can get some of the benefits without having the proper counter.

1. If the type you are checking has more base classes than the one that is being checked, then you can bypass the whole thing because unless the mro is entirely screwed up and it not a proper base class it is if the parent has 8 and the child has only 3 so no need to search the 3.
2. If you just arbitrarily assume that the bases is proper and the child has 5 and the parent has 3, if you check the 3rd entry in the child and it matches you are done. This however is only half way, as if you know it is a proper base class you can skip the search entirely and return false.

Unfortunately, for me I could not stop at the half way point because in my case I am making a huge number of checks that are expected to fail.

Sorry for my rambling as I know that you are focused on the exposed API.

[Thrameos]

Next up.... The lack of a factory from buffer type to buffer type in the C API was an rough spot. Likely out of scope for your work, but it was an area that created a lot of issue. My version is specialize in that it takes a Python buffer type string and a Java type code and gives back a function, but it could easily be reformulated to go from Python buffer type to Python buffer type strings.

jpype/native/common/jp_convert.cpp

Line 128 in f38a066

jconverter getConverter(const char* from, int itemsize, const char* to)

The code is slightly less cursed that the corresponding one in C Python which I could not access through the API at all. So I had to replicate the whole darn thing in my version.

(Again sorry for the wish list, but there are a lot of items that have plagued me over the years and I have a fair number of dead bodies that I have to recall where I buried.)

[encukou]

I'm not entirely sure what this does, or how to turn it into a generally useful API. Maybe it's something better left to libraries like NumPy?

[Thrameos]

The API is just returns a function for converting from one memory layout to another when doing a buffer copy. I implemented it because the corresponding code in C Python was private. Of course, it unfortunately drifted a lot from the code I was modeling as it became specialized to Java.

The code in CPython which was performing the same task was hitting the switch statement each time it was going through item by item in the conversion process. So I decided for performance reasons to have it go through the switch statement once and give back a static function which I can then use in a tight loop.

I will try to locate the original code where I started (though it has been years and I did the work while in noisy gym watching my sons wrestling tournament so I only have value recollections of where in CPython I was reading.)

[Thrameos]

Okay I found the code that I was trying to access when I wrote the converter function. There is a big blob of code in memory view called unpack_single and pack_single. When I am pushing a huge list python objects into Java, taking a huge list out of Java back into Python, or if I am moving lots of memory from one buffer to another, all of those operations go through the converter function here.

So for example in the method to_listbase it is taking a set of memory with identical items and pushing them one a time through the switch statement which is just hitting the same case over and over.

I was hoping to get an API which will have the following functions.

typedef void (*pymem_transfer)(void* target, void* src);  // A function that takes memory from source and pushes into target
typedef PyObject* (*pymem_unpacker)(void* target);      // A function that unpacks memory and produces a new reference
typedef void (*pymem_packer(void* target,  PyObject* src);  // A function that packs an object from source into memory.

pymem_transfer PyMem_GetTransfer(char* target_fmt, char* source_fmt);
pymem_unpacker PyMem_GetUnpacker(char* target_fmt, char* source_fmt);
pymem_packer PyMem_GetPacker(char* target_fmt, char* source_fmt);

This would make it very easy to perform buffer transfers from any memory layout that exists in another language in to Python, out of Python, and from the languages native to a Python memory buffer.

Yes, there it may be possible to do this with numpy, but I can't require numpy (there are lots of uses of Java that don't involve numpy) and this is really targeting making memoryview more useful.

[Thrameos]

@encukou So on to evil hacks.

jpype/native/python/pyjp_method.cpp

Line 401 in f38a066

PyFunction_Type.tp_flags |= Py_TPFLAGS_BASETYPE;

In this code I was trying to make Java methods compatible with Python ones such that they shared the API enough that they were always guaranteed to be recognized by tab completion engines. At the time, I extended the Python method function class from my own, but that class was not marked as a base class so I had to flip the flags in the function class, extend it, then flip it back. It is still unclear if extending the type as the user who was directing the work was pointing me in the wrong direction (since he was trying to get jedi to work).

But it points to usual problem that if something is assuming concrete types rather than protocols the evil hacks end up getting used.

[encukou]

Well, that'll definitely void your warranty :)
It looks like a workaround for jedi, though. Is there some more discussion about needing to do this on the jedi side?

[Thrameos]

I am not entirely convinced this evil hack was necessary. Jedi takes a horribly round about way to look at objects as it wants to probe them without triggering any actions. I could try to remove it though it would technically break compatibility is Java methods current pass certain checks for isinstance(function) when used with inspect.

My only question on this one was why are methods implementations final from the C API side. I can see why you don't want to extend them in Python, but in the C area if you pass the memory contract I am not sure why the restriction. The reason that the evil hack came about is that I derived it from function using the static API a few years back, then when I switched to heap types it suddenly failed. Static types don't check the BASETYPE flag, so it wasn't of the issue until I had a release out for a long while (and now I hesitate to remove it.)

[encukou]

Not sure about this case, but a common reason for having internal classes be final is that the type checking is faster.
But it could also be backwards compatibility: for things that predate the ability to subclass builtin classes, before setting the flag someone needs to audit existing code for assumptions that'd now be broken.

[Thrameos]

Err... Isn't that the whole reason that I did the fast subtype checker? If you have a bypass which checks the order once when the class mro is created, you can immediately check types with O(1) which a speed improvement across the board.

[Thrameos]

@encukou Next up nearly completely intractable problems....

My single worst issue is how to deal with multiple garbage collection systems. Java is a garbage collected language, and Python is garbage collected, but neither of them have a plug in to support foreign managed objects. This causes several sets of very complex issues, (some of them I solved and some I have no clue how to deal with).

1. Running out of memory due to resources held in the "other" machine. If Python creates a bunch of Java objects which are large, but they are minimum objects than the process will run out memory because Python doesn't know know to trigger its garbage collection, as it never reached the size necessary to trigger it. I have dealt with this by doing the "inform the other". Thus every time Java triggers a GC it is reflected back to Python so that it knows that it should look for garbage items (which allows Java to then collect those dead resources). This is a mostly solved problem, but it required a lot of engineering to determine when to trigger properly.

jpype/native/common/jp_gc.cpp

Line 182 in f38a066

void JPGarbageCollection::onEnd()

2. There is a similar problem when Java is holding Python resources hostage. Unfortunately, as Java and Python GC triggers under very different situations I can't just reflect the call to Java. Instead as Java is "Stop-the-world" which is very expensive I have to examine the memory size of each to determine how many resources we may be able to free. This is where I ran into an issue. Although the Python allocator has the sizes of all objects currently active in the pool, and method use the trace command can access objects as they come and go, there is no counter in Python to simply get the total memory. The allocate command knows how much was requested, the free command knows how much is being returned, but the trace command only gets the notification after then object is deallocated so there is no way I can compute a delta to maintain the count of active memory. Thus I have to monitor the total page sizes which is very indirect and very hard to interpret as it is the combination of both Java and Python pages.

3. Last is of course the absolutely impossible problem of "cross language reference loops". A Python object points to a Java Object which points to a Python object. Java needs a hard global reference (as a weak reference wont do) and the Java object has to hold a hard reference to Python. This can only be dealt with if the weak reference has a method which could check the other language to see if is a container which would then allow it to proxy the traversal call back to Python. I can try to do the same thing from the Java side by cross wiring the RMI which is meant for Java remote processes. But given that Python lacks any facilities to deal with this nasty corner case, I have simply thrown up my hands up and decide that without aid from Python developers it would just make my brain hurt.

[encukou]

Sadly, that's out of my depth, and also most likely specific to the JVM :(

[Thrameos]

It isn't really a JVM thing, but rather a Python and JVM thing. Java has mechanisms to allow for a "group" garbage collect where something goes over the boundary from one virtual machine to another (basically sends a query to the other VM to probe what objects are reachable with traverse), but locally it used stop the world block sweep. I can repurpose the Java one to try to use it to probe into Python which I may be able to use traverse to get back to find out who is at that edge.

But yes this is also way outside of my depth, and I can't find any papers or topics on how to deal with multiple garbage collectors properly. If I could point you to a protocol which would make them play nice together that could be implemented, it would make for a world better integration as currently I can't release a lot of integrations that cause memory leaks it not used carefully.

[Thrameos]

And then there is the slice bug.

static bool PySlice_CheckFull(PyObject *item)
{
        if (!PySlice_Check(item))
                return false;
        Py_ssize_t start, stop, step;
        int rc = PySlice_Unpack(item, &start, &stop, &step);
#if defined(ANDROID)
        return (rc == 0)&&(start == 0)&&(step == 1)&&((int) stop >= 0x7fffffff);
#else
        return (rc == 0)&&(start == 0)&&(step == 1)&&((int) stop == -1);
#endif
}

One idiom that we used in JPype is the simple check of did the user type JInt[1:5] or JInt[:]. Unfortunately, the pattern of end points for slice changes from machine to machine, so trying to decide if the range was unspecified becomes a nightmare of trying to determine what value will this machine use. I could write code that creates an empty slice in Python and pass it into my module so that I can determine what the pattern for an empty range will be, but that is a bunch of work fighting something that is really a C Python design item.

[encukou]

I guess we need a PySlice_GetStart PySlice_GetStep PySlice_GetStop that would give you None or something else? Or even one function for all three?
Is PyObject_GetAttrString(item, "stop") too expensive here?
Do you need to distinguish between JInt[:] and JInt[0::1]?

[Thrameos]

Yes, though if it did the some undefined action on the second case I could live with it.

As a point of understanding, I need to be able to define these in Java (which correspond to the following Python code). Assuming jtype is some Java type object.

• Case 1 - create a new array with a specified length
  Java: jtype[] a = new jtype[25]; => Python: a = jtype[25]
• Case 2 - declare a new array type with unspecified dimensions
  A = jtype[:]
  a = A(25)
• Case 3 - declare a new multiple dimensional array with specified dimensions
  a = jtype[10,2]
• Case 4 declare multiple dimensional array with some unspecified dimensions
  Java: jpype[][] a = new jtype[10][]; => Python: a = jpype[10,:]

[Thrameos]

So thus far I have only been discussing "negatives" while there are a few things that I see that are very positive in the C Python API that I would like to see more of.

First up, I made the effort to convert all my C implementations to heap types use specifications, rather than the older fixed types were some static memory is being used. I really liked the move from static structure to heaps and it when fairly smoothly except for a few corner cases were static structures would allowed things that were prohibited by heap types. Those limitations required me to create the heap type then alter the slot afterward. That said I found the heap type interface to be much better and safe now the memory was controlled by Python and thus allows easier changes behind the scenes. I would like this to expand throughout the API such that rather than having things that may be a pointer or a direct reference to a static structure, everything should be pointer types only so that they are consistent and it is irrelevant if they are heap or statically typed.

[Thrameos]

Next request.... This is not really a C API, but certainly a request toward standardization. Please standardize the Python byte codes and compiled class formats like Java has done. One of the key things that I have been trying to achieve is to make it so a compiled Python file with customizers for Java classes to be stored in the Jar files directly. If I have to place a different jar file for each Python version then it becomes a huge headache. My understanding is the Python byte codes are version specific by architecture portable (I could be wrong). Having a byte code system which is at least backward compatible (with a version number so that it is always clear which interpretation) would help towards being able to ship embedded Python which would be helpful for Android.

[encukou]

Oof... I don't think we can do that any time soon. Bytecodes are an implementation detail that would need to be carefully documented specified, before we could take on another big task of cross-version testing. There are some efforts in this direction, but nothing near the amount of work that would be needed.

[Thrameos]

If you can't store byte codes directly, I guess the best thing would be to just have a translator from some common byte code format to the current internal version used by a particular Python. I do a lot of work with Java byte codes (literally rewriting Java code on the fly as it is loaded). We also have people that are pressing to have a unit that takes python byte codes and converts them to Java so that we can speed up execution. But this work is all predicated on the concept that Python will have a supported byte code format at some point. Else we would have to supply a different translator for each Python version.

See:
#1059

[encukou]

Well, the common format is Python source. You might get somewhere by starting from that (or the AST), similarly to what CPython does?
Otherwise, there is just no equivalent of a separate JVM layer in Python, and I don't know of anyone who might want to put in the effort to add one :(

[Thrameos]

Next up, missing destructors on PyThread_create_key is missing a destructor. The reason this is an issue is that Java requires that every thread that access a Java resource must bind to a Java thread so that it can allocate the resources. Unfortunately, Java has no way to know when a Python thread which has been bound has terminated so one we bind resources to Java it will be leaked unless the thread manually calls a detach.

POSIX thread of course allow a dtor to be attached to a key using pthread_key_create. Similarly C11 allows this with tss_create. NT is the trouble spot as their thread specific storage (TlsAlloc) does not allow a destructor. There are two solutions for this

1. Spawning a second thread that simply waits for the first thread to complete and then calls all destructors. This solution is a pain because it is hard to coordinate to get the address of the resource being held by the other thread. And as it doubles the resources, it is impractical to use.
2. Use fiber API to create a fiber specific resource when will call a routine whenever a thread exits from which we can delegate to the dtors.

The code for this is not very complex. If the system doesn't support fibers, then it is fine because it will leak which is what we are already doing.

#include <iostream>
#include <Windows.h>


struct TssDtor
{
    struct TssDtor* td_next;
    DWORD            td_handle;
    void             (*td_dtor)(void*);
};

struct TssManager
{
    INIT_ONCE       tm_once;
    DWORD           tm_key = 0;
    struct TssDtor *tm_dtors;
};
static struct TssManager manager;

static void WINAPI tss_manager_dtor(void*)
{
    struct TssDtor* dtor = manager.tm_dtors;
    while (dtor != NULL)
    {
        void* contents = TlsGetValue(dtor->td_handle);
        if (contents != NULL)
            dtor->td_dtor(contents);
        dtor = dtor->td_next;
    }
}

static BOOL WINAPI tss_manager_init(PINIT_ONCE once, PVOID parameter, PVOID* context)
{
    manager.tm_key = FlsAlloc(&tss_manager_dtor);
    manager.tm_dtors = NULL;
    return TRUE;
}

void tss_init()
{
    InitOnceExecuteOnce(&manager.tm_once, &tss_manager_init, NULL, NULL);
}

typedef void (*tss_dtor_t)(void*);
typedef void* tss_t;

void tss_create(tss_t* key, tss_dtor_t dtor)
{
    tss_init();
    DWORD handle = TlsAlloc();
    *key = (tss_t) handle;
    if (dtor != NULL)
    {
        struct TssDtor* next = (struct TssDtor*)malloc(sizeof(struct TssDtor));
        if (next == NULL)
            return;

        next->td_next = manager.tm_dtors;
        next->td_dtor = dtor;
        next->td_handle = handle;
        manager.tm_dtors = next;
    }
}

void tss_set(tss_t key, void* value)
{
    // Connect to the fiber so that we will be notified of thread termination
    if (value !=NULL && manager.tm_dtors != NULL)
        FlsSetValue(manager.tm_key, &manager);
    TlsSetValue((DWORD)key, value);
}

void* tss_get(tss_t key)
{
    return TlsGetValue((DWORD) key);
}

[encukou]

I'll keep this in mind, but like with some other ideas, won't push it forward myself if it doesn't become a common request.
PRs are always welcome, of course -- and note that CPython now needs C11, so it might be possible to switch to a single implementation.

[Thrameos]

😢 I really wish my employer would get a clue.... I would have contributed all of these things two years ago, but have been stuck dejectedly waiting for others because of bureaucracy.

I honestly don't know how the Python thread specific storage is supposed to work. Thread storage is supposed to be something very small (like a pointer), so I assume that people must be putting allocated memory on it. But if you can't place a destructor to deallocate it or dereference it when the thread goes away, how would that memory every be released? I can see how Python may have ended up removing that argument from pthreads given it is not possible to do it with windows without creating a manager like I did here, but what is the use case that Python was going for if you can't store an allocated structure?

[Thrameos]

One entirely off the wall API discussion: I have been trying for a while to port the Java method of dealing with native referencing into Python (done about 3 different attempts). I am not sure this would help others as they generally end up using heap objects to hold the local reference frames or anonymous union hacks in C11.

The concept is simple, at the start of a function you declare a frame which will hold all references (and declares a goto statement where all the exceptions will be handled if I want to merge exception handling in), which is passed to the corresponding methods. (though it can be hidden if you use a tss). Instead of passing back a borrowed or new reference, the reference is held in frame for the user. The user can add their own hard reference to it to hold it, call "keep" in a return statement to push the object to the enclosed outer local reference frame, or do nothing in which case it will be destroyed with all the local references.

It works well with gcc in which you have the equivalent of a finally statement which removes logic from the function "exit" and "keep" routines. It can run without the finally, but then you have to be careful no never return without calling the keep or exit routine. It can be made to play nice with current system (if you are using the tss so you don't have an extra argument) though I am sure there are edge cases. If is unfortunately slightly slower because it needs to clean up the references with a for loop rather than having the user directly kill the references. (the extra instructions to check the remaining items costs a bit over hard coding). The anonymous union version will of cross crash if you do a for loop and fail to set up an inner frame as it will exhaust the frame slots. The heap version are slower as it has to pull from the local frame pool each call.

The macro magic is just written out copies of regular functions in most cases so it is possible create heap frames and use it without the macros. The result is Python C code that reads a bit more like Java/C++ in which there are not endless levels of if statements, and no need for the user to see the referencing. However, I fear that such a system is just too far from the current C API to ever be considered.

I have been writing it for my own vanity project (a compiled language like Java which supports Python like protocols and views), but if it were something that Python could consider I could try to put in some effort.

[Thrameos]

@encukou Searching through the code I found another set of rough spots. That is in general there are no methods to create new objects for certain primitives like longs with an extension class.

So for example when I need to create a JLong which is derived from long (which just the added memory for a slot and the new meta class). I end up having to replicate portions of the code.

// equivalent of long_subtype_new as it isn't exposed

PyObject *JPPrimitiveType::convertLong(PyTypeObject* wrapper, PyLongObject* tmp)
{
        if (wrapper == NULL)
                JP_RAISE(PyExc_SystemError, "bad wrapper");
        Py_ssize_t n = Py_SIZE(tmp);
        if (n < 0)
                n = -n;

        PyLongObject *newobj = (PyLongObject *) wrapper->tp_alloc(wrapper, n);
        if (newobj == NULL)
                return NULL;

        ((PyVarObject*) newobj)->ob_size = Py_SIZE(tmp);
        for (Py_ssize_t i = 0; i < n; i++)
        {
                newobj->ob_digit[i] = tmp->ob_digit[i];
        }
        return (PyObject*) newobj;
}

JPPyObject JPIntType::convertToPythonObject(JPJavaFrame& frame, jvalue val, bool cast)
{
        JPPyObject tmp = JPPyObject::call(PyLong_FromLong(field(val)));
        if (getHost() == NULL)
                return tmp;
        JPPyObject out = JPPyObject::call(convertLong(getHost(), (PyLongObject*) tmp.get()));
        PyJPValue_assignJavaSlot(frame, out.get(), JPValue(this, val));
        return out;
}

First I have to call PyLong_FromLong but as that doesn't take a type argument nor anything which knows about the meta class, I then have to call a second one with my allocator and copy the memory of the first onto the second. This is very round about, inefficient, and of course doesn't work with PyPy at all. It would be very nice if there were a way to specify a derived type in the C API so that such hacks are not required.

[encukou]

The idea for a solution is here: faster-cpython/ideas#553

[encukou]

There is now a wider effort for mapping out C API issues at: https://github.com/capi-workgroup/problems/issues

[Thrameos]

Thanks for keeping me posted.