Memory leak with psycopg-c (but not pure python)

[bufke]

I found a memory leak when using psycopg-c (not present in pure python version) when using with Django and Celery. I reported the issue to Django here but it is not clear exactly which project is at fault.

Here is a reproduction repo.

Django/Celery seems to call hstore.register_hstore(ti, connection.connection) on every celery task instead of just once. That might be a bug in one of those projects - but I would think psycopg should not memory leak when doing such. And the problem goes away with psycopg (sans c).

I will continue to try to create a more minimal reproduction set up. Please let me know if you have ideas to things to try out to help debug/fix the issue.

[dvarrazzo]

Hello,

this is surprising, because hstore has no C code.

Like in many register_*() functions, custom classes, customized according to the OID of the extension type, are created. The assumption is that such functions are not created often, however another softer assumption was that the classes are GC'd. Maybe this is not reliable enough.

Could you pleas try if f9c8878 solves the problem?

[bufke]

I applied the patch directly to a virtual environment's psycopg and confirmed the new code executes. However the leak still occurs. You may want to also look at this Sciagraph profile I took here.

[dvarrazzo]

Could you please try to count the object in memory and figure out which ones are growing unexpectedly? I use code like the following:

    gc.collect()
    objs = gc.get_objects()

    c = defaultdict(int)
    for o in objs:
        c[type(o)] += 1

or you could try c[type(o).__name__] += 1 to spot more easily leaks caused by classes with different identities but the same name, as I thought it was the case here.

Comparing the number of objects per class at different times often points clearly to the object or the code that is causing the leak.

Also, which version of Python are you using here?

[bufke]

I can reproduce the issue with Python 3.10 and 3.11 using both Ubuntu and the Python docker image. I tried both the original and name idea but not sure it produced anything helpful. I ran 2000 queries and then filtered the dict's value to only print those above 2000.

c[type(o)] += 1

2014: <class 'list'>
19718: <class 'dict'>
6629: <class 'getset_descriptor'>
12886: <class 'weakref.ReferenceType'>
6826: <class 'frozenset'>
28208: <class 'function'>
5169: <class 'type'>
35854: <class 'tuple'>
10778: <class 'cell'>
6715: <class '_abc._abc_data'>
6413: <class 'typing._ProtocolMeta'>

c[type(o).name] += 1

2014: list
19718: dict
6629: getset_descriptor
12886: ReferenceType
6826: frozenset
28208: function
5169: type
35854: tuple
10778: cell
6715: _abc_data
6421: _ProtocolMeta

Now again after 10k runs

c[type(o)] += 1

2382: <class 'list'>
51934: <class 'dict'>
22658: <class 'getset_descriptor'>
44936: <class 'weakref.ReferenceType'>
30826: <class 'frozenset'>
52464: <class 'function'>
13203: <class 'type'>
116549: <class 'tuple'>
34871: <class 'cell'>
30715: <class '_abc._abc_data'>
30413: <class 'typing._ProtocolMeta'>

c[type(o).name] += 1

2382: list
51934: dict
22658: getset_descriptor
44936: ReferenceType
30826: frozenset
52464: function
13203: type
116549: tuple
34871: cell
30715: _abc_data
30421: _ProtocolMeta

I also tried out memray

The lines under the User.objects.count from the reproduction repo grow as more celery tasks are run. Here's another after 20k tasks and we can see it exceeds the memory usage of everything else

Both memray and a celery task to print that gc info have been pushed to the reproduction repo.

[dvarrazzo]

I tried both the original and name idea but not sure it produced anything helpful.

Sorry, maybe I was unclear about how to use that snippet:

• run the loop n times, even for low n
• get a count of objects per type
• run the loop another single time
• get a count of the objects again

The difference between counts, if any, will tell which are the types of objects leaking.

From your examples I see that more or less one more type per iteration is created. So the next step to investigate could be to:

• run the loop n times
• get the list of types ([obj for obj in gc.get_objects() if type(obj) is type])
• run the loop another single time
• get the new list of types

there should be a single element more in the second list: which type it is?

[dvarrazzo]

(sorry, closed by mistake)

[bufke]

Indeed, there is exactly one additional type. <class 'psycopg.types.array.HstoreArrayLoader'>

[dvarrazzo]

Bingo! So we were on the right track, but it's the array loader to cause the problem. Which makes more sense, because the base of the array loaders is a Cython class, if the C extension is available. It looks like classes with a C base are not correctly GC'd.

I am working on a MR to cache every dynamic class generated by psycopg. Sending you something for testing soon.

[dvarrazzo]

@bufke could you please try if #649 fixes the issue?