Error deleting closed connection on exit

[cvickery]

psycopg 3.0.7
Python 3.10.1
Postgres 14.1
MacOS 12.1 (Monterey)
Mac mini (M1 2020)

Applications run to completion, then when exiting, throw the following exception:

Exception ignored in: <function BaseConnection.__del__ at 0x102356dd0>
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/connection.py", line 140, in __del__
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/connection.py", line 161, in closed
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/pq/pq_ctypes.py", line 205, in status
TypeError: 'NoneType' object is not callable
Exception ignored in: <function PGconn.__del__ at 0x1021bbb50>
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/pq/pq_ctypes.py", line 91, in __del__
TypeError: 'NoneType' object is not callable
Exception ignored in: <function PGresult.__del__ at 0x102232320>
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/pq/pq_ctypes.py", line 723, in __del__
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/pq/pq_ctypes.py", line 733, in clear
TypeError: 'NoneType' object is not callable

[dvarrazzo]

That's strange:

  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/pq/pq_ctypes.py", line 205, in status
TypeError: 'NoneType' object is not callable

impl.PQstatus is the only thing that gets called there

psycopg/psycopg/psycopg/pq/pq_ctypes.py

Lines 202 to 206 in fc21688

	@property
	def status(self) -> int:
	return impl.PQstatus(self._pgconn_ptr)

and I don't see how it can become None, because it's just a module member:

psycopg/psycopg/psycopg/pq/_pq_ctypes.py

Lines 201 to 205 in fc21688

	PQstatus = pq.PQstatus
	PQstatus.argtypes = [PGconn_ptr]
	PQstatus.restype = c_int

---

Even stranger:

  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/pq/pq_ctypes.py", line 91, in __del__
TypeError: 'NoneType' object is not callable

psycopg/psycopg/psycopg/pq/pq_ctypes.py

Lines 87 to 93 in fc21688

	def __del__(self) -> None:
	# Close the connection only if it was created in this process,
	# not if this object is being GC'd after fork.
	if os.getpid() == self._procpid:
	self.finish()

How can os.getpid become None?

---

This other one is similar to the first:

  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/pq/pq_ctypes.py", line 733, in clear
TypeError: 'NoneType' object is not callable

psycopg/psycopg/psycopg/pq/pq_ctypes.py

Lines 729 to 734 in fc21688

	def clear(self) -> None:
	self._pgresult_ptr, p = None, self._pgresult_ptr
	if p:
	impl.PQclear(p)

---

I have no idea how can the interpreter get in that state. Is this a multithread program? Or has it got any strange runtime characteristics?

[cvickery]

I'm just as mystified. It's not multithreaded, and all db queries (select/insert) work fine; results are committed okay at the end of the with blocks, etc. I tried putting an explicit call to exit() at the end of the program, wrapped in a try block, but couldn't catch the Exception.

[dvarrazzo]

If you could share a script to reproduce the problem it would be great :)

[cvickery]

I'm adding more info, in case it helps. I have a module called format_rules.py that caused the problem when run as __main__. I reverted that module to psycopg2 and eliminated the problem for that use case.

But when I import that module into a separate app (but don't use any of the functions it defines), the problem happens. The traceback in the original msg was from that separate app. If I remove import format_rules from the second app, the problem goes away. If I put it back in, I get the error again. I tried clearing __pycache__/format_rules* and still get this same scenario: the module, using psycopg2 works standalone without a problem, but can not be imported into a different app without causing the problem.

[tshirtman]

Can you try running python with -v so it shows when modules are loaded and unloaded? since the __del__ method is called at exit, maybe the os module has been cleaned up already. although in my quick test, it seems to be one of the last modules to go, after the deletion of my object, there might be something to it.

gabriel@gryphon:~> python -v
import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'posix' # <class '_frozen_importlib.BuiltinImporter'>
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'time' # <class '_frozen_importlib.BuiltinImporter'>
import 'zipimport' # <class '_frozen_importlib.FrozenImporter'>
# installed zipimport hook
# /usr/lib/python3.8/encodings/__pycache__/__init__.cpython-38.pyc matches /usr/lib/python3.8/encodings/__init__.py
# code object from '/usr/lib/python3.8/encodings/__pycache__/__init__.cpython-38.pyc'
# /usr/lib/python3.8/__pycache__/codecs.cpython-38.pyc matches /usr/lib/python3.8/codecs.py
# code object from '/usr/lib/python3.8/__pycache__/codecs.cpython-38.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeeb26400>
# /usr/lib/python3.8/encodings/__pycache__/aliases.cpython-38.pyc matches /usr/lib/python3.8/encodings/aliases.py
# code object from '/usr/lib/python3.8/encodings/__pycache__/aliases.cpython-38.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeeac29d0>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeeb26160>
# /usr/lib/python3.8/encodings/__pycache__/utf_8.cpython-38.pyc matches /usr/lib/python3.8/encodings/utf_8.py
# code object from '/usr/lib/python3.8/encodings/__pycache__/utf_8.cpython-38.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeeb26fd0>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# /usr/lib/python3.8/encodings/__pycache__/latin_1.cpython-38.pyc matches /usr/lib/python3.8/encodings/latin_1.py
# code object from '/usr/lib/python3.8/encodings/__pycache__/latin_1.cpython-38.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeeac2ac0>
# /usr/lib/python3.8/__pycache__/io.cpython-38.pyc matches /usr/lib/python3.8/io.py
# code object from '/usr/lib/python3.8/__pycache__/io.cpython-38.pyc'
# /usr/lib/python3.8/__pycache__/abc.cpython-38.pyc matches /usr/lib/python3.8/abc.py
# code object from '/usr/lib/python3.8/__pycache__/abc.cpython-38.pyc'
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeeac2f70>
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeeac2ca0>
# /usr/lib/python3.8/__pycache__/site.cpython-38.pyc matches /usr/lib/python3.8/site.py
# code object from '/usr/lib/python3.8/__pycache__/site.cpython-38.pyc'
# /usr/lib/python3.8/__pycache__/os.cpython-38.pyc matches /usr/lib/python3.8/os.py
# code object from '/usr/lib/python3.8/__pycache__/os.cpython-38.pyc'
# /usr/lib/python3.8/__pycache__/stat.cpython-38.pyc matches /usr/lib/python3.8/stat.py
# code object from '/usr/lib/python3.8/__pycache__/stat.cpython-38.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeea758e0>
# /usr/lib/python3.8/__pycache__/_collections_abc.cpython-38.pyc matches /usr/lib/python3.8/_collections_abc.py
# code object from '/usr/lib/python3.8/__pycache__/_collections_abc.cpython-38.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeea75970>
# /usr/lib/python3.8/__pycache__/posixpath.cpython-38.pyc matches /usr/lib/python3.8/posixpath.py
# code object from '/usr/lib/python3.8/__pycache__/posixpath.cpython-38.pyc'
# /usr/lib/python3.8/__pycache__/genericpath.cpython-38.pyc matches /usr/lib/python3.8/genericpath.py
# code object from '/usr/lib/python3.8/__pycache__/genericpath.cpython-38.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeeaab2b0>
import 'posixpath' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeea75a00>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeead9f40>
# /usr/lib/python3.8/__pycache__/_sitebuiltins.cpython-38.pyc matches /usr/lib/python3.8/_sitebuiltins.py
# code object from '/usr/lib/python3.8/__pycache__/_sitebuiltins.cpython-38.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeeaec8e0>
# /usr/lib/python3.8/__pycache__/_bootlocale.cpython-38.pyc matches /usr/lib/python3.8/_bootlocale.py
# code object from '/usr/lib/python3.8/__pycache__/_bootlocale.cpython-38.pyc'
import '_locale' # <class '_frozen_importlib.BuiltinImporter'>
import '_bootlocale' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeea754f0>
# /usr/lib/python3.8/__pycache__/types.cpython-38.pyc matches /usr/lib/python3.8/types.py
# code object from '/usr/lib/python3.8/__pycache__/types.cpython-38.pyc'
import 'types' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeeaab9a0>
# /usr/lib/python3.8/importlib/__pycache__/__init__.cpython-38.pyc matches /usr/lib/python3.8/importlib/__init__.py
# code object from '/usr/lib/python3.8/importlib/__pycache__/__init__.cpython-38.pyc'
# /usr/lib/python3.8/__pycache__/warnings.cpython-38.pyc matches /usr/lib/python3.8/warnings.py
# code object from '/usr/lib/python3.8/__pycache__/warnings.cpython-38.pyc'
import 'warnings' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeea24580>
import 'importlib' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeea24220>
# /usr/lib/python3.8/importlib/__pycache__/util.cpython-38.pyc matches /usr/lib/python3.8/importlib/util.py
# code object from '/usr/lib/python3.8/importlib/__pycache__/util.cpython-38.pyc'
# /usr/lib/python3.8/importlib/__pycache__/abc.cpython-38.pyc matches /usr/lib/python3.8/importlib/abc.py
# code object from '/usr/lib/python3.8/importlib/__pycache__/abc.cpython-38.pyc'
# /usr/lib/python3.8/importlib/__pycache__/machinery.cpython-38.pyc matches /usr/lib/python3.8/importlib/machinery.py
# code object from '/usr/lib/python3.8/importlib/__pycache__/machinery.cpython-38.pyc'
import 'importlib.machinery' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeea36dc0>
import 'importlib.abc' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeea36100>
# /usr/lib/python3.8/__pycache__/contextlib.cpython-38.pyc matches /usr/lib/python3.8/contextlib.py
# code object from '/usr/lib/python3.8/__pycache__/contextlib.cpython-38.pyc'
# /usr/lib/python3.8/collections/__pycache__/__init__.cpython-38.pyc matches /usr/lib/python3.8/collections/__init__.py
# code object from '/usr/lib/python3.8/collections/__pycache__/__init__.cpython-38.pyc'
# /usr/lib/python3.8/__pycache__/operator.cpython-38.pyc matches /usr/lib/python3.8/operator.py
# code object from '/usr/lib/python3.8/__pycache__/operator.cpython-38.pyc'
import '_operator' # <class '_frozen_importlib.BuiltinImporter'>
import 'operator' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbee9e9310>
# /usr/lib/python3.8/__pycache__/keyword.cpython-38.pyc matches /usr/lib/python3.8/keyword.py
# code object from '/usr/lib/python3.8/__pycache__/keyword.cpython-38.pyc'
import 'keyword' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbee9e9400>
# /usr/lib/python3.8/__pycache__/heapq.cpython-38.pyc matches /usr/lib/python3.8/heapq.py
# code object from '/usr/lib/python3.8/__pycache__/heapq.cpython-38.pyc'
import '_heapq' # <class '_frozen_importlib.BuiltinImporter'>
import 'heapq' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbee9e9580>
import 'itertools' # <class '_frozen_importlib.BuiltinImporter'>
# /usr/lib/python3.8/__pycache__/reprlib.cpython-38.pyc matches /usr/lib/python3.8/reprlib.py
# code object from '/usr/lib/python3.8/__pycache__/reprlib.cpython-38.pyc'
import 'reprlib' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbee9e9910>
import '_collections' # <class '_frozen_importlib.BuiltinImporter'>
import 'collections' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbee9cb4f0>
# /usr/lib/python3.8/__pycache__/functools.cpython-38.pyc matches /usr/lib/python3.8/functools.py
# code object from '/usr/lib/python3.8/__pycache__/functools.cpython-38.pyc'
import '_functools' # <class '_frozen_importlib.BuiltinImporter'>
import 'functools' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbee9cb6a0>
import 'contextlib' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeea367c0>
import 'importlib.util' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeea24400>
# possible namespace for /usr/lib/python3/dist-packages/ruamel
# /usr/lib/python3.8/__pycache__/sitecustomize.cpython-38.pyc matches /usr/lib/python3.8/sitecustomize.py
# code object from '/usr/lib/python3.8/__pycache__/sitecustomize.cpython-38.pyc'
# /usr/lib/python3/dist-packages/__pycache__/apport_python_hook.cpython-38.pyc matches /usr/lib/python3/dist-packages/apport_python_hook.py
# code object from '/usr/lib/python3/dist-packages/__pycache__/apport_python_hook.cpython-38.pyc'
import 'apport_python_hook' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbee9f0f70>
import 'sitecustomize' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeea413a0>
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbeead9910>
Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
# extension module 'readline' loaded from '/usr/lib/python3.8/lib-dynload/readline.cpython-38-x86_64-linux-gnu.so'
# extension module 'readline' executed from '/usr/lib/python3.8/lib-dynload/readline.cpython-38-x86_64-linux-gnu.so'
import 'readline' # <_frozen_importlib_external.ExtensionFileLoader object at 0x7fdbee9cb6d0>
import 'atexit' # <class '_frozen_importlib.BuiltinImporter'>
# /usr/lib/python3.8/__pycache__/rlcompleter.cpython-38.pyc matches /usr/lib/python3.8/rlcompleter.py
# code object from '/usr/lib/python3.8/__pycache__/rlcompleter.cpython-38.pyc'
import 'rlcompleter' # <_frozen_importlib_external.SourceFileLoader object at 0x7fdbee9f4d90>
>>> class t:
...     def __del__(self):
...             print("bye")
... 
>>> a = t()
>>> exit()
# clear builtins._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# clear sys.__interactivehook__
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
# cleanup[2] removing sys
# cleanup[2] removing builtins
# cleanup[2] removing _frozen_importlib
# cleanup[2] removing _imp
# cleanup[2] removing _warnings
# cleanup[2] removing _io
# cleanup[2] removing marshal
# cleanup[2] removing posix
# cleanup[2] removing _frozen_importlib_external
# cleanup[2] removing _thread
# cleanup[2] removing _weakref
# cleanup[2] removing time
# cleanup[2] removing zipimport
# destroy zipimport
# cleanup[2] removing _codecs
# cleanup[2] removing codecs
# cleanup[2] removing encodings.aliases
# cleanup[2] removing encodings
# destroy encodings
# cleanup[2] removing encodings.utf_8
# cleanup[2] removing _signal
# cleanup[2] removing __main__
# cleanup[2] removing encodings.latin_1
# cleanup[2] removing _abc
# cleanup[2] removing abc
# cleanup[2] removing io
# cleanup[2] removing _stat
# cleanup[2] removing stat
# cleanup[2] removing _collections_abc
# cleanup[2] removing genericpath
# cleanup[2] removing posixpath
# cleanup[2] removing os.path
# cleanup[2] removing os
# cleanup[2] removing _sitebuiltins
# cleanup[2] removing _locale
# cleanup[2] removing _bootlocale
# destroy _bootlocale
# cleanup[2] removing types
# cleanup[2] removing importlib._bootstrap
# cleanup[2] removing importlib._bootstrap_external
# cleanup[2] removing warnings
# cleanup[2] removing importlib
# destroy importlib
# cleanup[2] removing importlib.machinery
# cleanup[2] removing importlib.abc
# cleanup[2] removing _operator
# cleanup[2] removing operator
# destroy operator
# cleanup[2] removing keyword
# destroy keyword
# cleanup[2] removing _heapq
# cleanup[2] removing heapq
# cleanup[2] removing itertools
# cleanup[2] removing reprlib
# destroy reprlib
# cleanup[2] removing _collections
# cleanup[2] removing collections
# destroy collections
# cleanup[2] removing _functools
# cleanup[2] removing functools
# cleanup[2] removing contextlib
# destroy contextlib
# cleanup[2] removing importlib.util
# cleanup[2] removing mpl_toolkits
# destroy mpl_toolkits
# cleanup[2] removing ruamel
# destroy ruamel
# cleanup[2] removing apport_python_hook
# cleanup[2] removing sitecustomize
# destroy sitecustomize
# destroy apport_python_hook
# cleanup[2] removing site
# destroy site
# cleanup[2] removing readline
# cleanup[2] removing atexit
# cleanup[2] removing rlcompleter
# destroy rlcompleter
bye
# destroy time
# destroy _signal
# destroy _sitebuiltins
# destroy io
# destroy _locale
# destroy importlib.util
# destroy importlib.abc
# destroy functools
# destroy types
# destroy importlib.machinery
# destroy warnings
# destroy _collections_abc
# destroy heapq
# destroy _operator
# destroy _heapq
# destroy itertools
# destroy _collections
# destroy atexit
# destroy __main__
# destroy readline
# destroy _functools
# cleanup[3] wiping importlib._bootstrap_external
# cleanup[3] wiping importlib._bootstrap
# destroy _frozen_importlib_external
# cleanup[3] wiping os
# destroy abc
# destroy posixpath
# cleanup[3] wiping genericpath
# cleanup[3] wiping stat
# cleanup[3] wiping _stat
# destroy _stat
# cleanup[3] wiping _abc
# cleanup[3] wiping encodings.latin_1
# cleanup[3] wiping encodings.utf_8
# cleanup[3] wiping encodings.aliases
# cleanup[3] wiping codecs
# cleanup[3] wiping _codecs
# cleanup[3] wiping _weakref
# destroy _weakref
# cleanup[3] wiping _thread
# destroy _thread
# cleanup[3] wiping posix
# destroy posix
# cleanup[3] wiping marshal
# destroy marshal
# cleanup[3] wiping _io
# destroy io
# cleanup[3] wiping _warnings
# destroy _warnings
# cleanup[3] wiping _imp
# destroy _imp
# cleanup[3] wiping _frozen_importlib
# cleanup[3] wiping sys
# cleanup[3] wiping builtins
# destroy _abc
# destroy os
# destroy stat
# destroy genericpath
# destroy _frozen_importlib
# clear sys.audit hooks

[cvickery]

The output is too long (277K). I'll look at it after supper and see if I can see what you're looking for.

[tshirtman]

it seems tho os module is indeed cleaned up and destroyed before the __del__ code runs..

line 1723

# cleanup[2] removing os.path
# cleanup[2] removing os
# cleanup[2] removing _sitebuiltins
# cleanup[2] removing types
# cleanup[2] removing importlib._bootstrap

line 2881

# cleanup[3] wiping types
# cleanup[3] wiping os
# cleanup[3] wiping os.path
# destroy genericpath
# cleanup[3] wiping posixpath
# cleanup[3] wiping stat
# cleanup[3] wiping _stat
# destroy _stat
# cleanup[3] wiping __main__
# destroy argparse
# destroy csv
# destroy format_rules
# destroy openpyxl.worksheet
Exception ignored in: <function BaseConnection.__del__ at 0x10267d6c0>
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/connection.py", line 140, in __del__
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/connection.py", line 161, in closed
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/pq/pq_ctypes.py", line 205, in status
TypeError: 'NoneType' object is not callable
Exception ignored in: <function PGconn.__del__ at 0x1024de440>
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/pq/pq_ctypes.py", line 91, in __del__
TypeError: 'NoneType' object is not callable
Exception ignored in: <function PGresult.__del__ at 0x10255cc10>
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/pq/pq_ctypes.py", line 723, in __del__
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/pq/pq_ctypes.py", line 733, in clear
TypeError: 'NoneType' object is not callable
# cleanup[3] wiping io
# destroy abc
# cleanup[3] wiping _abc

As for why things happen in that order i'm not sure.

there is this very old essay from before python1.5 (so not sure how much still applies, but that's what turned up in a quick search), about cleanup order https://www.python.org/doc/essays/cleanup/ that might be useful.
and the code doing that is in this module of cpython, https://github.com/python/cpython/blob/9130a4d62032468e0d4949aaa49c29afb0d854ca/Python/pylifecycle.c#L1377

[dvarrazzo]

@tshirtman do you have a repro? What is format_rules? Can't google or pypi (verb) for it...

[tshirtman]

No, i just looked at the file shared by @cvickery, i don't know about the format_rules module either, it's internal to the app as Christopher said earlier, it would be interesting indeed to have the content to see if there is anything fishy.

This might not be a bug in psycopg.

edit: i guess it is this module https://github.com/cvickery/transfer-app/blob/cfdaa24fe78b0a95cfe15dae828c02b901f388dc/format_rules.py

Nothing obvious to me, hope this helps anyway… ++

[dvarrazzo]

@tshirtman did @cvickery share a file? ... oh yes, I got an email about it but can't see it on this page...

This bug is full of mysteries... 👀

As for your suggestion to import the objects in the module scope: we can surely try that, but the pq_ctypes module is deleted pretty early too, so it might trigger the same error...

Anyway the test is cheap. @cvickery could you please try applying this patch and see if the error is solved?

Thank you very much!

diff --git a/psycopg/psycopg/pq/pq_ctypes.py b/psycopg/psycopg/pq/pq_ctypes.py
index 42b86a55..9772bf16 100644
--- a/psycopg/psycopg/pq/pq_ctypes.py
+++ b/psycopg/psycopg/pq/pq_ctypes.py
@@ -8,7 +8,7 @@ implementation.
 
 # Copyright (C) 2020-2021 The Psycopg Team
 
-import os
+from os import getpid
 import logging
 from weakref import ref
 from functools import partial
@@ -23,6 +23,7 @@ from . import _pq_ctypes as impl
 from .misc import PGnotify, ConninfoOption, PGresAttDesc
 from .misc import error_message, connection_summary
 from ._enums import Format, ExecStatus
+from ._pq_ctypes import PQclear, PQfinish, PQfreeCancel
 
 if TYPE_CHECKING:
     from . import abc
@@ -83,12 +84,12 @@ class PGconn:
         )
         impl.PQsetNoticeReceiver(pgconn_ptr, self._notice_receiver, None)
 
-        self._procpid = os.getpid()
+        self._procpid = getpid()
 
     def __del__(self) -> None:
         # Close the connection only if it was created in this process,
         # not if this object is being GC'd after fork.
-        if os.getpid() == self._procpid:
+        if getpid() == self._procpid:
             self.finish()
 
     def __repr__(self) -> str:
@@ -122,7 +123,7 @@ class PGconn:
     def finish(self) -> None:
         self._pgconn_ptr, p = None, self._pgconn_ptr
         if p:
-            impl.PQfinish(p)
+            PQfinish(p)
 
     @property
     def pgconn_ptr(self) -> Optional[int]:
@@ -730,7 +731,7 @@ class PGresult:
     def clear(self) -> None:
         self._pgresult_ptr, p = None, self._pgresult_ptr
         if p:
-            impl.PQclear(p)
+            PQclear(p)
 
     @property
     def pgresult_ptr(self) -> Optional[int]:
@@ -863,7 +864,7 @@ class PGcancel:
         """
         self.pgcancel_ptr, p = None, self.pgcancel_ptr
         if p:
-            impl.PQfreeCancel(p)
+            PQfreeCancel(p)
 
     def cancel(self) -> None:
         """Requests that the server abandon processing of the current command.

[cvickery]

It helped, but was not a total fix. Here's the final output from the app that imports format_rules:

Exception ignored in: <function BaseConnection.__del__ at 0x10290ecb0>
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/connection.py", line 140, in __del__
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/connection.py", line 161, in closed
  File "/opt/homebrew/lib/python3.10/site-packages/psycopg/pq/pq_ctypes.py", line 206, in status
TypeError: 'NoneType' object is not callable

I'll tried reverting format_rules.py to use psycopg instead of psycopg2 to see if that would make a difference. If I now run it as a command, format_rules runs to completion and then raises the exception above. If I remove the import of format_rules from the other app, it runs to completion with no errors (no change from before the patch.)

Chris

[dvarrazzo]

I will report the issue upstream. It seems a Python error to me.

Did you try with older python versions? I've never seen it happening in 3.8 but haven't run 3.10 very much.

[cvickery]

It looks like you're right: the problem doesn't happen with Python 3.9.