Filedescriptor out of range error in v3.1.8

[Kim-Hammar]

I encounter the following error when running psycopg 3.1.8 on python 3.9 on a server with a large number of open file descriptors. This is the stack trace:

File "python3.9/site-packages/psycopg/cursor.py", line 719, in execute
    self._conn.wait(
  File python3.9/site-packages/psycopg/connection.py", line 957, in wait
    return waiting.wait(gen, self.pgconn.socket, timeout=timeout)
  File "python3.9/site-packages/psycopg/waiting.py", line 219, in wait_select
    rl, wl, xl = select.select(
ValueError: filedescriptor out of range in select()

ulimit have been set to allow for the large number of FDs so this is not the issue. Downgrading to psycopg v3.1.4 fixes the issue.

[dvarrazzo]

Hello,

which platform are you using?

I seem to understand you are using only the psycopg package, not psycopg[c] or [binary]; is that correct?

You shouldn't have this issue if you use the c or binary extension; however, it would be better if you check it.

We should identify better the condition of failures: you are not giving a lot of details. It's also curious that downgrading to 3.1.4 fixes the problem, because a change in 3.1.5 happened in the c/binary package and the select module shouldn't be used in the new code (poll should be used).

So, I appreciate that you are finding an issue, and it is a known issue (select on Linux only supports up to 1024 file descriptors). But this is worked around, at least in the C code, by the use of poll. We might have to make sure to not use select in the Python code either, but the Python implementation shouldn't have changed in 3.1.5.

Please help us to understand the problem better.

[Kim-Hammar]

Hello,

Thank you for the response. Based on your response and after some poking around I think the issue was that I did not have the psycopg[binary]. I got it to work now with v3.1.8 by reinstalling psycopg with the binary extension.

Closing the issue.

[dvarrazzo]

Thank you for checking! However, even the base psycopg module should work with a large FDs number. So, I'll keep the issue open to fix the problem in the Python path.

[Kim-Hammar]

Cool. The platform was Ubuntu 22, Postgres 15.2

[dvarrazzo]

Fixed in 531a040. Fix to be released in 3.1.9.

[bharadwajyarlagadda]

Hey @dvarrazzo, we are facing the same issue:

2023-04-20T13:28:04.417291Z |     1 | 139809410157368 | 1160808107 |    MainProcess-Thread-7729 (job) |                    base:36   | ERROR   | Unable to connect to the database <table_name> due to an unexpected error: filedescriptor out of range in select()
Traceback (most recent call last):
  File "/app/app/database/base.py", line 34, in connect
    yield conn
  File "/app/app/database/orion_job.py", line 24, in _table_exists
    cursor.execute(
  File "/usr/lib/python3.10/site-packages/psycopg/cursor.py", line 719, in execute
    self._conn.wait(
  File "/usr/lib/python3.10/site-packages/psycopg/connection.py", line 957, in wait
    return waiting.wait(gen, self.pgconn.socket, timeout=timeout)
  File "/usr/lib/python3.10/site-packages/psycopg/waiting.py", line 219, in wait_select
    rl, wl, xl = select.select(
ValueError: filedescriptor out of range in select()

Python version being used: 3.10
psycopg version: 3.1.8
Here are the dependencies that were installed as part of our docker container:

/app # pip freeze
Babel==2.11.0
certifi==2022.12.7
decorator==5.1.1
elasticsearch==7.14.1
elasticsearch-dsl==7.4.1
iso8601==1.1.0
kazoo==2.5.0
marshmallow==3.19.0
numpy==1.23.5
packaging==23.0
pandas==1.1.5
psycopg==3.1.8
py==1.11.0
pykafka==2.8.0
python-dateutil==2.8.2
pytimeparse==1.1.8
pytz==2022.6
retry==0.9.2
schedule==1.1.0
six==1.16.0
tabulate==0.9.0
typing_extensions==4.5.0
urllib3==1.26.13
zulu==2.0.0

The operating system for the docker container is:

/app # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.16.2
PRETTY_NAME="Alpine Linux v3.16"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

I see that you mentioned that the above fix is to be released in 3.1.9. Could you let us know when can this version be released? Currently we are facing this issue on critical production systems.

It would be great if the version 3.1.9 gets released today (or) is there any other way to bypass this?

[dvarrazzo]

I can release it next week, not today.

Today you can:

• install psycopg[binary] or psycopg[c] to work around the problem.
• export the env var PSYCOPG_WAIT_FUNC=wait_epoll to use a function that is not affected by the issue.

You are losing a lot of performance by not using [binary] or [c]. I suggest you add one of them to your container. See the install docs for eventual prerequisites.