Queries fail with a NetworkException in versions released after 3.1.0.

[mstfdkmn]

iRods server: rods5.0.2

We have encountered a NetworkException while running scripts that contain queries. To reproduce this issue, please execute the script provided below in the latest PRC version:

import os
import ssl
from irods.session import iRODSSession
from irods.models import DataObject, Collection
from irods.column import Criterion

env_file = os.path.expanduser("~/.irods/irods_environment.json")
ssl_context = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH, cafile=None, capath=None, cadata=None)
ssl_settings = {'ssl_context': ssl_context}

s = iRODSSession(irods_env_file=env_file, **ssl_settings)
col = s.collections.get('/vsc/home/public')
query = s.query(Collection, DataObject)
query.filter(Criterion("=", Collection.name, col.path))
results = [r for r in query] #HERE IS WHERE I GET THE ERROR
print(results)

The result is:

root@aaaed9abf832:/# python3 test.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/irods/connection.py", line 166, in recv
    msg = iRODSMessage.recv(self.socket)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/irods/message/__init__.py", line 346, in recv
    rsp_header_size = _recv_message_in_len(sock, 4)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/irods/message/__init__.py", line 251, in _recv_message_in_len
    raise socket.error(msg)
OSError: Read 0 bytes from socket instead of expected 4 bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "//test.py", line 21, in <module>
    results = [r for r in query]
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/irods/query.py", line 249, in get_results
    for result_set in self.get_batches():
  File "/usr/local/lib/python3.12/dist-packages/irods/query.py", line 233, in get_batches
    result_set = self.execute()
                 ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/irods/query.py", line 206, in execute
    result_message = conn.recv()
                     ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/irods/connection.py", line 177, in recv
    raise NetworkException("Could not receive server response")
irods.exception.NetworkException: Could not receive server response

Following an investigation, we initially suspected pam_interactive authentication as the cause however, native authentication also failed. I have since confirmed that the script functions correctly using python-irodsclient==3.1.0, as demonstrated below.

root@aaaed9abf832:/# pip uninstall python-irodsclient --break-system-packages
Found existing installation: python-irodsclient 3.3.0
Uninstalling python-irodsclient-3.3.0:
  Would remove:
    /usr/local/bin/prc_write_irodsA.py
    /usr/local/lib/python3.12/dist-packages/build/lib/irods/*
    /usr/local/lib/python3.12/dist-packages/irods/*
    /usr/local/lib/python3.12/dist-packages/python_irodsclient-3.3.0.dist-info/*
    /usr/local/lib/python3.12/dist-packages/venv313/bin/prc_write_irodsA.py
Proceed (Y/n)? Y
  Successfully uninstalled python-irodsclient-3.3.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@aaaed9abf832:/# pip install python-irodsclient==3.1.0 --break-system-packages
Collecting python-irodsclient==3.1.0
  Using cached python_irodsclient-3.1.0-py3-none-any.whl.metadata (73 kB)
Requirement already satisfied: PrettyTable>=0.7.2 in /usr/local/lib/python3.12/dist-packages (from python-irodsclient==3.1.0) (3.16.0)
Requirement already satisfied: defusedxml in /usr/local/lib/python3.12/dist-packages (from python-irodsclient==3.1.0) (0.7.1)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.12/dist-packages (from PrettyTable>=0.7.2->python-irodsclient==3.1.0) (0.2.13)
Using cached python_irodsclient-3.1.0-py3-none-any.whl (263 kB)
Installing collected packages: python-irodsclient
Successfully installed python-irodsclient-3.1.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@aaaed9abf832:/# python3 test.py
[]

We looked into the different versions but aren't exactly sure where the issue is coming from, even though 3.2.0 has some changes related to connections.

[korydraughn]

Does anything appear in the server's log file?

[mstfdkmn]

No, the server logs show nothing relevant.

[mstfdkmn]

I am seeing entries that appear to be related, though I cannot confirm they are caused by this issue.

[2026-04-07T15:45:34.862Z][hpc-p-cloud-2] {"processname": "irods", "groupname": "irods", "pid": "110", "channel": "stderr", "msg": "free(): invalid pointer"}
[2026-04-07T15:45:47.689Z][hpc-p-cloud-2] {"processname": "proxy", "groupname": "proxy", "pid": "348", "channel": "stderr", "msg": "2026/04/07 17:45:47 error forwarding connection from [2a02:2c40:0:240a:3::7]:35232 at port 1247: out side closed a second ago"}

[korydraughn]

That free(): invalid pointer message stands out. Looks like an agent crashed.

We might be able to determine if it's the agent we care about by using msi_get_agent_pid.

• https://docs.irods.org/5.0.2/doxygen/get__agent__pid_8cpp.html#a0cf53fa9b2955dd97324a33e16cad03d

Before running the query, you'd invoke the microservice to get the PID of the agent. However, if the PRC contains a pool of connections and uses different ones on every interaction with the server, then your only way of confirming the PID is to invoke the microservice on each connection in the pool.

---

Do you see the same problem when SSL isn't involved?

[d-w-moore]

@mstfdkmn In iRODS, GenQuery1 cannot currently handle querying all columns of the Collection and DataObject models together. I've been able to duplicate that problem not just in PRC but also using iquest. You'll have to exclude at least one column from either Collection or DataObject ....

Look at the first example in the README of releases following v3.2.0 -- there is an example of how to do this using the syntactic sugar of "negated" columns in your parameter list.
and in v3.2.0 of PRC you can do the following to exclude the status column (DB column name is D_DATA_STATUS) which unlike replica status is currently an unused column

Here's a trivial example that might be of use:

list( session.query( Collection, DataObject, -DataObject.status ))

This should help you to avoid the NetworkException by preventing the agent from crashing.

[mstfdkmn]

Do you see the same problem when SSL isn't involved?

Hi @korydraughn, we tried and it looks like ssl has no impact on this bug.

[mstfdkmn]

Thank you, @d-w-moore. Below are my observations regarding this issue:

• I am unable to reproduce the issue using iquest.
• The script functions correctly when using the syntax you specified (excluding data object status column): list( session.query( Collection, DataObject, -DataObject.status )).
• A bit strange but it looks like the behavior is inconsistent (at least for the empty collections); the issue does not always occur with the same collection (e.g., the public collection) under identical circumstances. I have no further details to provide at this time.

[d-w-moore]

Thank you, @d-w-moore. Below are my observations regarding this issue:

* I am unable to reproduce the issue using iquest.

* The script functions correctly when using the syntax you specified (excluding data object status column): `list( session.query( Collection, DataObject, -DataObject.status ))`.

* A bit strange but it looks like the behavior is inconsistent (at least for the empty collections); the issue does not always occur with the same collection (e.g., the public collection) under identical circumstances. I have no further details to provide at this time.

@mstfdkmn
To reproduce using iquest , consider the following Bash script:

iquest "select
DATA_ID,
DATA_COLL_ID,
DATA_NAME,
DATA_REPL_NUM,
DATA_VERSION,
DATA_TYPE_NAME,
DATA_SIZE,
DATA_RESC_NAME,
DATA_PATH,
DATA_OWNER_NAME,
DATA_OWNER_ZONE,
DATA_REPL_STATUS,
DATA_STATUS,
DATA_CHECKSUM,
DATA_EXPIRY,
DATA_MAP_ID,
DATA_COMMENTS,
DATA_CREATE_TIME,
DATA_MODIFY_TIME,
DATA_RESC_HIER,
DATA_RESC_ID,
DATA_ACCESS_TIME,
COLL_ID,
COLL_NAME,
COLL_PARENT_NAME,
COLL_OWNER_NAME,
COLL_OWNER_ZONE,
COLL_MAP_ID,
COLL_INHERITANCE,
COLL_COMMENTS,
COLL_CREATE_TIME,
COLL_MODIFY_TIME"

Invoking this on iRODS 5.0+ as a shell command returns SYS_INTERNAL_ERR (probably due to the same agent crash as created the NetworkError in the Python client.) But deleting any of the lines that name a column allows the query to function without error.

I am glad you were able to get the suggestion to work!

As to the variable behavior on the same collection, perhaps it is a statistical error that sometimes depends on row content. But I have not, thus far, been able to track down why the server does this.

[korydraughn]

Related to irods/irods#8574.