AutoReconnect: [Errno 22] Invalid argument

[antonst]

While one of my runs were queuing on Stampede I have lost internet connection for a brief moment, which resulted in the following error:

Exception in thread PilotLauncherWorker-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/antons/env8-no-opt/local/lib/python2.7/site-packages/radical.pilot-0.33-py2.7.egg/radical/pilot/controller/pilot_launcher_worker.py", line 302, in run
    "$push": {"statehistory": {"state": LAUNCHING, "timestamp": ts}}}
  File "build/bdist.linux-x86_64/egg/pymongo/collection.py", line 1738, in find_and_modify
    **kwargs)
  File "build/bdist.linux-x86_64/egg/pymongo/database.py", line 439, in command
    uuid_subtype, compile_re, **kwargs)[0]
  File "build/bdist.linux-x86_64/egg/pymongo/database.py", line 338, in _command
    for doc in cursor:
  File "build/bdist.linux-x86_64/egg/pymongo/cursor.py", line 1076, in next
    if len(self.__data) or self._refresh():
  File "build/bdist.linux-x86_64/egg/pymongo/cursor.py", line 1020, in _refresh
    self.__uuid_subtype))
  File "build/bdist.linux-x86_64/egg/pymongo/cursor.py", line 933, in __send_message
    res = client._send_message_with_response(message, **kwargs)
  File "build/bdist.linux-x86_64/egg/pymongo/mongo_client.py", line 1217, in _send_message_with_response
    raise AutoReconnect(str(e))
AutoReconnect: [Errno 22] Invalid argument

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/antons/env8-no-opt/local/lib/python2.7/site-packages/radical.pilot-0.33-py2.7.egg/radical/pilot/controller/pilot_manager_controller.py", line 300, in run
    pilot_list = self._db.get_pilots(pilot_manager_id=self._pm_id)
  File "/home/antons/env8-no-opt/local/lib/python2.7/site-packages/radical.pilot-0.33-py2.7.egg/radical/pilot/db/database.py", line 411, in get_pilots
    for obj in cursor:
  File "build/bdist.linux-x86_64/egg/pymongo/cursor.py", line 1076, in next
    if len(self.__data) or self._refresh():
  File "build/bdist.linux-x86_64/egg/pymongo/cursor.py", line 1020, in _refresh
    self.__uuid_subtype))
  File "build/bdist.linux-x86_64/egg/pymongo/cursor.py", line 933, in __send_message
    res = client._send_message_with_response(message, **kwargs)
  File "build/bdist.linux-x86_64/egg/pymongo/mongo_client.py", line 1217, in _send_message_with_response
    raise AutoReconnect(str(e))
AutoReconnect: [Errno 22] Invalid argument

I don't have any other info available, but hope this might be helpful

[andre-merzky]

I'm afraid I can't help with this one: this is a pymongo layer error which we hit from time to time, usually on long-running experiments. I do not know how to avoid it. It might be worth to open pymongo ticket - but their ticket system (https://jira.mongodb.org/browse/PYTHON/?selectedTab=com.atlassian.jira.jira-projects-plugin:issues-panel) does not allow to add issues (publicly)... Would you mind trying to find out how to report it? Thanks!

[antonst]

Will do.

[andre-merzky]

Thanks! :)

[oleweidner]

I have experienced this in a completely different context. This is the pymongo layer auto-reconnect failing. You might want to catch that error on application level (RP level) and do a manual connection close / connection open as a last resort, possible with some simple timeout mechanism. This worked for me.

[antonst]

Thanks for the insightful comment Ole!

[andre-merzky]

closed in favor of #442

[tumikosha]

I have this error when Linux do not have free sockets.
this code do not close linux sockets properly:

client = pymongo.MongoClient(connection_str, maxIdleTimeMS=TIMEOUT, socketTimeoutMS=TIMEOUT,
                                 connectTimeoutMS=TIMEOUT, serverSelectionTimeoutMS=TIMEOUT)
    try:
        db_names = client.list_database_names()
    except Exception as e:
         print('Exception: ', str(e))
    client.close()

[andre-merzky]

@tumikosha : Hi - are you sure you replied on the right ticket? This is closed since about 3 years... ;-)