Exiting Python session generates TypeError

[jyu00]

Describe the bug

After using IBMRuntimeService in a Python CLI session, when exiting it generated a bunch of TypeError:

Exception ignored in: <function RetrySession.__del__ at 0x13fd5cd30>
Traceback (most recent call last):
  File "/Users/jessieyu/Documents/Q/Github/qiskit-ibm-runtime/qiskit_ibm_runtime/api/session.py", line 173, in __del__
  File "/Users/jessieyu/.pyenv/versions/qiskit-ibm-runtime/lib/python3.8/site-packages/requests/sessions.py", line 747, in close
  File "/Users/jessieyu/.pyenv/versions/qiskit-ibm-runtime/lib/python3.8/site-packages/requests/adapters.py", line 325, in close
  File "/Users/jessieyu/.pyenv/versions/qiskit-ibm-runtime/lib/python3.8/site-packages/urllib3/poolmanager.py", line 222, in clear
  File "/Users/jessieyu/.pyenv/versions/qiskit-ibm-runtime/lib/python3.8/site-packages/urllib3/_collections.py", line 100, in clear
  File "/Users/jessieyu/.pyenv/versions/qiskit-ibm-runtime/lib/python3.8/site-packages/urllib3/poolmanager.py", line 173, in <lambda>
  File "/Users/jessieyu/.pyenv/versions/qiskit-ibm-runtime/lib/python3.8/site-packages/urllib3/connectionpool.py", line 494, in close
TypeError: catching classes that do not inherit from BaseException is not allowed
Exception ignored in: <function RetrySession.__del__ at 0x13fd5cd30>
Traceback (most recent call last):

Steps to reproduce

>>> from qiskit_ibm_runtime import IBMRuntimeService
>>> service = IBMRuntimeService()
>>> backend = service.least_busy()
<ctrl-D>

Expected behavior

Suggested solutions

Additional Information

• Qiskit IBM Runtime version:
• Python version: 3.8.1
• Operating system:

[daka1510]

That's a tricky one. Nailed it down as below and will continue the investigation early next week.

good-case.py

from requests import Session


class CustomSession(Session):
    def __del__(self) -> None:
        self.close()


session = CustomSession()
result = session.get("https://auth.quantum-computing.ibm.com/api/version").json()
print(result)

bad-case.py

from qiskit_ibm_runtime.api.session import RetrySession


session = RetrySession(
    base_url="https://auth.quantum-computing.ibm.com",
    auth=None,
)
# TypeError only seen when a request is made
result = session.get("/api/version").json()
print(result)

Running python good-case.py succeeds, while python bad-case.py fails with the reported TypeError. I.e. something within RetrySession must be the culprit...

[daka1510]

Investigation Results

Observations

• Code fragment that is causing TypeError: catching classes that do not inherit from BaseException. In the failing case, queue.Empty is None.

# File ".../site-packages/urllib3/connectionpool.py", line 494, in close
def close(self):
   # ...
-> except queue.Empty:
      pass

• Entrypoint from code within qiskit-ibm-runtime that leads to the exception:

# File ".../qiskit_ibm_runtime/api/session.py", line 173, in __del__
class RetrySession(Session):
    def __del__(self) -> None:
->        self.close()

• To debug this further I instrumented the custom session code with a print statement as shown below.

import queue
class CustomSession(Session):
    def __del__(self) -> None:
        print("CustomSession#__del__", queue.Empty)
        self.close()

# output in success case: CustomSession#__del__ <class '_queue.Empty'>
# output in failing case: RetrySession#__del__ None

Conclusions

Our code calls session cleanup code from within the destructor of a custom session subclass. The destructor (__del__) is called when the object is destroyed. At this point in time it can apparently happen that queue.Empty returns None. Maybe because Python has cleaned up other objects as well.

I can't find any references which indicate that triggering the cleanup of session information from within the destructor is a supported best practice. At the same time we can't switch to using a session context manager without breaking the APIs that consumers are used to.

Proposed Solution

Ignore errors and do not spend additional time understanding how python cleans up module references under the covers (see #103)

[daka1510]

With #103 the "high priority" part of this issue has been addressed.

Based on the above investigation we can reprioritize the work and come up with a proper fix in a next step. Need a second pair of eyes to help with that and think about clean solutions (cc: @rathishcholarajan)