bug: when connecting to databricks pyspark sql endpoint, geting invalid handler error from server

[ruiyang2015]

What happened?

Getting this warning logs when use the databricks pyspark compute sql endpoint

WARNING:root:Error closing operation handle: Invalid OperationHandle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=17690c3f-24ee-4dd9-922e-8855c709db53]
WARNING:root:Error closing operation handle: Invalid OperationHandle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=066c6504-93a2-4a70-82a7-f739a911c800]
WARNING:root:Error closing operation handle: Invalid OperationHandle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=1f8c2158-375e-4f27-8c7a-ada651204e90]

and some times getting the exception like this:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/.venv/lib/python3.12/site-packages/ibis/backends/databricks/__init__.py", line 222, in create_table
    with self._safe_raw_sql(create_stmt) as cur:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/app/.venv/lib/python3.12/site-packages/ibis/backends/databricks/__init__.py", line 327, in _safe_raw_sql
    with self.con.cursor() as cur:
         ^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/client.py", line 473, in __exit__
    self.close()
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/client.py", line 1204, in close
    self._close_and_clear_active_result_set()
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/client.py", line 616, in _close_and_clear_active_result_set
    self.active_result_set.close()
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/client.py", line 1534, in close
    self.thrift_backend.close_command(self.command_id)
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/thrift_backend.py", line 1095, in close_command
    resp = self.make_request(self._client.CloseOperation, req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/thrift_backend.py", line 486, in make_request
    ThriftBackend._check_response_for_error(response)
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/thrift_backend.py", line 263, in _check_response_for_error
    raise DatabaseError(response.status.errorMessage)
databricks.sql.exc.DatabaseError: Invalid OperationHandle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=d7537beb-c663-46c4-bef4-8055c88c020f]

they are raised from the Cursor.close() call, it seems when trying to close the cursor to a pyspark sql endpoint, server side will raise error. Ideally ibis lib should handle this error gracefully. If server does not support close, library should not hard fail entirely

What version of ibis are you using?

10.5.0

What backend(s) are you using, if any?

Databricks.

Relevant log output

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/.venv/lib/python3.12/site-packages/ibis/backends/databricks/__init__.py", line 222, in create_table
    with self._safe_raw_sql(create_stmt) as cur:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/app/.venv/lib/python3.12/site-packages/ibis/backends/databricks/__init__.py", line 327, in _safe_raw_sql
    with self.con.cursor() as cur:
         ^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/client.py", line 473, in __exit__
    self.close()
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/client.py", line 1204, in close
    self._close_and_clear_active_result_set()
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/client.py", line 616, in _close_and_clear_active_result_set
    self.active_result_set.close()
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/client.py", line 1534, in close
    self.thrift_backend.close_command(self.command_id)
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/thrift_backend.py", line 1095, in close_command
    resp = self.make_request(self._client.CloseOperation, req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/thrift_backend.py", line 486, in make_request
    ThriftBackend._check_response_for_error(response)
  File "/app/.venv/lib/python3.12/site-packages/databricks/sql/thrift_backend.py", line 263, in _check_response_for_error
    raise DatabaseError(response.status.errorMessage)
databricks.sql.exc.DatabaseError: Invalid OperationHandle: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=d7537beb-c663-46c4-bef4-8055c88c020f]

Code of Conduct

• I agree to follow this project's Code of Conduct

[cpcloud]

Thanks for opening an issue.

It's not clear from your report that close isn't supported. The error is Invalid OperationHandle, which doesn't seem to necessarily mean "close isn't a supported operation", it could mean that whatever handle the API is trying to access doesn't exist for whatever reason.

Even if it did mean that close wasn't a supported operation, that seems like a Databricks problem, not an Ibis problem. We're using the context manager protocol that's implemented in the Databricks library. If close() is a no-op, or isn't supported then either the functionality or error should reflect that.

[ruiyang2015]

turn out to be an issue with databricks-connector-sql, the 4.0.4 has a but, and it is fixed in 4.0.5