Connectivity of PyIceberg to Hive Metastore on HPE

[dev-sajal]

Question

I am trying to connect with Hive Metastore with HPE, PyIceberg isn't able to connect with the same. While PySpark could connect to the HMS on the same platform. We are using a mapr keytab. I would like to check if the functionality is supported by PyIceberg at the moment.

Our debugging on the issue is: Kerberized environment without HTTP (default for HPE) isn't supported by thrift or pyiceberg.

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "hive",
    **{
        "uri": config["THRIFT_SERVER"],
        "s3.endpoint": config["ENDPOINT_URL"],
        "s3.access-key-id": config["ACCESS_KEY_ID"],
        "s3.secret-access-key": config["SECRET_ACCESS_KEY"],
        "hive.kerberos-authentication": "true",
        "hive.metastore.sasl.enabled": "true",
        "hive.kerberos-service-name": "host"
    }
)

Error log:

Traceback (most recent call last):
  File "/opt/hpe_venv/lib64/python3.11/site-packages/pyiceberg/catalog/hive.py", line 186, in __enter__
    self._transport.open()
  File "/opt/hpe_venv/lib64/python3.11/site-packages/thrift/transport/TTransport.py", line 397, in open
    raise TTransportException(
thrift.transport.TTransport.TTransportException: Bad SASL negotiation status: 3 (b'GSS initiate failed')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/hpe_venv/lib64/python3.11/site-packages/pyiceberg/catalog/hive.py", line 733, in list_tables
    with self._client as open_client:
  File "/opt/hpe_venv/lib64/python3.11/site-packages/pyiceberg/catalog/hive.py", line 190, in __enter__
    self._transport.open()
  File "/opt/hpe_venv/lib64/python3.11/site-packages/thrift/transport/TTransport.py", line 397, in open
    raise TTransportException(
thrift.transport.TTransport.TTransportException: Bad SASL negotiation status: 3 (b'GSS initiate failed')

[kevinjqliu]

Hi @dev-sajal thanks for raising this issue. Kerberos support is a bit in the grey area. I have tried to fix a related issue in #1747 but I'm not familiar with kerberos and dont have the necessary infra to test against.

Would love to get additional context from you to resolve this together.

While PySpark could connect to the HMS on the same platform

Could you post what the pyspark configs look like? It might help us debug the issue

Also, is there any guidance from HPE on how to connect to kerberos?

[dev-sajal]

Hi @kevinjqliu, thanks for responding.
The config provided to PySpark are same as I mentioned above without any specifc security mechanism i.e., ignoring these parameters "hive.kerberos-authentication", "hive.metastore.sasl.enabled" and "hive.kerberos-service-name".

No specific guidance as of now, the only possibility we could think of is unavailability of HTTP mechanism inside the HPE environment. The reason for saying so, before we tested this in HPE, the kerberos key used to work and we were able to have a clear communication setup. The HPE team is also looking to check what could be the issue, but thought of checking here, if anyone has already found a fix.

Apologies, I can't provide much details on it as its more for use in organization.

[kevinjqliu]

Thanks @dev-sajal
Heres the code that deals with kerberos and HMS

iceberg-python/pyiceberg/catalog/hive.py

Lines 142 to 196 in b0880c8

	class _HiveClient:
	"""Helper class to nicely open and close the transport."""
	_transport: TTransport
	_ugi: list[str] | None
	def __init__(
	self,
	uri: str,
	ugi: str | None = None,
	kerberos_auth: bool | None = HIVE_KERBEROS_AUTH_DEFAULT,
	kerberos_service_name: str | None = HIVE_KERBEROS_SERVICE_NAME,
	):
	self._uri = uri
	self._kerberos_auth = kerberos_auth
	self._kerberos_service_name = kerberos_service_name
	self._ugi = ugi.split(":") if ugi else None
	self._transport = self._init_thrift_transport()
	def _init_thrift_transport(self) -> TTransport:
	url_parts = urlparse(self._uri)
	socket = TSocket.TSocket(url_parts.hostname, url_parts.port)
	if not self._kerberos_auth:
	return TTransport.TBufferedTransport(socket)
	else:
	return TTransport.TSaslClientTransport(socket, host=url_parts.hostname, service=self._kerberos_service_name)
	def _client(self) -> Client:
	protocol = TBinaryProtocol.TBinaryProtocol(self._transport)
	client = Client(protocol)
	if self._ugi:
	client.set_ugi(*self._ugi)
	return client
	def __enter__(self) -> Client:
	"""Make sure the transport is initialized and open."""
	if not self._transport.isOpen():
	try:
	self._transport.open()
	except (TypeError, TTransport.TTransportException):
	# Close the old transport before reinitializing to prevent resource leaks
	try:
	self._transport.close()
	except Exception:
	pass
	# reinitialize _transport
	self._transport = self._init_thrift_transport()
	self._transport.open()
	return self._client() # recreate the client
	def __exit__(self, exctype: type[BaseException] | None, excinst: BaseException | None, exctb: TracebackType | None) -> None:
	"""Close transport if it was opened."""
	if self._transport.isOpen():
	self._transport.close()

The hive_metastore client implementation is in https://github.com/apache/iceberg-python/tree/main/vendor/hive_metastore

Hope this helps with debugging with HPE. And please let us know if you found a solution 😄

[dev-sajal]

Thanks for the code reference! Will definitely share as I find a fix.