Issue with Iceberg metadata ingestion configuration

[divyagowdab]

From ui go to Database Services -> Add new service -> Select Iceberg and click next to provide connection details.
Choose "RestCatalogConnection" and provide details. I have used polaris rest catalog and provided the polaris uri, clientid and clientsecret. When i test the connection it fails. I also tried saving it and then triggering pipeline then also it fails with below error.
Provided screenshot of the request which has the credentials set properly.

Note: It works if "token" is provided. But the token is short lived and it does not refresh automatically after it expires

[2024-09-11T06:26:22.676+0000] {server_mixin.py:75} INFO - OpenMetadata client running with Server version [1.4.4] and Client version [1.4.4.1]
[2024-09-11T06:26:22.856+0000] {taskinstance.py:1937} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/pyiceberg/catalog/rest.py", line 285, in _fetch_config
response.raise_for_status()
File "/home/airflow/.local/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://polaris.polaris.svc.cluster.local:8181/api/catalog/v1/config?warehouse=polaris10k

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/python.py", line 192, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/operators/python.py", line 209, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/home/airflow/.local/lib/python3.10/site-packages/openmetadata_managed_apis/workflows/ingestion/common.py", line 207, in metadata_ingestion_workflow
workflow = MetadataWorkflow.create(config)
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/workflow/ingestion.py", line 103, in create
return cls(config)
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/workflow/ingestion.py", line 79, in init
super().init(
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/workflow/base.py", line 107, in init
self.post_init()
return cls(config)
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/workflow/ingestion.py", line 79, in init
super().init(
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/workflow/base.py", line 107, in init
self.post_init()
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/workflow/ingestion.py", line 98, in post_init
self.set_steps()
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/workflow/metadata.py", line 35, in set_steps
self.source = self._get_source()
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/workflow/metadata.py", line 66, in _get_source
source: Source = source_class.create(
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/ingestion/source/database/iceberg/metadata.py", line 94, in create
return cls(config, metadata)
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/ingestion/source/database/iceberg/metadata.py", line 79, in init
self.iceberg = get_connection(self.service_connection)
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/ingestion/source/connections.py", line 49, in get_connection
return get_connection_fn(connection)(connection)
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/ingestion/source/database/iceberg/connection.py", line 35, in get_connection
return IcebergCatalogFactory.from_connection(
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/ingestion/source/database/iceberg/catalog/init.py", line 65, in from_connection
return catalog_type.get_catalog(catalog)
File "/home/airflow/.local/lib/python3.10/site-packages/metadata/ingestion/source/database/iceberg/catalog/rest.py", line 84, in get_catalog
return load_rest(catalog.name, parameters)
File "/home/airflow/.local/lib/python3.10/site-packages/pyiceberg/catalog/init.py", line 88, in load_rest
return RestCatalog(name, **conf)
File "/home/airflow/.local/lib/python3.10/site-packages/pyiceberg/catalog/rest.py", line 199, in init
self._fetch_config()
File "/home/airflow/.local/lib/python3.10/site-packages/pyiceberg/catalog/rest.py", line 287, in _fetch_config
self._handle_non_200_response(exc, {})
File "/home/airflow/.local/lib/python3.10/site-packages/pyiceberg/catalog/rest.py", line 356, in _handle_non_200_response
raise exception(response) from exc
pyiceberg.exceptions.UnauthorizedError: RESTError 401: Could not decode json payload: Credentials are required to access this resource.
[2024-09-11T06:26:22.861+0000] {taskinstance.py:1400} INFO - Marking task as FAILED.

[divyagowdab]

The load_rest func call from pyiceberg library is throwing an error in this file https://github.com/open-metadata/OpenMetadata/blob/b14288284860bad04da5d4220f3e4307590b9f8f/ingestion/src/metadata/ingestion/source/database/iceberg/catalog/rest.py
I have tried that func call locally and it works. sample code snippet
`
from pyiceberg.catalog import load_rest

catalog_conf = {
"type": "rest",
"uri": "http://localhost:8184/api/catalog",
"credential": "125c4546e024e8ea:140e4bf12b103645e9a19939d5a5d76e",
"scope": "PRINCIPAL_ROLE:ALL",
"warehouse": "polaris10k"
}

catalog = load_rest('polaris10k', catalog_conf)
`

In addition to clientid and clientsecret, "scope" is also needed to be set. There is no option in UI to provide scope value. In OM code it uses "catalog" as the default scope if not provided. That scope does not work