Recomended way to do credentials interpolation

[jkolash]

Feature Request / Improvement

I would like to be able to use the .pyiceberg.yaml but not have hardcoded credentials.

I want to be able to refer to a secrets manager key/value

Currently I can do

pre_interpolated_config = pyiceberg.catalog._ENV_CONFIG.get_catalog_config("iceberg_rest")

# Looks up secrets in a secrets manager based on some pattern such as op://vault/secret/field for 1password.
config = interpolate_configs_and_secrets(pre_interpolated_config) 

pyiceberg.catalog.load_catalog("iceberg_rest", **config)

But that relies on using _ENV_CONFIG which is internal. How can I do this via a supported way? I would like to be able to write my own credentials interpolation function and not have that part of pyiceberg, as that would be specific to each users secrets manager setup such as hashicorp vault, 1password, aws secrets manager, etc..

I see two potential approaches

1. Allow getting the config before load_catalog() in an officially supported way, not _ENV_CONFIG.
2. Officially support interpolating credentials via a custom interpolator that can be registered. It simply remaps each entry in a config map.)

I think approach 1 is the lighter lift.

[jkolash]

For reference this is something basic that I used to get secrets interpolation from 1 password working.

secure_catalog_load.py

import subprocess
from pyiceberg.catalog import load_catalog
from pyiceberg.catalog import _ENV_CONFIG as DEFAULT_CONFIGS

def get_op_secret(name):
   result = subprocess.run(["op", "read", name], capture_output=True, text=True, check=True)
   return result.stdout.strip()

def interpolate_secrets(config):
   def interpolate_secret(key, value):
      if value.startswith("op://"):
        return get_op_secret(value)
      else:
        return value
      
   return { key: (interpolate_secret(key, value) )
     for key, value in config.items() }

def catalog_load(catalog_name):
    raw_configs = DEFAULT_CONFIGS.get_catalog_config(catalog_name)
    configs = interpolate_secrets(raw_configs)
    return load_catalog(catalog_name, **configs)

[Fokko]

@jkolash, first of all, thanks for raising this. I think it should merge the configs. What about:

pyiceberg.catalog.load_catalog("iceberg_rest", credential='abc')

[kevinjqliu]

+1 to what Fokko said. I think load_catalog can be helpful here

iceberg-python/pyiceberg/catalog/__init__.py

Lines 247 to 248 in b63278b

	env = _ENV_CONFIG.get_catalog_config(name)
	conf: RecursiveDict = merge_config(env or {}, cast(RecursiveDict, properties))

load_catalog takes care of reading catalog configs, from .pyiceberg.yaml, ENV vars, and custom properties dict.

I think this would work

secrets = {...}
load_catalog(catalog_name, **{**catalog_configs, **secrets})

[jkolash]

So in my snippet above I do use load_catalog and that is fine. However I still have the problem of where do I store credential references.

My options are

1. Store them in code (Not good)
2. Store them in a separate config file (Ok. This requires setting up a separate config format and resolution from pyicebergs, or re-implement it)
3. Store them in the .pyiceberg.yaml config (Best. The config format/resolution logic is the same as pyiceberg)

In order to go with option 3 I need a way to read the catalog configs without loading the catalog so a new function inside catalog/init.py called read_catalog_configs would work.

def read_catalog_configs(catalog_name):
    return _ENV_CONFIG.get_catalog_config(catalog_name)

I can today just directly access _ENV_CONFIG and go with option 3 but I'd prefer that there be a more proper function to call vs getting a direct reference to a global and calling a funciton on that which may break in the future as it is probably not a supported api call.