Feature request: allow intercepting or lazily blocking secret fetching in `GoogleSecretManagerSettingsSource`

[fede-bello]

I'm using GoogleSecretManagerSettingsSource, and I would like to lazily load some secrets so they are only fetched when actually accessed. The current implementation fetches every single secret defined in the Settings class from Google Secret Manager during initialization, regardless of whether those secrets will actually be used.

This eager loading is the expected behavior, but it causes some isues:

• Performance penalty: Each secret fetch is a network call to GCP
• Unnecessary costs: GCP charges per secret access, so fetching unused secrets wastes money
• No conditional loading: Can't skip secrets based on environment (dev vs prod) or feature flags

Ideally, the solution would allow the following:

• Avoid loading all GCP secrets when instantiating a settings object
• Intercept secret access before calling the GCP API
• Implement conditional/lazy loading in applications that may not need all settings

For example, in my application some modules use only a subset of settings. Currently, importing the settings class triggers GCP secret access even if the code path will not require those secrets.

[fede-bello]

I believe making a Lazy Proxy interface might be the best approach. Something like this (doesn't work like this, but I think it might be useful):

from functools import cached_property
from typing import Any, Optional

class LazySecret:
    """Proxy that fetches secret on first access"""
    def __init__(self, secret_name: str, client, project_id: str):
        self._secret_name = secret_name
        self._client = client
        self._project_id = project_id
        self._value = None
        self._fetched = False
    
    def get_value(self):
        """Fetch secret value when actually needed"""
        if not self._fetched:
            try:
                path = self._client.secret_version_path(
                    self._project_id, self._secret_name, "latest"
                )
                response = self._client.access_secret_version(name=path)
                self._value = response.payload.data.decode('UTF-8')
            except Exception:
                self._value = None
            self._fetched = True
        return self._value
    
    def __str__(self):
        return self.get_value() or ""
    
    def __repr__(self):
        return f"LazySecret({self._secret_name})"


class LazyGoogleSecretManagerSource(GoogleSecretManagerSettingsSource):
    def get_field_value(self, field: FieldInfo, field_name: str) -> tuple[Any, str, bool]:
        # Check if field should be lazy
        if field.json_schema_extra and field.json_schema_extra.get("lazy"):
            # Return a lazy proxy instead of fetching immediately
            secret_name = self._get_secret_name(field_name)
            lazy_proxy = LazySecret(secret_name, self._secret_client, self._project_id)
            return lazy_proxy, field_name, False
        
        # Non-lazy fields fetch immediately (existing behavior)
        return super().get_field_value(field, field_name)


# Usage
class Settings(BaseSettings):
    app_name: str  # Fetched immediately
    
    # This returns a LazySecret proxy
    analytics_token: Annotated[str, Field(json_schema_extra={"lazy": True})]
    
    @property
    def get_analytics_token(self) -> str:
        """Access the actual value when needed"""
        if isinstance(self.analytics_token, LazySecret):
            return self.analytics_token.get_value()
        return self.analytics_token

[hramezani]

Thanks @fede-bello for this feature.

I am not very familiar with GoogleSecretManagerSettingsSource TBH. but your suggestion seems reasonable.

We can probably add a feature flag to the settings source and use lazy loading instead, without breaking the existing implementation.
This needs a PR with proper tests and documentation. feel free to open a PR if you are interested.

[sebastian-correa]

@hramezani I obviously don't have the whole context and don't know about Pydantic's innards too well, but if users are willing to have some "custom indicator", what would you think about an override to __getattribute__?

from pydantic import BaseModel

class SecretFetchingBaseModel(BaseModel):
    x: str
    y: int
    z: float
    def __getattribute__(self, name: str):
        # 1. Retrieve the actual value using standard behavior
        val = super().__getattribute__(name)

        # 2. Safety Check: We only want to intercept Pydantic fields.
        try:
            fields = super().__getattribute__("model_fields")
        except AttributeError:
            return val

        # 3. Apply logic ONLY if the attribute is a declared field
        if name not in fields:
            return val
        if isinstance(val, str) and val.startswith("__gcp_secret"):
            # Expect `__gcp_secret:NAME:VERSION` strings.
            _, secret_name, version = val.split(":")
            # Go fetch the secret!
            setattr(self, name, "secret_value")
            return "secret_value"
        return val

A bit crude, but we've been doing something similar with another library, and it's been rock solid. For Pydantic we'd have to use that new model on each submodel, since outer models holding complex structures wouldn't know to go fetch secrets of inner models (unless we made a recursive step if isinstance(val, BaseModel, SecretFetchingBaseModel, dict) or something like that).

The main issue is that Pydantic is a validation library, so what we're trying to do here is a bit out of scope, but seems like a reasonable problem to have. If we were to add lazy loading via Settings Sources, would you defer validation on values loaded from those sources until they're lazy-loaded? Secrets in particular don't really need much validation, but this feature kinda opens up to lazy-loading any source (maybe desriable or not, not sure!).