Feature: Add lazy load support in GCP

[fede-bello]

Lazy Loading Support for Pydantic Settings Sources

Summary

This PR implements lazy loading for settings sources, deferring field value resolution until fields are accessed rather than fetching all values during initialization. This enables significant performance improvements for expensive operations such as API calls to cloud secret managers.

Solves #713

What Changed

• Added a new lazy_load parameter to GCP Secret Manager settings source (GCPSecretManagerSettingsSource) to opt into lazy loading.
• Implemented lazy loading support in the base class, ensuring all providers can benefit with minimal changes.
• Introduced the internal LazyMapping mechanism for deferring and caching per-field lookups.
• Updated tests to cover lazy loading behavior and environment-based sources.
  • Performed integration testing specifically for the GCP provider.

Problem

Currently, all settings sources eagerly fetch values for every field during Settings instantiation, even if those fields are never accessed. This is problematic for expensive operations:

• API calls to cloud secret managers (GCP Secret Manager, AWS Secrets Manager, Azure Key Vault)
• Large file reads from secrets directories
• Network roundtrips that could be avoided

Solution: LazyMapping

The implementation introduces a LazyMapping class, a dict-like mapping that:

• Defers field value resolution until keys are accessed via __getitem__()
• Caches computed values to avoid redundant operations
• Implements the Mapping ABC for compatibility with Pydantic's initialization

When lazy_load=True:

• Settings sources return an empty dict from __call__()
• A LazyMapping is stored on source._lazy_mapping
• Field values are only fetched when explicitly accessed

Test Coverage

• unit tests covering LazyMapping behavior and GCP Secret Manager

Note: Integration tests were performed for GCP Secret Manager. The fix is implemented at the PydanticBaseEnvSettingsSource class level, so all inheriting providers automatically support lazy loading. The parameter was only added for GCP Secret Manager, but extending it to new providers should be as simple as adding the parameter.

Why LazyMapping

Backward Compatibility

lazy_load defaults to False, preserving eager loading behavior.

Alternative Approaches Considered

I don’t think this is the most intuitive implementation, and I initially wanted something simpler. However, I've discussed some other options and nothing convinced me:

1. Lazy attribute access on Settings (__getattr__)

   • Idea: Fetch values only when you access them (e.g., settings.db_password)
   • Problem: Requires hacky code that intercepts all field access. Your IDE won't know what fields exist, autocomplete breaks, and it breaks every time Pydantic updates.

2. Separate LazySettings class

   • Idea: Have two different Settings classes—one eager, one lazy. Pick which to use.
   • Problem: Users have to decide at import time. Can't mix lazy and eager sources together.

3. Property-based field access

   • Idea: Turn each Settings field into a function/property that fetches on demand
   • Problem: Users would have to change how they define every single field in their code. Your IDE won't understand the types anymore.

4. Async initialization

   • Idea: Use async def __init__() to fetch values asynchronously
   • Problem: Would break existing code massively. Every Settings instantiation would need await. Too invasive.

[hramezani]

Thanks @fede-bello for the PR.

I think we only want it for GoogleSecretManagerSettingsSource.

I think it doesn't make sense to have lazy loading for the env source or dotenv.

People usually initialize settings on application startup and they usually do it once.

[fede-bello]

Do we want it for other cloud providers? AWS or Azure? Or just GCP that it's what I was able to test?

[hramezani]

Do we want it for other cloud providers? AWS or Azure? Or just GCP that it's what I was able to test?

Let's do it for GCP now because you can test it and probably maintain it later.

[hramezani]

Two questions:

1. What happens if other sources' values have more priority than GCP settings source?
2. What happens if the value provided by a source is not a valid value?

[fede-bello]

1. Not sure in what sense you mean. If I understand your question correctly, higher priority sources shadow lower ones. So if theres a key in a higher priority source, that one will be loaded and the one for GCP won't be consulted
2. Trying to access will return None. It will enter the get_field_value method in EnvSettingsSource, the field_value will not be found and will return None

PS: left a fix with an issue with the model dump that wasn't loading the lazy fields

[hramezani]

Trying to access will return None. It will enter the get_field_value method in EnvSettingsSource, the field_value will not be found and will return None

I mean if the value is not a valid value for the field. like you defined an int field but the value is string. or you put some limitation on the string length.

[hramezani]

should we keep this here? we agreed to enable lazy loading for GCP secret source

[fede-bello]

I left it there so eventually is easier to implement for other sources providers and it's not a only-gcp fix. It the parameter is not passed to the base class the logic shouldn't change

[hramezani]

@fede-bello I think this is going to be complicated. Like we need to add and maintain a lot of code, and the lazy loading values from GCP is not important IMHO.
pydantic-settings model usually will be initialized once at startup and will be part of the bootstrap process, which generally takes time.

As pydantic-settings lets you have your own custom settings source, and GCP secret source is not one of our most used source, it would probably be good not to add lazy loading to the package.

What do you think?

[fede-bello]

@hramezani

I don't know if there's a more straightforward way to implement the feature

The thing is that we live far from our gcp instances are, and each secret fetch can take up to 2-3 seconds. It's not the worst but it can get annoying when having a lot of secrets and performing short tests. A 20 second start up just to try a path that doesn't use a secret

Maybe we can add a wrapper to the gcp ourselves and not even have to modify the library, but seems like a issue that more people might have in the future

[sebastian-correa]

Hey @hramezani. Just some context, I'm working with @fede-bello on a feature that needs a solution to the issue he described in his last comment; that's why you see some comments from me from time to time :).

I agree this functionality doesn't super align with Pydantic's ethos of validating data and erroring out. It's also been an issue with FastAPI apis we've had to produce where we have some list[Model] attribute and users want to send some erroneous data that we must report on, but process the correct ones anyways. In that sense, we'd be totally cool if you didn't want to integrate this into the package.

Implementing lazy loading for this cloud provider is a bit of a slippery slope as well. Adding one lazy loader may open up to implementing all sources as lazy, and thay will probably bring issues in the future, since Pydantic is designed with front-loading and validation in mind.

However, I'd be surprised if we're the only ones facing this limitation. Aside from the latency Fede described (imagine having a CLI and having to wait many seconds to see the --help), here are some reasons why not having lazy loading may be an issue:

• Monorepos. Sometimes, people have a single Python package where some parts need access to some secret and some others don't. In principle, when running locally this isn't an issue since we usually have either a Service Account or a personal account with access to everything. However, when deploying to production, each of those subcomponents may run in a separate container, with a purpose-built SA that has limited permissions. This may cause issues if we try to front-load all secrets, since that specific service only needs (and has access to) a subset of the secrets.
• We'd be incurring costs for short tests, as Fede mentioned. Probably not substancial enough to warrant the feature, but something to consider.

Aside from the caveats I mentioned, I don't see why doing this #713 (comment) wouldn't work for us. Maybe this is just a matter of documenting this "hack" and its caveats, instead of altering all the core and increasing the maintenance burden on you guys, which isn't ideal.

In essence what we're looking for is a post-process hook that runs just before returning the retrieved attribute and has access to the instance. That's why my suggestion was an override to the __getattribute__, but a hook can be systematized by Pydantic I would imagine.

[hramezani]

Hey @sebastian-correa. Thanks for your explanation.

I agree this functionality doesn't super align with Pydantic's ethos of validating data and erroring out. It's also been an issue with FastAPI apis we've had to produce where we have some list[Model] attribute and users want to send some erroneous data that we must report on, but process the correct ones anyways. In that sense, we'd be totally cool if you didn't want to integrate this into the package.

Yeah, the whole point of pydantic-settings is collecting values from different sources and validating them by Pydantic based on defined fields. But in the case of lazy loading, we need to skip the validation.
#718 (comment)

Monorepos. Sometimes, people have a single Python package where some parts need access to some secret and some others don't. In principle, when running locally this isn't an issue since we usually have either a Service Account or a personal account with access to everything. However, when deploying to production, each of those subcomponents may run in a separate container, with a purpose-built SA that has limited permissions. This may cause issues if we try to front-load all secrets, since that specific service only needs (and has access to) a subset of the secrets.

I was working on a code base like the thing that you described. at the end we solved the problem by splitting our settings and not loading unrelated values in containers. each container only receives the needed values. then only containers that needs values from e.g. GCP secret have to wait for it.

Aside from the caveats I mentioned, I don't see why doing this #713 (comment) wouldn't work for us. Maybe this is just a matter of documenting this "hack" and its caveats, instead of altering all the core and increasing the maintenance burden on you guys, which isn't ideal.

Aside from the complexity and future maintenance problems, the fact that we can't validate the value provided in GCP secret(I mentioned it in the first paragraph) is important for me because having a lazy gcp settings source means ignoring validation for gcp secret source.

Probably not related, take a look at pydantic validator doc. pre field and model validator might be helpful for you

[sebastian-correa]

@hramezani yes, setting (secret) splitting can solve the issue, I agree. In fact, our settings are split in reality (not a single file).

To clarify: I didn't mean for my #713 (comment) to be incorporated into the codebase. I was looking for validation of the code from someone who probably knows the internals better than we do, to see if the implementation makes sense. If it does, we'd be providing this internally to users without any changes to Pydantic or pydantic-settings. In our codebase that uses dynaconf we do this:

def _setting_is_cloud_secret(name: str, value: str) -> bool:  # noqa: ARG001 needed to comply with interface.
    """Check if the given setting (as name and value) is special.

    Currently, a setting is special if its value starts with "__cloud_secret:".

    Args:
        name (str): Setting name.
        value (str): Setting value.

    Returns:
        bool: `True` if special.
    """
    return value.startswith("__cloud_secret:")


def _maybe_return_from_gcp_and_update(getter: Callable) -> Callable:
    """If the value got by `getter` `_setting_is_cloud_secret`, replace it from from GCP.

    This should decorate methos of classes, as the returned funtion expects `self` as the first arg.

    Args:
        getter (Callable): The getter, used to find the current value.

    Returns:
        Callable: A callable, to replace the previous one, that performs the
            `_setting_is_cloud_secret` on the name and current value of the setting, and fetches
            the setting from GCP if it is special.
    """

    # This is intended as a decorator of `Dynabox` and `BoxList`, so the first arg must be `self`.
    @wraps(getter)
    def check_and_replace(self, item, *args, **kwargs):
        current_value = getter(self, item, *args, **kwargs)
        if isinstance(current_value, str) and _setting_is_cloud_secret(item, current_value):
            _, name, version = current_value.split(":")
            new_value = _get_secret_from_gcp(name, version)
            self[item] = new_value
            return new_value
        return current_value

    return check_and_replace

# Monkey patch Dynaconf's container classes to include our custom post-processing.
dynaconf.utils.boxing.DynaBox.__getattr__ = _maybe_return_from_gcp_and_update(
    dynaconf.utils.boxing.DynaBox.__getattr__
)

dynaconf.utils.boxing.DynaBox.__getitem__ = _maybe_return_from_gcp_and_update(
    dynaconf.utils.boxing.DynaBox.__getitem__
)
dynaconf.vendor.box.box_list.BoxList.__getitem__ = _maybe_return_from_gcp_and_update(
    dynaconf.vendor.box.box_list.BoxList.__getitem__
)

dynaconf.utils.boxing.DynaBox.get = _maybe_return_from_gcp_and_update(
    dynaconf.utils.boxing.DynaBox.get
)

The bit about hooks was a suggestion. It seems like Pydantic could provide some mechanism to hook into access, with validation incorporated. But, again, this would probably go against Pydantic's ethos. It would also add some overhead to access which may be undesirable.

Model and field validators hook into the validation process, not the access process. Unfortunately, those wouldn't help us solve the issue.