GCP Cloud Run/Function only: GoogleSecretManagerSettingsSource tries to read unrelated secrets even if all values are set in .env or env vars

[zaphod72]

I am using the example code at https://docs.pydantic.dev/latest/concepts/pydantic_settings/#basic-usage

---

Note: this does NOT fail if I do not use a nested model. I.e.:

class Settings(BaseSettings):
    password: str
    user: str

    model_config = SettingsConfigDict(env_nested_delimiter='__')

    @classmethod
    def settings_customise_sources(
...

or if I remove the GoogleSecretManagerSettingsSource
or if I run outside of the GCP.

---

When I deploy this into Google Cloud Run or Google Cloud Functions it fails trying to read an unrelated secret.

Repro:

1. Delete database__user and database__password secrets from your GCP.
2. Create some unrelated secret and ensure that the Cloud Run/Function service account does NOT have read permissions on that secret.
3. Create a Cloud Function with Python 3.12
   Set variables on the container:

DATABASE__PASSWORD: db_pwd
DATABASE__USER: db_user

Files:
.env:

DATABASE__USER=dotenv_db_user
DATABASE__PASSWORD=dotenv_db_password

requirements.txt:

functions-framework==3.*
pydantic-settings[gcp-secret-manager]==2.12.*
pydantic

main.py:

import functions_framework
from pydantic import BaseModel
from pydantic_settings import (
    BaseSettings,
    GoogleSecretManagerSettingsSource,
    PydanticBaseSettingsSource,
    SettingsConfigDict,
)

class Database(BaseModel):
    password: str
    user: str


class Settings(BaseSettings):
    database: Database

    model_config = SettingsConfigDict(env_nested_delimiter='__')

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        # Create the GCP Secret Manager settings source
        gcp_settings = GoogleSecretManagerSettingsSource(
            settings_cls,
            # If not provided, will use google.auth.default()
            # to get credentials from the environemnt
            # credentials=your_credentials,
            # If not provided, will use google.auth.default()
            # to get project_id from the environemnt
            project_id='<your-project-id>',   # Yes, I did replace this with my project id
        )

        return (
            init_settings,
            env_settings,
            dotenv_settings,
            file_secret_settings,
            gcp_settings,
        )

@functions_framework.http
def test(request):
    settings = Settings()
    return settings.model_dump_json(indent=2)

4. Deploy your Cloud Function.
5. curl the CF URL.
6. Check the logs for the error:

  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/functions_framework/execution_id.py", line 157, in wrapper
    result = view_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/functions_framework/__init__.py", line 142, in view_func
    return function(request._get_current_object())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/functions_framework/__init__.py", line 121, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/main.py", line 50, in test
    settings = Settings()
               ^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/pydantic_settings/main.py", line 195, in __init__
    **__pydantic_self__._settings_build_values(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/pydantic_settings/main.py", line 438, in _settings_build_values
    source_state = source()
                   ^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/pydantic_settings/sources/base.py", line 510, in __call__
    field_value = self.prepare_field_value(field_name, field, field_value, value_is_complex)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/pydantic_settings/sources/providers/env.py", line 114, in prepare_field_value
    env_val_built = self.explode_env_vars(field_name, field, self.env_vars)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/pydantic_settings/sources/providers/env.py", line 226, in explode_env_vars
    for env_name, env_val in env_vars.items():
                             ^^^^^^^^^^^^^^^^
  File "<frozen _collections_abc>", line 894, in __iter__
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/pydantic_settings/sources/providers/gcp.py", line 78, in __getitem__
    raise KeyError(key)
KeyError: '<one of your secrets>'

I also tried with just the .env and no env vars, and just the env vars and no .env

[zaphod72]

Root Cause Analysis

The crash occurs due to a specific combination of env_nested_delimiter, GCP Permissions/Secret State, and the implementation of GoogleSecretManagerSettingsSource.

1. env_nested_delimiter Triggers Iteration:
   When you set env_nested_delimiter='__', pydantic-settings (specifically explode_env_vars) must iterate over every single key available in the settings source to determine if it maps to a nested field (e.g., checking if unrelated_secret matches database__...).

2. List vs. Access Discrepancy:

   • The GoogleSecretManagerSettingsSource.__iter__ method calls the GCP API to list all secrets in the project.
   • The GoogleSecretManagerSettingsSource.__getitem__ method calls the GCP API to access the latest version of a specific secret.

3. The Failure Chain:

   • My Cloud Run Service Account has the Secret Manager Viewer role on the project. This allows it to see that unrelated_secret exists, so __iter__ yields it.
   • However, the Service Account lacks permission to access the value of unrelated_secret (missing Secret Manager Secret Accessor role), or the secret is disabled or has no versions.
   • When the loop in explode_env_vars reaches unrelated_secret, it tries to retrieve its value to check for nested delimiters.
   • The __getitem__ method attempts to access the secret version. The GCP API returns an error (e.g., PermissionDenied or NotFound).
   • The GoogleSecretManagerSettingsSource catches this API error and re-raises it as a KeyError.
   • Since explode_env_vars does not expect a KeyError when iterating over keys provided by the source itself, the application crashes.

Why it works locally:
My local user credentials has the Owner or Editor role, giving me full access to read the values of all secrets.

Why it works without env_nested_delimiter:
Without the delimiter, Pydantic only looks up the specific fields defined in your model (e.g., DATABASE). It never iterates over the "unrelated" secrets, so it never triggers the permission error.

[hramezani]

Thanks @zaphod72 for reporting the issue.

@ezwiefel Could you take a look when you have time?

[zaphod72]

After my root cause analysis I can repro locally. I just create a secret and remove my GCP user's read permissions on it.

[hramezani]

After my root cause analysis I can repro locally. I just create a secret and remove my GCP user's read permissions on it.

Does it mean we don't need any change on the code?

It would be great if you could create a fix patch with tests(if we need a fix)

[zaphod72]

Does it mean we don't need any change on the code?

No, just means I found out why I was previously not able to repro locally previously.
So this bug will occur if the user or service account running the application does not have permission to read ANY secret from GCS, which is very likely in a secure prod deployment.

I'll work on a fix patch.

[zaphod72]

I have a fix, should I wait for this to be tagged as a bug?

[hramezani]

No, please create a PR.

[zaphod72]

#712