Setting a BaseSettings field repeated times using different aliases in different settings sources results in ValidationError

[sujuka99]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

I have 2 settings sources that refer to the same variable with different aliases and expect that the first one would be prioritized. Instead, the second time a settings source tries to set the variable under the different alias a ValidationError is thrown that says that the variable is extra which it isn't.

Pytest output

__pydantic_self__ = FakeConfig(), _case_sensitive = None, _env_prefix = None, _env_file = PosixPath('.'), _env_file_encoding = None, _env_nested_delimiter = None, _secrets_dir = None
values = {'bar_init': 'anything'}

    def __init__(
        __pydantic_self__,
        _case_sensitive: bool | None = None,
        _env_prefix: str | None = None,
        _env_file: DotenvType | None = ENV_FILE_SENTINEL,
        _env_file_encoding: str | None = None,
        _env_nested_delimiter: str | None = None,
        _secrets_dir: str | Path | None = None,
        **values: Any,
    ) -> None:
        # Uses something other than `self` the first arg to allow "self" as a settable attribute
>       super().__init__(
            **__pydantic_self__._settings_build_values(
                values,
                _case_sensitive=_case_sensitive,
                _env_prefix=_env_prefix,
                _env_file=_env_file,
                _env_file_encoding=_env_file_encoding,
                _env_nested_delimiter=_env_nested_delimiter,
                _secrets_dir=_secrets_dir,
            )
        )
E       pydantic_core._pydantic_core.ValidationError: 1 validation error for FakeConfig
E       bar_env
E         Extra inputs are not permitted [type=extra_forbidden, input_value='env', input_type=str]
E           For further information visit https://errors.pydantic.dev/2.4/v/extra_forbidden

.venv/lib/python3.11/site-packages/pydantic_settings/main.py:71: ValidationError

Example Code

from typing import Any
from typing_extensions import override
from pydantic import AliasChoices, Field
from pydantic.fields import FieldInfo
from pydantic_settings import BaseSettings, PydanticBaseSettingsSource

class FakeEnvSettingsSource(PydanticBaseSettingsSource):

    @override
    def __call__(self):
        d = {}

        return {"bar_env": "env"}
    
    @override
    def get_field_value(self, field: FieldInfo, field_name: str) -> tuple[Any, str, bool]:
        return None, "", False
    
class FakeInitSettingsSource(PydanticBaseSettingsSource):

    @override
    def __call__(self):
        d = {}

        return {"bar_init": "init"}
    
    @override
    def get_field_value(self, field: FieldInfo, field_name: str) -> tuple[Any, str, bool]:
        return None, "", False
    
class FakeConfig(BaseSettings):
    bar: str = Field(validation_alias=AliasChoices("bar_init", "bar_env"))

    @override
    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ):
        return FakeEnvSettingsSource(settings_cls), FakeInitSettingsSource(settings_cls)

def test_alias_choices():
    c = FakeConfig(bar_init="anything")
    assert c == "env"

Python, Pydantic & OS Version

pydantic version: 2.4.2
        pydantic-core version: 2.10.1
          pydantic-core build: profile=release pgo=true
                 install path: /home/sujuka99/path/to/project/.venv/lib/python3.11/site-packages/pydantic
               python version: 3.11.5 (main, Sep  2 2023, 14:16:33) [GCC 13.2.1 20230801]
                     platform: Linux-6.4.12-arch1-1-x86_64-with-glibc2.38
             related packages: pydantic-settings-2.0.3 pyright-1.1.314 typing_extensions-4.7.1

[sydney-runkle]

Hi @sujuka99,

Thanks for reporting this 😄. The reasoning behind this is a bit complicated. So pydantic-settings parses your various environment settings sources correctly, and effectively generates the following input to your FakeConfig class:

{'bar_env': 'env', 'bar_init': 'init'}

However, the difficulty here is that then the following is occurring:

from pydantic import AliasChoices, BaseModel, ConfigDict, Field


class FakeConfig(BaseModel):
    bar: str = Field(validation_alias=AliasChoices('bar_env', 'bar_init'))

    model_config = ConfigDict(extra='forbid')


fc = FakeConfig.model_validate({'bar_env': 'env', 'bar_init': 'init'})

However, the nature of BaseSettings models is that they have the setting extra='forbid' set by default (shown explicitly above). So what's really happening is that bar is set to env, but then bar_init is recognized as an extra field. One option would be to set extra='allow' on your FakeConfig model, but then you'd get an extra bar_init field set on your model.

Hopefully that explanation of the behavior makes some more sense. We're set up to prioritize fields specified in more highly prioritized sources, but not in the case of different aliases.

If this is behavior that you'd like to see added, I suggest we move this to a feature request on the pydantic-settings repo 😄.

[sujuka99]

Hey @sydney-runkle, thanks for the thorough explanation! I would certainly like to see the behavior added 😄, should I move the request?

Ultimately I would want to replicate what Field(..., env=...) did in V1. Is there a way to achieve it in V2 currently?

[sydney-runkle]

@sujuka99,

Thanks for the prompt response. After thinking about this more, I do think this is actually a bug. We guarantee that if an environment variable is specified in 2 locations (without an alias), we do prioritize and select appropriately, so I don't think the case should be any different when aliases are used.

I'll move this issue to pydantic-settings and we can work on a fix there 😄.

At the moment, you could set extra='allow' within the model_config on your FakeConfig class, but it's an unclean workaround for the time being.

[hramezani]

similar to #148

[D1maD1m0nd]

Hello, please tell me what I'm doing wrong, this code worked before update 2.1.0 -> 2.2.1

from pydantic_settings import BaseSettings, SettingsConfigDict


class Base(BaseSettings):
    model_config = SettingsConfigDict(env_file='.env', env_file_encoding='utf-8')


class Settings(Base):
    model_config = SettingsConfigDict(env_prefix='APP_')
    ENVIRONMENT: str = ""
    REDIS_ADDRESS: str = ""
    MEANING_OF_LIFE: int = 0
    MY_VAR: str = ""


class ServerSettings(Base):
    model_config = SettingsConfigDict(env_prefix='SERV_')
    ADDR: str = ""


class GlobalSettings:
    app_settings = Settings()
    srv_settings = ServerSettings()


g_settings = GlobalSettings()

print(g_settings.app_settings.model_dump())
print(g_settings.srv_settings.model_dump())

.env

APP_ENVIRONMENT="production"
APP_REDIS_ADDRESS=localhost:6379
APP_MEANING_OF_LIFE=42
APP_MY_VAR='Hello world'


SERV_ADDR='localhost'

[hramezani]

@D1maD1m0nd probably related to #245

If you get ValidationError complaining about extra fields, you can fix it by adding extra='ignore'