error parsing value for field <...> from source "EnvSettingsSource"

[fredrikaverpil]

Hi 👋

I just saw the FastAPI 0.100.0 prerelease announcement, the Pydantic v2 migration guide and wanted to give this all a try with one of my project's test suites with the latest and greatest.

I ended up getting this error:

self = EnvSettingsSource(env_nested_delimiter=None, env_prefix_len=0)

    def __call__(self) -> dict[str, Any]:
        data: dict[str, Any] = {}
    
        for field_name, field in self.settings_cls.model_fields.items():
            try:
                field_value, field_key, value_is_complex = self.get_field_value(field, field_name)
            except Exception as e:
                raise SettingsError(
                    f'error getting value for field "{field_name}" from source "{self.__class__.__name__}"'
                ) from e
    
            try:
                field_value = self.prepare_field_value(field_name, field, field_value, value_is_complex)
            except ValueError as e:
>               raise SettingsError(
                    f'error parsing value for field "{field_name}" from source "{self.__class__.__name__}"'
                ) from e
E               pydantic_settings.sources.SettingsError: error parsing value for field "ui_languages" from source "EnvSettingsSource"

And this is my code that triggered the error when calling LanguageSettings().ui_languages:

from pathlib import Path
from pydantic import Field
from pydantic_settings import BaseSettings


def comma_separated_string_to_set(raw_value: str) -> set[str]:
    if not raw_value:
        msg = f"{raw_value} must be a comma separated string"
        raise ValueError(msg)
    return {item.strip() for item in raw_value.split(",")}


class LanguageSettings(BaseSettings):
    ui_languages: set[str] = Field(env="UI_LANGUAGES")

    class Config:
        env_file = Path("envs/test.env")

        @classmethod
        def parse_env_var(cls, field_name: str, raw_val: str) -> Any:
            if field_name == "ui_languages":
                return comma_separated_string_to_set(raw_val)

Here's my poetry config:

[tool.poetry.dependencies]
fastapi = {extras = ["all"], version = "^0.100.0b1", allow-prereleases = true}
pydantic = {version = "^2.0b3", allow-prereleases = true}
pydantic-settings = {version = "^2.0b1", allow-prereleases = true}

Is all this expected when moving from old/current Pydantic, or is there a bug in pydantic-settings?

[hramezani]

Hi @fredrikaverpil 👋

Thanks for reporting this.

Could you please provide the envs/test.env env file content?

Is it working in Pydantic V1?

[fredrikaverpil]

Could you please provide the envs/test.env env file content?

So unfortunately I cannot share the actual file contents, as this is a closed source proprietary project at work. But I can try to obfuscate as much as possible from the file if you can give me any hints on what it might be you are looking for?

The line in this env file, which talks about UI_LANGUAGES, looks like this:

UI_LANGUAGES=sv,en

Is it working in Pydantic V1?

Yes. The code I provided above has worked fine for a long time in production which we have currently sitting at Pydantic 1.10.8.

[fredrikaverpil]

@hramezani do you think there's a problem that we are mixing pydantic_settings.BaseSettings and pydantic.Field (since they are not from the same package)?

[hramezani]

In pydantic-settings, we have settings sources as separate classes. So, you can have your own settings source class by inheriting from one of the existing source classes and adding your custom implementation.

Here is your custom comma-separated parser in pydantic-settings

from typing import Any, Tuple, Type, Set

from pydantic import Field
from pydantic.fields import FieldInfo
from pydantic_settings import (
    BaseSettings,
    DotEnvSettingsSource,
    PydanticBaseSettingsSource,
)


class CommaSepratedSettingsSource(DotEnvSettingsSource):

    def prepare_field_value(
        self, field_name: str, field: FieldInfo, value: Any, value_is_complex: bool
    ) -> Any:
        if not value:
            msg = f"{value} must be a comma separated string"
            raise ValueError(msg)
        return {item.strip() for item in value.split(",")}


class LanguageSettings(BaseSettings):
    ui_languages: Set[str] = Field(validation_alias="UI_LANGUAGES")


    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: Type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> Tuple[PydanticBaseSettingsSource, ...]:
        return (
            init_settings,
            env_settings,
            CommaSepratedSettingsSource(settings_cls, env_file='test.env', env_file_encoding=None),
            file_secret_settings,
        )



print(LanguageSettings())
#> ui_languages={'sv', 'en'}

Here is the doc about customise settings sources

[fredrikaverpil]

Oh, okay, hmmm 🤔

The parse_env_var classmethod is no longer available at all then?
So I won't be able to use multiple parsers anymore, like in the (somewhat contrived) example below?

class LanguageSettings(BaseSettings):
    ui_languages_as_set: set[str] = Field(env="UI_LANGUAGES")
    ui_languages_as_list: list[str] = Field(env="UI_LANGUAGES")

    class Config:
        env_file = Path("envs/test.env")

        @classmethod
        def parse_env_var(cls, field_name: str, raw_val: str) -> Any:
            if field_name == "ui_languages_as_set":
                return comma_separated_string_to_set(raw_val)  # <--- to set
            if field_name == "ui_languages_as_list":
                return comma_separated_string_to_list(raw_val)  # <--- to list

And I actually used to do yet one more thing in this LanguageSettings class which also doesn't seem to work anymore, which was to define a default factory:

def is_in_features(raw_value: str) -> bool:
    if raw_value in FeatureSettings().features:  # type:ignore
        return True
    return False

def comma_separated_string_to_set(raw_value: str) -> set[str]:
    if not raw_value:
        msg = f"{raw_value} must be a comma separated string"
        raise ValueError(msg)
    return {item.strip() for item in raw_value.split(",")}


class FeatureSettings(BaseSettings):
    features: set[str] = Field(env="FEATURES")

    class Config:
        env_file = Path("envs/test.env")

        @classmethod
        def parse_env_var(cls, field_name: str, raw_val: str) -> Any:
            if field_name == "features":
                return comma_separated_string_to_set(raw_val)


class LanguageSettings(BaseSettings):
    feature_x_enabled: bool = Field(
        default_factory=lambda: is_in_features("feature_x")
    )

Is all this correct, that I can no longer "collect" all my language settings (that uses different parsers and default factories) underneath the same class LanguageSettings?

[hramezani]

The parse_env_var classmethod is no longer available at all then?

No. it is not available.

So I won't be able to use multiple parsers anymore, like in the (somewhat contrived) example below?

You are able to use it. you can handle different parsers in prepare_field_value:

class CommaSepratedSettingsSource(DotEnvSettingsSource):

    def prepare_field_value(
        self, field_name: str, field: FieldInfo, value: Any, value_is_complex: bool
    ) -> Any:
            if field_name == "ui_languages_as_set":
                return comma_separated_string_to_set(value)  # <--- to set
            if field_name == "ui_languages_as_list":
                return comma_separated_string_to_list(value)  # <--- to list

And I actually used to do yet one more thing in this LanguageSettings class which also doesn't seem to work anymore, which was to define a default factory:

default_factory still working as before

[fredrikaverpil]

Thank you for that, I came up with this configuration:

from typing import Any, Set, Tuple, Type

from pydantic import Field
from pydantic.fields import FieldInfo
from pydantic_settings import (
    BaseSettings,
    DotEnvSettingsSource,
    PydanticBaseSettingsSource,
)
from pydantic_settings.main import SettingsConfigDict

from conversations.helpers.dotenv import detect_dotenv_file


class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file="envs/test.env",
        env_file_encoding="utf-8",
        extra="ignore",
    )

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: Type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> Tuple[PydanticBaseSettingsSource, ...]:
        return (
            init_settings,
            env_settings,
            CustomSettingsSource(
                settings_cls,
                env_file="envs/test.env",
                env_file_encoding=None,
            ),
            file_secret_settings,
        )


class CustomSettingsSource(DotEnvSettingsSource):
    def prepare_field_value(
        self,
        field_name: str,
        field: FieldInfo,
        value: Any,
        value_is_complex: bool,
    ) -> Any:
        if field_name in ["features", "ui_languages"]:
            if not value:
                msg = f"{value} must be a comma separated string"
                raise ValueError(msg)
            return {item.strip() for item in value.split(",")}

        return value


class FeatureSettings(Settings):
    features: set[str] = Field("FEATURES")


class LanguageSettings(Settings):
    ui_languages: set[str]
    feature_x_enabled: bool = Field(
        default_factory=lambda: is_in_features("feature_x")
    )


def is_in_features(raw_value: str) -> bool:
    features = FeatureSettings().features
    if raw_value in features:
        return True
    return False

This seems to work now for me. What I've set out to achieve is to load any non-existing variable from the env file, and then supply custom value parsers via the CustomSettingsSource class.

I'm noticing I have to pass in the env file in two places. Does this setup look reasonable to you?

[hramezani]

I'm noticing I have to pass in the env file in two places. Does this setup look reasonable to you?

I think it's a problem in pydantic-settings and I will prepare a patch for fixing.

[fredrikaverpil]

Ok, awesome, thank you!

One more question: I used to have a few environment variables that might not exist in either the environment or in the .env file, like this:

# given the Settings class posted in the above comment: https://github.com/pydantic/pydantic-settings/issues/80#issuecomment-1602324766

class MySettings(Settings):
    might_not_exist: str | None = Field("MIGHT_NOT_EXIST")

This does not seem to work anymore and I can't find anything in the docs on this.

Is it possible to have MySettings().might_not_exist set to None if it doesn't exist in neither environment or .env file?

[hramezani]

This should work. I haven't checked your settings classes in detail. but as an example:

from typing import Optional

from pydantic_settings import BaseSettings

from pydantic import Field


class MySettings(BaseSettings):
    might_not_exist: Optional[str] = Field("MIGHT_NOT_EXIST")


print(MySettings())  # might_not_exist='MIGHT_NOT_EXIST'

[fredrikaverpil]

Now I'm a bit confused. Shouldn't MySettings().might_not_exist in your example print out None if the environment variable does not exist?

When I run your example, without having that variable registered in my environment I get this:

>>> MySettings().might_not_exist
'MIGHT_NOT_EXIST'

[hramezani]

@fredrikaverpil
I've created a PR to fix your mentioned problem

I'm noticing I have to pass in the env file in two places. Does this setup look reasonable to you?

[hramezani]

Field("MIGHT_NOT_EXIST")

The first param in Field is the default value. So, Field("MIGHT_NOT_EXIST") set the default value for might_not_exist to MIGHT_NOT_EXIST

[fredrikaverpil]

The first param in Field is the default value.

🤦 oh yes of course!

Thanks for putting together that PR!