Using annotated BeforeValidator may cause JSONDecodeError from .env

[ttti07]

# in .env file
MYAPP_TOKEN=Bearer abcdefg123456789

def parse_token(token: str) -> HTTPAuthorizationCredentials:
    scheme, credentials = token.split(" ")
    return HTTPAuthorizationCredentials(scheme=scheme, credentials=credentials)


HTTPAuthToken = Annotated[HTTPAuthorizationCredentials, BeforeValidator(parse_token)]


class MyappSettings(BaseSettings):
    token: HTTPAuthToken

    model_config = SettingsConfigDict(env_file=".env", env_prefix="MYAPP_")

Becausetoken field uses BeforeValidator,
its actual value type in .env file should be str (i.e. param type of parse_token)
but inside EnvSettingsSource.prepare_field_value it just concludes that it is a complex type.
I think it would be better to consider the input type of the validation function, especially in case of the validator is BeforeValidator.

Selected Assignee: @hramezani

[hramezani]

Thanks @ttti07 for reporting this.

Yes, pydantic-settings considers HTTPAuthToken as a complex field.

You can have your desired behavior by a custom settings source that doesn't parse value to json:

from typing import Any

from fastapi.security import HTTPAuthorizationCredentials
from pydantic import BeforeValidator
from pydantic.fields import FieldInfo
from typing_extensions import Annotated

from pydantic_settings import BaseSettings, DotEnvSettingsSource, PydanticBaseSettingsSource, SettingsConfigDict


class MyDotEnvSettingsSource(DotEnvSettingsSource):
    def prepare_field_value(self, field_name: str, field: FieldInfo, value: Any, value_is_complex: bool) -> Any:
        return value


def parse_token(token: str) -> HTTPAuthorizationCredentials:
    scheme, credentials = token.split(" ")
    return HTTPAuthorizationCredentials(scheme=scheme, credentials=credentials)


HTTPAuthToken = Annotated[HTTPAuthorizationCredentials, BeforeValidator(parse_token)]


class MyappSettings(BaseSettings):
    token: HTTPAuthToken

    model_config = SettingsConfigDict(env_file=".env", env_prefix="MYAPP_")

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        return init_settings, env_settings, MyDotEnvSettingsSource(settings_cls), file_secret_settings

print(MyappSettings())

Or, you can change the env variable value in .env to a valid json like:
MYAPP_TOKEN=["Bearer abcdefg123456789"]
and change parse_token to load from a list like:

from fastapi.security import HTTPAuthorizationCredentials
from typing_extensions import Annotated
from pydantic import BeforeValidator
from pydantic_settings import BaseSettings, SettingsConfigDict

def parse_token(token: str) -> HTTPAuthorizationCredentials:
    scheme, credentials = token[0].split(" ")
    return HTTPAuthorizationCredentials(scheme=scheme, credentials=credentials)


HTTPAuthToken = Annotated[HTTPAuthorizationCredentials, BeforeValidator(parse_token)]


class MyappSettings(BaseSettings):
    token: HTTPAuthToken

    model_config = SettingsConfigDict(env_file=".env", env_prefix="MYAPP_")

print(MyappSettings())