Usage of Field and BaseSettings, migration to v2

[CharlesPerrotMinotHCHB]

Hi,
I'm trying to migrate our projects to pydantic v2, but have some issues in the Fields / BaseSettings usage (although it very likely relies around Field). The following used to work, but not any more, and I didn't find a correct way to fix this.

from pydantic.networks import HttpUrl
from pydantic_settings import BaseSettings, SettingsConfigDict

class Url(HttpUrl):
  ...

class AppSettings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
    )

class Settings(AppSettings):
    SOME_URL: Url = Field("http://something:7007")

It fails complaining about the type of SOME_URL

E   pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
E   SOME_URL
E     Input should be an instance of Url [type=is_instance_of, input_value='http://something:7007', input_type=str]
E       For further information visit https://errors.pydantic.dev/2.4/v/is_instance_of

I Have tried using Annotated without success

SOME_URL: Annotated[
  Url,
  Field("http://something:7007"),
]

The only way that works, but that I'd like to avoid, is to instantiate the object in the Settings.

SOME_URL: Url = Field(
    Url("http://something:7007")
)

For some reason, the original code works if I use directly HttpUrl instead of the Url subclass.

from pydantic.networks import HttpUrl
from pydantic_settings import BaseSettings, SettingsConfigDict

class AppSettings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
    )

class Settings(AppSettings):
    SOME_URL: HttpUrl = Field("http://something:7007")

[hramezani]

As the Url class is an arbitrary class, Pydantic checks the default value to be an instance of that class.

[CharlesPerrotMinotHCHB]

Okay, but that wasn't the case before (with pydantic v1), and the doc doesn't explain how to remedy this.
So now we have to pass a constructed instance in that case?

Also, it's not quite arbitrary since it's a child class of HttpUrl from pydantic.networks, isn;t it?

[CharlesPerrotMinotHCHB]

Anyone else has a clue here?
It's blocking my migration because of the following.

Before, we were able to do this:

class JWK:
  ...
class JWKPlaceholder(JWK):
  ...

PLATFORM_JWK: JWK = Field(
    JWKPlaceholder(),
    env="JWT_KEY",
)

Where pydantic would take the JWT_KEY value from env and use it to create a JWK. But now it doesn't work at all, partly because of the change of behavior in the initialization of the Field (see main question). I tried changing to the new json_schema_extra (json_schema_extra={"env": "JWT_KEY"}), but it didn't solve.

[CharlesPerrotMinotHCHB]

So, getting back to trying to migrate to pydantic v2, and still hitting this issue.

I think it relates to this: https://docs.pydantic.dev/latest/concepts/pydantic_settings/#parsing-environment-variable-values

Sub model has to inherit from pydantic.BaseModel, Otherwise pydantic-settings will initialize sub model, collects values for sub model fields separately, and you may get unexpected results.

My issue being, BaseSettings really doesn't work at all if the type is not a BaseModel, and even simply inherit from str or of of the pydantic other models like HttpUrl (see example in first post). The only workaround I have found so far, is to create a model which contains my variable type, and have the settings as a json. For example:

class MyModel(BaseModel):
    url: HttpUrl

class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
    )

    MY_VAR: MyModel

With the following .env:

MY_VAR = '{"url": "https://google.com"}'

It's quite bothersome (because of the added json around the value), and I'm not even sure how to make it work when variables are injected in the env file, like from a k8s config for example.

[hramezani]

I think you can use the environment variables and env_nested_delimiter config. like:

from pydantic import BaseModel, HttpUrl
from pydantic_settings import BaseSettings, SettingsConfigDict

class MyModel(BaseModel):
    url: HttpUrl

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_nested_delimiter='__')
    MY_VAR: MyModel

print(Settings())

So, by setting MY_VAR__URL=https://google.com in env, you will get MY_VAR=MyModel(url=Url('https://google.com/'))

[CharlesPerrotMinotHCHB]

So, it does solve the config injection, indeed. That being said...

• It still requires having to create a container class for non-BaseModel classes
• Forces to change the configurations, which I would rather avoid

I'm going to try if there's a way around that as well

[CharlesPerrotMinotHCHB]

I tested it further and my previous workaround is causing more issues than I though, largely around having to change all the configurations, plus part of the code to support the fact that the class is no longer a subclass of the type but a container, so what used to be a direct access needs to call out the attribute instead, which is clearly not ideal.

Digging into the documentation (https://docs.pydantic.dev/latest/concepts/pydantic_settings/#parsing-environment-variable-values), I came up with another workaround:

from pydantic.fields import FieldInfo
from pydantic_settings import (
    BaseSettings,
    EnvSettingsSource,
    PydanticBaseSettingsSource,
    SettingsConfigDict,
)
from pydantic_core import PydanticUndefined, Url

class MyCustomParser(EnvSettingsSource):
    def set_settings_source(
        self,
        settings: PydanticBaseSettingsSource,
    ) -> "MyCustomParser":
        self.settings = settings
        return self

    def prepare_field_value(
        self,
        field_name: str,
        field: FieldInfo,
        value: Any,
        _complex: bool,
    ) -> Any:
        config_value = self.settings.get_field_value(field, field_name)[0]
        if value is None:
            if field.default is not PydanticUndefined:
                value = field.default
            elif field.default_factory is not None:
                return field.default_factory()
            elif config_value is None:
                return None
            else:
                value = config_value
            value = config_value
        if field.annotation: and (issubclass(field.annotation, Url) or (
            field.annotation.__class__.__module__ == "builtins"
        )):
            return field.annotation(value)
        return json.loads(value)


class AppSettings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env",
        env_file_encoding="utf-8",
    )

    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        return (MyCustomParser(settings_cls).set_settings_source(dotenv_settings),)

The main benefits:

• No configuration change needed
• Supports any subclass of Url or built-in python type, so existing classes don't need to be changed

Caveats:

• Only supports the default settings value + .env (our use case), though it's upgradeable to support more

[CharlesPerrotMinotHCHB]

@samuelcolvin I feel like it'd be way easier if integrated in pydantic like it used to be (in v1), to support all sources of settings, and without everyone having to do it. Basically, when a settings attribute is a subclass of a built-in class, or a subclass of a pydantic model (like pydantic.networks.Url), cast the string value from the settings to that class.