Optional (or Union) types cannot be loaded from JSON secrets

[daBrado]

If one tries to use an optional settings field with a complex type, it cannot be assigned to from a secrets file that contains JSON. For example:

from os import environ
from os.path import join
from tempfile import TemporaryDirectory

from pydantic import BaseModel
from pydantic_settings import BaseSettings


class Foo(BaseModel):
    a: int


class Settings(BaseSettings):
    foo: Foo


class SettingsOptional(BaseSettings):
    foo: Foo | None = None


with TemporaryDirectory() as tmpdir:
    with open(join(tmpdir, "foo"), "w") as f:
        f.write('{"a":10}')

    bar = Settings(_secrets_dir=tmpdir)
    print(bar)  # foo=Foo(a=10), as expected.

    # Errors out with:
    #   Traceback (most recent call last):
    #     File "/.../example.py", line 41, in <module>
    #       bar2 = SettingsOptional(_secrets_dir=tmpdir)
    #              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    #     File "/.../python3.11/site-packages/pydantic_settings/main.py", line 71, in __init__
    #       super().__init__(
    #     File "/.../lib/python3.11/site-packages/pydantic/main.py", line 164, in __init__
    #       __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
    #   pydantic_core._pydantic_core.ValidationError: 1 validation error for SettingsOptional
    #   foo
    #     Input should be a valid dictionary or instance of Foo [type=model_type, input_value='{"a":10}', input_type=str]
    #       For further information visit https://errors.pydantic.dev/2.5/v/model_type
    bar2 = SettingsOptional(_secrets_dir=tmpdir)

This seems inconsistent. I wouldn't expect making the foo field optional to disallow the ability to load its value as JSON.

[hramezani]

Thanks @daBrado for reporting this issue and sorry for the late response.

Yeah, seems like a bug to me and the reason is by making foo optional(foo: Foo | None = None), pydantic-settings does not consider the field as a complex field.

But this is not the case in env source. if you set FOO='{"a":10}' env variable it SettingsOptional works as expected.