Howto have Optional Submodels for Settings

[mxab]

Checks

• I added a descriptive title to this issue
• I have searched (google, github) for similar issues and couldn't find anything
• I have read and followed the docs and couldn't find an answer
• After submitting this, I commit to one of:
  • Look through open issues and helped at least one other person
  • Hit the "watch" button on this repo to receive notifications and I commit to help at least 2 people that ask questions in the future
  • Implement a Pull Request for a confirmed bug

Question

In my use case I have optional configuration parts, like for example and auth block. The auth part can be optional, but when provided it should contain username/password. Therefore I want to create a submodel that is consistent.

The BaseSettings seems to require that submodels need to be initiated directly. Which means it forces me to have an Auth model that contains Optional Fields.

Is there a way to get the usecase to work as described below?

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.6.1
            pydantic compiled: True
                 install path: /home/mafr/.cache/pypoetry/virtualenvs/sese-rest-g0otWJoB-py3.6/lib/python3.6/site-packages/pydantic
               python version: 3.6.10 (default, Jul 29 2020, 11:44:44)  [GCC 9.3.0]
                     platform: Linux-4.19.104-microsoft-standard-x86_64-with-debian-bullseye-sid
     optional deps. installed: ['typing-extensions']

...

from _pytest.monkeypatch import MonkeyPatch
from pydantic import BaseSettings, BaseModel
from typing import Optional


class AuthConfig(BaseModel):
    username: str
    password: str


class DemoSettings(BaseSettings):
    foo: str
    auth: Optional[AuthConfig]

# this one passes currently
def test_settings(monkeypatch: MonkeyPatch):

    monkeypatch.setenv("FOO", "hello")

    settings = DemoSettings()

    assert settings.foo == "hello"

# this doesn't
def test_settings2(monkeypatch: MonkeyPatch):

    monkeypatch.setenv("FOO", "hello")
    monkeypatch.setenv("AUTH_USERNAME", "user123")
    monkeypatch.setenv("AUTH_PASSWORD", "pass123")

    settings = DemoSettings()

    assert settings.foo == "hello"
    assert settings.auth.username == "user123"
    assert settings.auth.password == "pass123"

The second test fails as the auth is None

assert settings.auth.username == "user123"
E       AttributeError: 'NoneType' object has no attribute 'username'

[mxab]

Ok I just noticed it works but only if the submodel is json

def test_settings2(monkeypatch: MonkeyPatch):

    monkeypatch.setenv("FOO", "hello")
    monkeypatch.setenv("AUTH", '{"username": "user123", "password": "pass123"}')

    settings = DemoSettings()

    assert settings.foo == "hello"
    assert settings.auth.username == "user123"
    assert settings.auth.password == "pass123"

[mxab]

Ok just saw this in the documentation:

Complex types like list, set, dict, and sub-models are populated from the environment by treating the environment variable's value as a JSON-encoded string.

[mxab]

So my question now is if there is another way besides having to encode complex types as json

[StephenBrown2]

Not easily. There is a PR open for allowing space/comma separated values, but for something like a Key-Value setting with particular requirements, JSON is the best way to do so.

In your code example, the environment variables don't line up with how they would be gathered, however you can make them do so with a little tweaking to the definition, using a validator instead of a sub-model:

import os
from pydantic import BaseSettings, Field, validator
from typing import Optional

class DemoSettings(BaseSettings):
    foo: str
    username: Optional[str] = Field(None, env="auth_username")
    password: Optional[str] = Field(None, env="auth_password")

    @validator("password", always=True)
    def username_and_password(cls, v, values):
        # If both values are provided, good to pass
        if values.get("username") and v:
            return v
        # If neither are provided, also good
        if not (values.get("username") or v):
            return None
        # We only got one of username or password, raise!
        raise ValueError("Username and Password must be provided")

# this one passes still
def test_settings():

    os.environ["FOO"] = "hello"

    settings = DemoSettings()

    assert settings.foo == "hello"
    assert settings.username is None
    assert settings.password is None

# this one raises ValidationError
def test_settings2():
  
    os.environ["FOO"] = "hello"
    os.environ["AUTH_USERNAME"] = "user123"

    settings = DemoSettings()

    assert settings.foo == "hello"
    assert settings.username == "user123"
    assert settings.password == None

ValidationError: 1 validation error for DemoSettings
password
  Username and Password must be provided (type=value_error)

# this works
def test_settings3():

    os.environ["FOO"] = "hello"
    os.environ["AUTH_USERNAME"] = "user123"
    os.environ["AUTH_PASSWORD"] = "pass123"

    settings = DemoSettings()

    assert settings.foo == "hello"
    assert settings.username == "user123"
    assert settings.password == "pass123"

Others may have another way, but that's my recommendation.

[mxab]

Thanks for that recommendation, I'll try this approach.

Do you think it makes sense to create feature request for such a use case with submodel non json optional submodel settings?

[daviskirk]

We use "__" as a hierarchy seperator for environment variables in https://github.com/daviskirk/climatecontrol#usage as a solution for the key/value integration (it has pydantic integration built in). You could probably do something similar with pydantics builtin "env_prefix".

[samuelcolvin]

Thanks for using pydantic. 🙏

As part of a migration to using discussions and cleanup old issues, I'm closing all open issues with the "question" label. 💭 🆘 🚁

I hope you've now found an answer to your question. If you haven't, feel free to start a 👉 question discussion.