Cannot load JSON from secret files

[nat212]

Checks

• I added a descriptive title to this issue
• I have searched (google, github) for similar issues and couldn't find anything
• I have read and followed the docs and still think this is a bug

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.8.2
            pydantic compiled: True
                 install path: /home/natasha/.local/share/virtualenvs/draperweb3-7iyfNlF4/lib/python3.8/site-packages/pydantic
               python version: 3.8.6 (default, May 27 2021, 13:28:02)  [GCC 10.2.0]
                     platform: Linux-5.11.0-7614-generic-x86_64-with-glibc2.32
     optional deps. installed: ['dotenv', 'email-validator', 'typing-extensions']

When loading settings values from a secrets file, values encoded in JSON do not get correctly parsed, leading to a ValidationError. After looking through the code, I discovered that SecretsSettingsSource simply reads the secret value as text instead of checking if it is a complex type. I'm not sure if this is intended, or if it's a bug, but either way, I think it would be beneficial if it were possible to load JSON from a secret file.

Example:

pydantic.error_wrappers.ValidationError: 1 validation error for Settings
cloud_config
  value is not a valid dict (type=type_error.dict)

Minimal example

from pydantic import BaseSettings, BaseModel


class CloudConfig(BaseModel):
    client_id: str
    client_secret: str
    token_url: str
    auth_url: str
    api_url: str


class Settings(BaseSettings):
    cloud_config: CloudConfig

    class Config:
        case_sensitive = False
        secrets_dir = "/run/secrets"

settings = Settings()
print(settings.cloud_config.client_id)

This works if I have an environment variable CLOUD_CONFIG='{"client_id": "...", ...}', but not if I have the same string in a secrets file

[PrettyWood]

Hi @nat212
Your minimal example is not complete. Can you add what is actually run?
(I'm surprised your CloudConfig is not a BaseSettings)

[nat212]

Sure, I've updated it to include a little more detail. The example in the Settings management docs use a BaseModel as the submodel, so that is why it is not a BaseSettings. Would it be better to make it a BaseSettings?

[PrettyWood]

If you look at the example, you can see that the BaseModel doesn't contain any secret and it's explain in the comment that you need to stringify a dict to override it.
If you want to write only your client_secret into /run/secrets/client_secret, CloudConfig needs to be an BaseSettings and have the right Config declared

[nat212]

Yes, the issue is that when using a JSON string as an environment variable, it sets the cloud_config model successfully. So the environment variable should get detected by the Settings model, and parsed as a JSON string, and used to set the cloud_config model. However, if I have a secret called cloud_config with the same JSON string, the secret just gets read as plain text, and the Settings model complains that it has not received a dict

[davidmreed]

I'm encountering the same issue, and want to clarify that (at least as I understand it) the problem is not with using a nested BaseModel within a BaseSettings, but using any complex type within a BaseSettings that needs to be loaded from a secrets file.

Here's an example where the BaseSettings property is a dict:

class OAuth2ServiceConfig(BaseModel):
    client_id: str
    client_secret: str
    auth_uri: str
    token_uri: str

class APISettings(BaseSettings):
    oauth2_services: Dict[str, OAuth2ServiceConfig] = {}
    # ... other properties ...

    class Config:
        secrets_dir = "/run/secrets"

In docker-compose.yml:

version: '3'

services:
  myapp:
    build: .
    secrets:
      - oauth2_services
secrets:
  oauth2_services:
    file: ./secrets/oauth2_services.txt

secrets/oauth2_services.txt:

{
    "foo": {
        "client_id": "<snipped>",
        "client_secret": "<snipped>",
        "auth_uri": "<snipped>",
        "token_uri": "<snipped>"
    }
}

The outcome is

pydantic.error_wrappers.ValidationError: 1 validation error for APISettings

oauth2_services

value is not a valid dict (type=type_error.dict)