allow use of a `.env`-style files in BaseSettings (#607) #1011

acnebs · 2019-11-19T17:07:44Z

Change Summary

Update BaseSettings class to allow the use of .env-style files for storing sensitive environment variables.

Just pass in the path (either absolute or relative to the working directory) to such a file in the Config subclass, and pydantic will do the rest for you.

.env

foo=baz

settings.py

from pydantic import BaseSettings

class Settings(BaseSettings):
    foo: str = 'bar'

    class Config:
        env_file = '.env'

test.py

from settings import Settings

s = Settings()
print(s.foo)
# output: baz

Related issue number

#607

Checklist

Unit tests for the changes exist
Tests pass on CI and coverage remains at 100%
Documentation reflects the changes where applicable
changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details)

codecov · 2019-11-19T17:18:17Z

Codecov Report

❗ No coverage uploaded for pull request base (master@23c9730). Click here to learn what that means.
The diff coverage is 100%.

@@           Coverage Diff            @@
##             master   #1011   +/-   ##
========================================
  Coverage          ?    100%           
========================================
  Files             ?      20           
  Lines             ?    3507           
  Branches          ?     683           
========================================
  Hits              ?    3507           
  Misses            ?       0           
  Partials          ?       0

Impacted Files	Coverage Δ
pydantic/datetime_parse.py	`100% <ø> (ø)`
pydantic/errors.py	`100% <100%> (ø)`
pydantic/class_validators.py	`100% <100%> (ø)`
pydantic/version.py	`100% <100%> (ø)`
pydantic/typing.py	`100% <100%> (ø)`
pydantic/tools.py	`100% <100%> (ø)`
pydantic/dataclasses.py	`100% <100%> (ø)`
pydantic/networks.py	`100% <100%> (ø)`
pydantic/validators.py	`100% <100%> (ø)`
pydantic/schema.py	`100% <100%> (ø)`
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 23c9730...04c51df. Read the comment docs.

samuelcolvin

Looking good.

Also needs tests and documentation.

pydantic/env_settings.py

- allow specifying `_env_file` kwarg in instantiation * overrides any `env_file` specified in the `Config` class - cast `os.environ` as a dict for better consistenty of behavior - `env_path` should be a `Path` type - replace `with open()` with `read_text` - use regex for parsing the dotenv files and throw error on invalid line - factor out `read_env_file` into separate file for easier testing

acnebs · 2019-11-20T15:33:28Z

Still haven't written any tests yet, but I figure it's best to leave that until we've settled on the rest of the implementation.

pydantic/env_settings.py

pydantic/read_env_file.py

pydantic/env_settings.py

euri10 · 2019-11-21T17:30:39Z

Sorry to highjack the PR, I'm currently having a simple function to push .env fies into objects inheriting BaseSettings using load_dotenv from the dotenv package.

Stuff like this for instance:

class DatabaseSetting(BaseSettings):
    host: str = "localhost"
    port: int = 5432
    user: str
    password: str
    db: str

    class Config:
        env_prefix = "POSTGRES_"

def get_db_settings(dotenv_path: Union[str, PathLike]) -> DatabaseSetting:
    loaded = load_dotenv(dotenv_path=dotenv_path, verbose=True, override=True)
    return DatabaseSetting()

maybe the "loading" part could take inspiration from it, just an idea,

well either way that's a good idea and will reduce dependencies so thankls for that !

samuelcolvin

Also need tests and documentation.

pydantic/env_settings.py

acnebs · 2019-11-23T07:35:22Z

@euri10 Thanks for the suggestion! I think in the end the dotenv approach is probably a bit much for this builtin functionality at the moment, which is why I initially went for something as simple as possible. I had also found this library while researching different approaches, which does something similar to what you're doing.

But yeah, since Settings handling is a first-class use-case of Pydantic, I felt that it was much nicer to have it integrated into the lib without dependencies.

acnebs · 2019-11-23T08:06:18Z

Ok, I think I've sorted everything on the PR checklist. Let me know if anything else needs changing.

docs/usage/settings.md

samuelcolvin · 2019-11-25T09:51:25Z

docs/usage/settings.md

+#### `.env`
+
+```
+# ignore comment


I think this needs an example of python code to read this file so the stripping of quotes etc. is shown.

As in just include the actual code we're using to parse env files in the docs, or...?

samuelcolvin · 2019-11-25T09:54:15Z

tests/env_files/test_settings_1.env

@@ -0,0 +1,6 @@
+# this is a comment


Better to use tmp_path than create actual files like this.

Is it? It creates a lot of repetition in the tests since we're testing various different things but can re-use the same files. Will make the changes though if you think it is worth it.

tests/test_settings.py

pydantic/env_settings.py

tests/test_settings.py

acnebs · 2019-11-26T02:06:38Z

@samuelcolvin Do we really want existing environment variables to take precedence over the .env file?

One of the major benefits of using dotenv is that it makes it far easier to ensure consistency of your settings. In an actual environment, there can be many different places / programs / scripts etc. modifying your vars. If you use the .env file, you can be sure that everything in that file will take precedence. The idea is that you're taking the file, and if it's valid, you're setting all the values inside it as env vars. At least that's how other libs like python-dotenv behave.

That makes the most sense to me, as otherwise you might have a bunch of vars in this file you're loading that are silently ignored, which seems like it would do more harm than good.

euri10 · 2019-11-26T07:58:30Z

One of the major benefits of using dotenv is that it makes it far easier to ensure consistency of your settings. In an actual environment, there can be many different places / programs / scripts etc. modifying your vars. If you use the .env file, you can be sure that everything in that file will take precedence. The idea is that you're taking the file, and if it's valid, you're setting all the values inside it as env vars. At least that's how other libs like python-dotenv behave.

Don't quote me because on that, I'm not too sure about it, but I think in python-dotenv you need to be explicit about precedence using the override kwarg.

samuelcolvin · 2019-11-26T11:01:25Z

No, environment variables should definitely take precedence over variables in .env. As they do in other libraries:

with create-react-app environment variables take priority, see here
with starlette environment variables take priority, see here
I've just tried with python-dotenv, as expected environment variables take priority:

(env) pydantic 1 333ms ➤  pip install python-dotenv
...
(env) pydantic 0 971ms ➤  echo FOO=whatever >> .env
(env) pydantic 0  13ms ➤  echo BAR=other >> .env
(env) pydantic 05ms ➤  cat .env 
FOO=whatever
BAR=other
(env) pydantic 08ms ➤  export FOO=different
(env) pydantic 06ms ➤  ipython

Python 3.7.4
import asyncio, base64, math, hashlib, json, os, pickle, random, re, secrets, sys, time
from datetime import datetime, date, timedelta, timezone
from pathlib import Path

autoreload enabled, use `%autoreload 0` to disable

In [1]: from dotenv import load_dotenv

In [2]: load_dotenv()
Out[2]: True

In [3]: import os

In [4]: os.getenv('FOO')
Out[4]: 'different'

In [5]: os.getenv('BAR')
Out[5]: 'other'

samuelcolvin · 2019-11-26T11:02:10Z

In an actual environment, there can be many different places / programs / scripts etc. modifying your vars.

That's what the env variable prefix is for.

samuelcolvin · 2019-12-09T10:06:54Z

Hi @acnebs, are you still okay to finish this?

acnebs · 2019-12-09T19:14:56Z

@samuelcolvin Yep, just been a bit busy over the past couple weeks. Should be able to get to it soon.

pydantic/env_settings.py

sm-Fifteen · 2020-01-13T15:47:00Z

Any news about that PR? I'm pretty excided about getting rid of my custom .env config data extraction and casting logic so I can replace it with Pydantic instead.

samuelcolvin · 2020-01-13T16:00:03Z

me too, if there's no progress I'll try and finish it myself when I get a chance.

@sm-Fifteen if you'd like to take over this, by all means create a new PR from this work. I'll make sure we give credit to both developers.

Co-Authored-By: Samuel Colvin <samcolvin@gmail.com>

acnebs · 2020-01-13T20:42:02Z

Sorry about the long delay, getting back to this now!

sm-Fifteen · 2020-01-13T21:13:20Z

Looking at the code more closely, shouldn't we just use python-dotenv (as an optional dependency) for the file parsing instead of making our own spin on it, possibly with its own quirks and issues? Because parsing those files correctly looks awfully tricky.

I just checked and python-dotenv's quoting behavior doesn't even match dotenv's since mismatched quotes (see this file) appear to assume the current line continues as a multiline string in the python lib, but the JS lib treats the lone quote as a literal character. I don't think the current logic implemented in this PR even handles mismatched quotes at all.

python-dotenv would at least ensure that the parsing is done consistently with other Python applications, without adding an extra maintenance burden on pydantic.

import dotenv
# A dict, which can then be used to populate BaseSettings
env_dict = dotenv.dotenv_values('.env')

EDIT: See theskumar/python-dotenv#170 for some of the differences between parsers.

samuelcolvin · 2020-01-13T21:33:50Z

No. For reasons I think I explained above.

We can cover 99.9% of usage very easily, if you want more you should probably use yaml or toml.

samuelcolvin · 2020-01-19T14:45:48Z

upon reflection, I've decided to take on @sm-Fifteen's comment and switch to using python-dotenv. While I don't like the extra (optional) dependency for one use, I can't ignore the problems you discuss regarding quotes etc.

@acnebs I'm sorry we spent so long building a dotenv parser that's now not being used, but I really do think this is a better approach.

sm-Fifteen · 2020-01-19T18:59:23Z

I'm not seeing tests for types other than strings, does the current implementation support other types of field, such as ints, floats, booleans, optionals and possibly JSON-formatted BaseModels?

samuelcolvin · 2020-01-19T19:07:13Z

Yes of course, same as pydantic always does. Those values will be coerced to the field's type.

There is definitely at least one test which demonstrates this.

Lists etc. would require the Json type.

sm-Fifteen · 2020-01-19T19:12:56Z

Yeah, ok, thank you for confirming this. That pretty much covers what I needed this to do, then.

acnebs · 2020-01-21T14:43:26Z

@samuelcolvin Sounds good. Thanks for finishing it up!

layday · 2020-01-24T14:01:50Z

BaseSettings._build_values used to be part of the public interface. The docs said:

By default BaseSettings considers field values in the following priority ... This behaviour can be changed by overriding the _build_values method on BaseSettings.

Mention of this was removed in v1 and this PR breaks code which relied on _build_values to change the evaluation order.

samuelcolvin · 2020-01-24T14:43:50Z

sorry about that I wasn't aware that _build_values had at one point been publicly documented.

PR welcome if you wish to update the release notes to include this.

layday · 2020-01-24T14:52:14Z

Would you also accept a PR to allow customising the evaluation order in a future version?

samuelcolvin · 2020-01-24T15:05:38Z

I guess, though I imagine it could be quite complex.

Quite related: I also want to allow multiple dot-env files with different priorities which are then merged, same as CRA.

…#1011)

allow use of a .env-style files in BaseSettings (#607)

2eb5939

samuelcolvin reviewed Nov 19, 2019

View reviewed changes

samuelcolvin reviewed Nov 20, 2019

View reviewed changes

pydantic/env_settings.py Outdated Show resolved Hide resolved

move back into a single file; revert typing changes; use regex better

9b9a4a9

samuelcolvin reviewed Nov 22, 2019

View reviewed changes

pydantic/env_settings.py Outdated Show resolved Hide resolved

acnebs added 3 commits November 23, 2019 01:49

pass _env_file argument around instead of setting a class attribute

dfe090f

add dotenv docs

ff38cec

add dotenv tests

e5e9326

Add changes file

b00ee01

Flesh out the docs a bit

9c43ef7

samuelcolvin reviewed Nov 25, 2019

View reviewed changes

samuelcolvin reviewed Dec 10, 2019

View reviewed changes

pydantic/env_settings.py Outdated Show resolved Hide resolved

Apply suggestions from @samuelcolvin's code review

3e112ad

Co-Authored-By: Samuel Colvin <samcolvin@gmail.com>

samuelcolvin added 5 commits January 19, 2020 13:44

wrap docs

5476690

add not about priority

23c9730

Merge branch 'master' into add-env-file-support

2f1f6d4

fix tests and imports

3d71d02

fix tests

0957304

samuelcolvin added 5 commits January 19, 2020 14:50

switch to python-dotenv

24dba06

cleanup, test example

9533a4a

more docs tweaks

030abb0

typo

95a3316

fix tests for dotenv

04c51df

samuelcolvin merged commit 730d842 into pydantic:master Jan 19, 2020

phy25 mentioned this pull request Jan 23, 2020

.env file like crea when using BaseSettings #607

Closed

dmfigol mentioned this pull request Jan 26, 2020

pydantic 1.4 breaks nornir nornir-automation/nornir#473

Closed

acnebs deleted the add-env-file-support branch January 30, 2020 17:15

fullc0ntr0l mentioned this pull request Apr 4, 2020

BaseSettings can't read from .env file #1368

Closed

layday mentioned this pull request Oct 20, 2020

BaseSettings: Customization of field value priority via Config pydantic/pydantic-settings#32

Closed

4 tasks

alexdrydew pushed a commit to alexdrydew/pydantic that referenced this pull request Dec 23, 2023

Fix regex_engine being rejected by validate_core_schema (pydantic…

8e66bd9

…#1011)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

allow use of a `.env`-style files in BaseSettings (#607) #1011

allow use of a `.env`-style files in BaseSettings (#607) #1011

acnebs commented Nov 19, 2019 •

edited

codecov bot commented Nov 19, 2019 •

edited

samuelcolvin left a comment

acnebs commented Nov 20, 2019

euri10 commented Nov 21, 2019

samuelcolvin left a comment

acnebs commented Nov 23, 2019

acnebs commented Nov 23, 2019 •

edited

samuelcolvin Nov 25, 2019 •

edited

acnebs Jan 13, 2020

samuelcolvin Nov 25, 2019 •

edited

acnebs Jan 13, 2020 •

edited

acnebs commented Nov 26, 2019

euri10 commented Nov 26, 2019

samuelcolvin commented Nov 26, 2019

samuelcolvin commented Nov 26, 2019 •

edited

samuelcolvin commented Dec 9, 2019

acnebs commented Dec 9, 2019

sm-Fifteen commented Jan 13, 2020

samuelcolvin commented Jan 13, 2020

acnebs commented Jan 13, 2020

sm-Fifteen commented Jan 13, 2020 •

edited

samuelcolvin commented Jan 13, 2020

samuelcolvin commented Jan 19, 2020 •

edited

sm-Fifteen commented Jan 19, 2020

samuelcolvin commented Jan 19, 2020

sm-Fifteen commented Jan 19, 2020

acnebs commented Jan 21, 2020

layday commented Jan 24, 2020

samuelcolvin commented Jan 24, 2020

layday commented Jan 24, 2020

samuelcolvin commented Jan 24, 2020

allow use of a .env-style files in BaseSettings (#607) #1011

allow use of a .env-style files in BaseSettings (#607) #1011

Conversation

acnebs commented Nov 19, 2019 • edited

Change Summary

.env

settings.py

test.py

Related issue number

Checklist

codecov bot commented Nov 19, 2019 • edited

Codecov Report

samuelcolvin left a comment

Choose a reason for hiding this comment

acnebs commented Nov 20, 2019

euri10 commented Nov 21, 2019

samuelcolvin left a comment

Choose a reason for hiding this comment

acnebs commented Nov 23, 2019

acnebs commented Nov 23, 2019 • edited

samuelcolvin Nov 25, 2019 • edited

Choose a reason for hiding this comment

acnebs Jan 13, 2020

Choose a reason for hiding this comment

samuelcolvin Nov 25, 2019 • edited

Choose a reason for hiding this comment

acnebs Jan 13, 2020 • edited

Choose a reason for hiding this comment

acnebs commented Nov 26, 2019

euri10 commented Nov 26, 2019

samuelcolvin commented Nov 26, 2019

samuelcolvin commented Nov 26, 2019 • edited

samuelcolvin commented Dec 9, 2019

acnebs commented Dec 9, 2019

sm-Fifteen commented Jan 13, 2020

samuelcolvin commented Jan 13, 2020

acnebs commented Jan 13, 2020

sm-Fifteen commented Jan 13, 2020 • edited

samuelcolvin commented Jan 13, 2020

samuelcolvin commented Jan 19, 2020 • edited

sm-Fifteen commented Jan 19, 2020

samuelcolvin commented Jan 19, 2020

sm-Fifteen commented Jan 19, 2020

acnebs commented Jan 21, 2020

layday commented Jan 24, 2020

samuelcolvin commented Jan 24, 2020

layday commented Jan 24, 2020

samuelcolvin commented Jan 24, 2020

allow use of a `.env`-style files in BaseSettings (#607) #1011

allow use of a `.env`-style files in BaseSettings (#607) #1011

acnebs commented Nov 19, 2019 •

edited

codecov bot commented Nov 19, 2019 •

edited

acnebs commented Nov 23, 2019 •

edited

samuelcolvin Nov 25, 2019 •

edited

samuelcolvin Nov 25, 2019 •

edited

acnebs Jan 13, 2020 •

edited

samuelcolvin commented Nov 26, 2019 •

edited

sm-Fifteen commented Jan 13, 2020 •

edited

samuelcolvin commented Jan 19, 2020 •

edited