Implement proper support for nested complex env values #4216

robertaistleitner · 2022-07-06T12:25:05Z

Change Summary

Pydantic didn't support setting fields containing "complex values" like dicts or lists from environment variables using nesting. This PR contains an implementation addressing this issue by checking which field the nested environment variable targets to, checking if the field is a complex value and if so, trying to parse the environment value as a JSON.

This already worked if explicitly using env=... keyword arg for a field but not for nested environment variables.

Related issue number

There is no issue created yet.

fix #2304 fix #4389

Checklist

Unit tests for the changes exist
Tests pass on CI and coverage remains at 100%
Documentation reflects the changes where applicable
changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details)
My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

robertaistleitner · 2022-07-06T12:50:07Z

Hey there, please review.

robertaistleitner · 2022-07-06T12:51:09Z

Oops I missed the failing coverage check, will fix this.

robertaistleitner · 2022-07-15T07:43:54Z

Please note this PR fixed the same missing feature described in https://github.com/samuelcolvin/pydantic/issues/3971: #4235

What this PR also includes is checking the target field (is it really a complex field) and some more test cases.

changes/4216-raistleitner.md

pydantic/env_settings.py

hramezani · 2022-07-15T14:32:32Z

Thanks @robertaistleitner for the patch 👍

I left some comments.

Also, I think we can improve the Parsing environment variable values doc and improve the example there with some examples from your change.

hramezani · 2022-07-15T14:32:40Z

please update

Simplify complex change description Co-authored-by: Hasan Ramezani <hasan.r67@gmail.com>

Improve typing Co-authored-by: Hasan Ramezani <hasan.r67@gmail.com>

Merge conditions with same outcome Co-authored-by: Hasan Ramezani <hasan.r67@gmail.com>

Improve typing Co-authored-by: Hasan Ramezani <hasan.r67@gmail.com>

robertaistleitner · 2022-07-19T15:23:00Z

I left some comments.

I worked the comments in as they were because they were all valid as is.

Also, I think we can improve the Parsing environment variable values doc and improve the example there with some examples from your change.

I added a simple example showing the implemented feature.

Please review again.

hramezani

Thanks @robertaistleitner for updating. LGTM 👍

samuelcolvin

Thanks so much for this, I think it looks like a great addition.

My only concern is whether it should be included in V1.10 or wait for V2? (We've got a bit of heat recently for introducing breaking changes in minor and patch updates, hence my paranoia)

There are two potential risks:

known unknowns: someone has a very weird schema for their settings and a field (e.g. Union[str, List[str]]) get's a different value after this PR vs. before
unknown unknowns: something else unintended changes

I'll think about it, but any input from @robertaistleitner @AngellusMortis @hramezani @PrettyWood on whether this is a suitable change for a minor release would be very welcome.

samuelcolvin · 2022-08-09T09:28:34Z

pydantic/env_settings.py

+    def next_field(field: Optional[ModelField], key: str) -> Optional[ModelField]:
+        if not field or field.sub_fields:


This method is only called in one place, are you sure it needs to be a separate function?

If so, please add a docstring explaining what we're doing here.

This method gets called twice, that's why I moved it to a method:

https://github.com/pydantic/pydantic/pull/4216/files/82f5b971d3a201fea1402eb5ae8b210a03ad5e2a#diff-67b5bf5da243ce3b23618214851e35a3f0951de739f3747800daf6d1273e4962R248

https://github.com/pydantic/pydantic/pull/4216/files/82f5b971d3a201fea1402eb5ae8b210a03ad5e2a#diff-67b5bf5da243ce3b23618214851e35a3f0951de739f3747800daf6d1273e4962R252

Sorry, you're right, my mistake.

I don't think there are collisions with existing setting schemas due to the fact that setting complex values from an environment variable never really worked.

Nevertheless, it's up to you - for now I'm using a monkeypatch that does the job pretty well for me.

I understand where you're coming from, I'm just a bit paranoid at the moment about "breaking" changes, hence deferring this.

hramezani · 2022-08-09T11:19:42Z

My only concern is whether it should be included in V1.10 or wait for V2?

V2 makes more sense to me. The feature itself is great but the implementation is complex and it may introduce breaking changes that is not visible now.

PrettyWood · 2022-08-10T21:49:51Z

I would also differ this for v2

samuelcolvin · 2022-12-29T11:08:45Z

Closing since:

this is deferred to V2
settings logic has moved to pydantic-settings
We need to resolve [WIP] Improved source management pydantic-settings#10 which might solve this.

samuelcolvin · 2022-12-29T11:09:25Z

Thanks so much @robertaistleitner for the proposed change, I'd love you to contribute to pydantic-settings or indeed pydantic in the future.

Sorry we couldn't accept this this time.

hramezani · 2023-04-26T08:26:04Z

@robertaistleitner I've created a new PR in pydantic-settings based on your idea here.

robertaistleitner · 2023-05-14T17:04:15Z

@hramezani that's awesome, thanks for your effort!

Robert Aistleitner added 3 commits July 6, 2022 14:11

Implement proper support for nested complex env values

86b324a

Add changes file for PR

00d5e66

Remove second paragraph for changes file

47aa0e9

github-actions bot added the ready for review label Jul 6, 2022

github-actions bot assigned hramezani, PrettyWood and samuelcolvin Jul 6, 2022

Improve tests for coverage and code cleanup

5a31ebd

hramezani reviewed Jul 15, 2022

View reviewed changes

changes/4216-raistleitner.md Outdated Show resolved Hide resolved

hramezani reviewed Jul 15, 2022

View reviewed changes

pydantic/env_settings.py Outdated Show resolved Hide resolved

hramezani reviewed Jul 15, 2022

View reviewed changes

pydantic/env_settings.py Outdated Show resolved Hide resolved

hramezani reviewed Jul 15, 2022

View reviewed changes

pydantic/env_settings.py Outdated Show resolved Hide resolved

github-actions bot added awaiting author revision and removed ready for review labels Jul 15, 2022

github-actions bot assigned robertaistleitner and unassigned hramezani, samuelcolvin and PrettyWood Jul 15, 2022

hramezani mentioned this pull request Jul 15, 2022

Fix nested fields not json parsed when they are complex #3971 #4235

Closed

5 tasks

robertaistleitner and others added 6 commits July 19, 2022 17:06

Update changes/4216-raistleitner.md

a6df189

Simplify complex change description Co-authored-by: Hasan Ramezani <hasan.r67@gmail.com>

Update pydantic/env_settings.py

cad5c1c

Improve typing Co-authored-by: Hasan Ramezani <hasan.r67@gmail.com>

Update pydantic/env_settings.py

0622281

Merge conditions with same outcome Co-authored-by: Hasan Ramezani <hasan.r67@gmail.com>

Update pydantic/env_settings.py

ffbf310

Improve typing Co-authored-by: Hasan Ramezani <hasan.r67@gmail.com>

Add nested complex env variable example to docs

6c16022

Provide default value for example

82f5b97

github-actions bot added ready for review and removed awaiting author revision labels Jul 19, 2022

github-actions bot assigned hramezani, PrettyWood and samuelcolvin and unassigned robertaistleitner Jul 19, 2022

hramezani approved these changes Jul 19, 2022

View reviewed changes

samuelcolvin approved these changes Aug 9, 2022

View reviewed changes

samuelcolvin added the Feedback Wanted label Aug 9, 2022

samuelcolvin mentioned this pull request Aug 9, 2022

V1.10 release #4324

Closed

14 tasks

samuelcolvin added deferred deferred until future release or until something else gets done and removed Feedback Wanted ready for review labels Aug 11, 2022

samuelcolvin unassigned hramezani, samuelcolvin and PrettyWood Aug 11, 2022

samuelcolvin mentioned this pull request Aug 12, 2022

fix ability to mix/merge json env variables with env_nested_delimiter… #3640

Closed

5 tasks

jooola mentioned this pull request Oct 4, 2022

Allow recursive env variables loading based on models jsonschema pydantic/pydantic-settings#5

Open

14 tasks

hramezani mentioned this pull request Aug 17, 2022

Error with BaseSettings from a .env when trying to load a dictionary in a nested sub-model. #4389

Closed

16 tasks

samuelcolvin closed this Dec 29, 2022

hramezani mentioned this pull request Apr 25, 2023

Implement proper support for nested complex env values pydantic/pydantic-settings#22

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement proper support for nested complex env values #4216

Implement proper support for nested complex env values #4216

robertaistleitner commented Jul 6, 2022 •

edited by samuelcolvin

robertaistleitner commented Jul 6, 2022

robertaistleitner commented Jul 6, 2022 •

edited

robertaistleitner commented Jul 15, 2022 •

edited

hramezani commented Jul 15, 2022

hramezani commented Jul 15, 2022

robertaistleitner commented Jul 19, 2022

hramezani left a comment

samuelcolvin left a comment

samuelcolvin Aug 9, 2022

robertaistleitner Aug 11, 2022

samuelcolvin Aug 11, 2022

robertaistleitner Aug 11, 2022

samuelcolvin Aug 11, 2022

hramezani commented Aug 9, 2022

PrettyWood commented Aug 10, 2022

samuelcolvin commented Dec 29, 2022

samuelcolvin commented Dec 29, 2022

hramezani commented Apr 26, 2023

robertaistleitner commented May 14, 2023

		def next_field(field: Optional[ModelField], key: str) -> Optional[ModelField]:
		if not field or field.sub_fields:

Implement proper support for nested complex env values #4216

Implement proper support for nested complex env values #4216

Conversation

robertaistleitner commented Jul 6, 2022 • edited by samuelcolvin

Change Summary

Related issue number

Checklist

robertaistleitner commented Jul 6, 2022

robertaistleitner commented Jul 6, 2022 • edited

robertaistleitner commented Jul 15, 2022 • edited

hramezani commented Jul 15, 2022

hramezani commented Jul 15, 2022

robertaistleitner commented Jul 19, 2022

hramezani left a comment

Choose a reason for hiding this comment

samuelcolvin left a comment

Choose a reason for hiding this comment

samuelcolvin Aug 9, 2022

Choose a reason for hiding this comment

robertaistleitner Aug 11, 2022

Choose a reason for hiding this comment

samuelcolvin Aug 11, 2022

Choose a reason for hiding this comment

robertaistleitner Aug 11, 2022

Choose a reason for hiding this comment

samuelcolvin Aug 11, 2022

Choose a reason for hiding this comment

hramezani commented Aug 9, 2022

PrettyWood commented Aug 10, 2022

samuelcolvin commented Dec 29, 2022

samuelcolvin commented Dec 29, 2022

hramezani commented Apr 26, 2023

robertaistleitner commented May 14, 2023

robertaistleitner commented Jul 6, 2022 •

edited by samuelcolvin

robertaistleitner commented Jul 6, 2022 •

edited

robertaistleitner commented Jul 15, 2022 •

edited