Add back support for `json_encoders` #6811

adriangb · 2023-07-23T10:41:21Z

Closes #6726, closes #6375,

Selected Reviewer: @dmontagu

cloudflare-workers-and-pages · 2023-07-23T10:51:21Z

Deploying with Cloudflare Pages

Latest commit:	`2e756f4`
Status:	✅ Deploy successful!
Preview URL:	https://9c93e4ad.pydantic-docs2.pages.dev
Branch Preview URL:	https://json-encoders.pydantic-docs2.pages.dev

View logs

tests/mypy/outputs/1.1.1/mypy-plugin_ini/pydantic_settings.py

adriangb · 2023-07-24T14:50:51Z

please review

adriangb · 2023-07-24T16:37:10Z

tests/mypy/outputs/1.1.1/mypy-plugin-strict-no-any_ini/dataclass_no_any.py

+@dataclass
+# MYPY: error: Expression type contains "Any" (has type overloaded function)  [misc]
+class Foo:
+    foo: int
+
+
+@dataclass(config=ConfigDict(title='Bar Title'))
+# MYPY: error: Expression type contains "Any" (has type "Type[ConfigDict]")  [misc]
+# MYPY: error: Expression type contains "Any" (has type "ConfigDict")  [misc]
+class Bar:
+    bar: str


@hramezani did you intentionally add this error?

No, It was generated by the script. Actually, we can say by your change.

The point of this test was specifically to make sure this error didn't get created. However, I've discovered that using a type alias for the callable in json_encoders seems to make this go away. (At least with mypy 1.1.1...), which I've just pushed in a commit. So hopefully that resolves the issue.

dmontagu · 2023-07-24T21:59:08Z

pydantic/_internal/_generate_schema.py

+        if 'ref' in schema:
+            return core_schema.definition_reference_schema(schema_ref=schema['ref'], serialization=serialization)  # type: ignore


This seems problematic to me — it's not currently hit by any of the tests (as determined by putting if 'ref' in schema: assert False, and it seems like in principle it could result in a definition getting removed from the schema. Does that make sense? I'm trying to look further into it now, maybe it happens to be the case that this can't cause issues...

I haven't been able to reproduce it (though investigating did lead me to the other comment I made, about self-referencing encoders), but yeah this does seem to make the assumption that the original definition will be present elsewhere, I'm not sure whether that is a safe assumption or not, maybe you know based on the way definitions get stored in GenerateSchema or whatever.

You're right, it's not needed. We're pretty good about always retuning a definitions-reference schema and putting the schema into definitions so we never really end up passing a round a schema with a 'ref'.

dmontagu · 2023-07-24T22:08:07Z

Maybe enough of an edge case to ignore, but it might be nice to improve the way you can use a self-referential JSON encoder.

In particular, the following works:

from pydantic import BaseModel


class Model1(BaseModel):
    model_config = {'json_encoders': {}}
    x: 'Model1 | None'


Model1.model_config['json_encoders'][Model1] = lambda x: x.x if x.x is not None else 1
del Model1.__pydantic_core_schema__
Model1.model_rebuild(force=True)

print(Model1(x=Model1(x=None)).model_dump_json())
#> {"x":1}

But obviously this is a lot worse than doing something like model_config = {'json_encoders': {'Model1': lambda x: x.x}}

I guess maybe we can postpone this as a feature request for the future.

pydantic/_internal/_generate_schema.py

dmontagu

Other than the comments I've made, this all looks good to me, so I'll approve for now so you can merge after reviewing those comments and making changes if appropriate.

pydantic/_internal/_generate_schema.py

Co-authored-by: David Montague <35119617+dmontagu@users.noreply.github.com>

adriangb · 2023-07-25T07:32:24Z

Maybe enough of an edge case to ignore, but it might be nice to improve the way you can use a self-referential JSON encoder.
I guess maybe we can postpone this as a feature request for the future.

Yeah I don't think this worked in v1 so let's table it.

samuelcolvin

otherwise LGTM. Do we need to add, alter or remove something in the migration guide?

pydantic/_internal/_generate_schema.py

pydantic/_internal/_config.py

Co-authored-by: Samuel Colvin <s@muelcolvin.com>

samuelcolvin · 2023-07-25T09:05:35Z

pydantic/_internal/_generate_schema.py

+            f'`json_encoders` is deprecated. See https://docs.pydantic.dev/{VERSION}/usage/serialization/#custom-serializers for alternatives',
+            PydanticDeprecatedSince20,


this is incorrect, it should only be the major and minor versions.

Maybe the solution is to have a proper warnings page like we have for user and validation errors. I can work on this today.

Fixing this should not block release.

It actually redirects, e.g. https://docs.pydantic.dev/2.0.2/usage/serialization/#custom-serializers works

only for specific versions see #6527.

I'll provide a general fix.

Ah I see. Okay then we can just strip the patch version.

adriangb · 2023-07-25T10:25:41Z

please review

samuelcolvin

LGTM.

adriangb commented Jul 24, 2023

View reviewed changes

tests/mypy/outputs/1.1.1/mypy-plugin_ini/pydantic_settings.py Outdated Show resolved Hide resolved

adriangb and others added 5 commits July 24, 2023 11:21

Add support for json_encoders

b104956

Add test

df2a478

Fix mypy test

c469d3e

Update MyPy output

34355a4

Revert mypy output update

2c900f2

hramezani force-pushed the json-encoders branch 2 times, most recently from 114a4c6 to a6c4e3b Compare July 24, 2023 09:43

Add missed mypy output

a432267

hramezani force-pushed the json-encoders branch from a6c4e3b to a432267 Compare July 24, 2023 09:53

pydantic-hooky bot added the ready for review label Jul 24, 2023

pydantic-hooky bot assigned dmontagu Jul 24, 2023

adriangb commented Jul 24, 2023

View reviewed changes

dmontagu added 2 commits July 24, 2023 10:48

Fix issue with expression-type-contains-any

917411c

Undo changes to mypy tests

8d0d38a

dmontagu reviewed Jul 24, 2023

View reviewed changes

pydantic/_internal/_generate_schema.py Outdated Show resolved Hide resolved

dmontagu approved these changes Jul 24, 2023

View reviewed changes

dmontagu reviewed Jul 24, 2023

View reviewed changes

pydantic/_internal/_generate_schema.py Outdated Show resolved Hide resolved

dmontagu reviewed Jul 24, 2023

View reviewed changes

pydantic/_internal/_generate_schema.py Outdated Show resolved Hide resolved

adriangb and others added 3 commits July 25, 2023 02:30

Update pydantic/_internal/_generate_schema.py

4e6b800

Co-authored-by: David Montague <35119617+dmontagu@users.noreply.github.com>

Update pydantic/_internal/_generate_schema.py

5603305

Co-authored-by: David Montague <35119617+dmontagu@users.noreply.github.com>

Update pydantic/_internal/_generate_schema.py

735a1dc

Co-authored-by: David Montague <35119617+dmontagu@users.noreply.github.com>

adriangb added 3 commits July 25, 2023 09:33

Remove unused path

b0c3121

Add warning

fdcd9ad

Move imports

54c3f08

samuelcolvin added this to the 2.0.X bugfixes milestone Jul 25, 2023

samuelcolvin reviewed Jul 25, 2023

View reviewed changes

pydantic/_internal/_generate_schema.py Outdated Show resolved Hide resolved

pydantic/_internal/_generate_schema.py Outdated Show resolved Hide resolved

pydantic/_internal/_generate_schema.py Outdated Show resolved Hide resolved

pydantic/_internal/_config.py Show resolved Hide resolved

adriangb and others added 3 commits July 25, 2023 04:04

Update pydantic/_internal/_generate_schema.py

75b52c0

Co-authored-by: Samuel Colvin <s@muelcolvin.com>

Update pydantic/_internal/_generate_schema.py

03f9fca

Co-authored-by: Samuel Colvin <s@muelcolvin.com>

Update pydantic/_internal/_generate_schema.py

8b74182

Co-authored-by: Samuel Colvin <s@muelcolvin.com>

samuelcolvin requested changes Jul 25, 2023

View reviewed changes

pydantic-hooky bot added awaiting author revision awaiting changes from the PR author and removed ready for review labels Jul 25, 2023

pydantic-hooky bot assigned adriangb and unassigned dmontagu Jul 25, 2023

Add when used

efd45b5

samuelcolvin changed the title ~~Add support for json_encoders~~ Add back support for json_encoders Jul 25, 2023

adriangb added 3 commits July 25, 2023 11:07

Add when used

067934d

Add docstring

cc1f975

fix tests

2e756f4

adriangb enabled auto-merge (squash) July 25, 2023 09:55

pydantic-hooky bot added ready for review and removed awaiting author revision awaiting changes from the PR author labels Jul 25, 2023

pydantic-hooky bot assigned dmontagu and unassigned adriangb Jul 25, 2023

samuelcolvin approved these changes Jul 25, 2023

View reviewed changes

adriangb merged commit 413fea3 into main Jul 25, 2023

adriangb deleted the json-encoders branch July 25, 2023 10:58

camillol mentioned this pull request Apr 7, 2024

json_encoders is back pydantic/bump-pydantic#162

Open

		if 'ref' in schema:
		return core_schema.definition_reference_schema(schema_ref=schema['ref'], serialization=serialization) # type: ignore

		f'`json_encoders` is deprecated. See https://docs.pydantic.dev/{VERSION}/usage/serialization/#custom-serializers for alternatives',
		PydanticDeprecatedSince20,

Uh oh!

Add back support for json_encoders #6811

Add back support for json_encoders #6811

Uh oh!

Conversation

adriangb commented Jul 23, 2023 • edited by pydantic-hooky bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloudflare-workers-and-pages bot commented Jul 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Pages

Uh oh!

Uh oh!

adriangb commented Jul 24, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmontagu commented Jul 24, 2023

Uh oh!

Uh oh!

dmontagu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

adriangb commented Jul 25, 2023

Uh oh!

samuelcolvin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adriangb commented Jul 25, 2023

Uh oh!

samuelcolvin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Add back support for `json_encoders` #6811

Add back support for `json_encoders` #6811

adriangb commented Jul 23, 2023 •

edited by pydantic-hooky bot

Loading

cloudflare-workers-and-pages bot commented Jul 23, 2023 •

edited

Loading