Make it so callable JSON schema extra works #6798

dmontagu · 2023-07-21T20:28:55Z

Note that as mentioned here, the better solution for FastAPI is to use a custom GenerateJsonSchema that generates the JSON schema differently for Optional query params, etc.. But this at least makes it possible to express what you want on the pydantic side via:

from typing import Annotated

from pydantic import BaseModel, Field
from pydantic.json_schema import SkipJsonSchema


class Model(BaseModel):
    x: int | SkipJsonSchema[None] = Field(default=None, json_schema_extra=lambda x: x.pop('default'))

print(Model.model_json_schema())
#> {'properties': {'x': {'title': 'X', 'type': 'integer'}}, 'title': 'Model', 'type': 'object'}

Selected Reviewer: @samuelcolvin

cloudflare-pages · 2023-07-21T20:30:46Z

Deploying with Cloudflare Pages

Latest commit:	`6745019`
Status:	✅ Deploy successful!
Preview URL:	https://19f679d4.pydantic-docs2.pages.dev
Branch Preview URL:	https://field-json-schema-extra-call.pydantic-docs2.pages.dev

View logs

dmontagu · 2023-07-21T20:58:01Z

please review

dmontagu · 2023-07-21T20:59:20Z

pydantic/fields.py

        for field_info in field_infos:
            new_kwargs.update(field_info._attributes_set)
+            for x in field_info.metadata:
+                if not isinstance(x, FieldInfo):
+                    metadata[type(x)] = x


I ran into a case where this was necessary to get proper merging of FieldInfos that use gt, lt, etc.

This stuff is getting really annoying. @samuelcolvin not having to do this stuff is one of the big advantages of the Output enum thing we were discussing the other day.

dmontagu · 2023-07-21T21:00:32Z

pydantic/_internal/_generate_schema.py

@@ -1480,6 +1462,37 @@ def json_schema_update_func(
            return maybe_updated_schema
        return original_schema

+    def apply_single_annotation_json_schema(


I had to break this out as a separate function in order to ensure the callable JSON schema ran after all modifications to the core schema that influence the json schema are performed. Otherwise, for example, you can't remove the default in the callable.

dmontagu · 2023-07-21T21:01:26Z

pydantic/_internal/_generate_schema.py

@@ -1493,6 +1506,7 @@ def _get_wrapped_inner_schema(
        def new_handler(source: Any) -> core_schema.CoreSchema:
            schema = metadata_get_schema(source, get_inner_schema)
            schema = self.apply_single_annotation(schema, annotation)
+            schema = self.apply_single_annotation_json_schema(schema, annotation)


As you can see here, everything related to JSON schema now runs after everything not related to JSON schema.

No typo as far as I can tell — the self.apply_single_annotation does everything not related to JSON schema, then after that, self.apply_single_annotation_json_schema does everything related to json schema. I think that's what I said in the comment above?

dmontagu · 2023-07-21T21:04:14Z

tests/test_json_schema.py

+        d: Annotated[int, Field(default=4), Field(json_schema_extra=pop_default)]
+        # TODO: it doesn't work properly to have both annotation and assigned value of FieldInfo:
+        # e: Annotated[int, Field(json_schema_extra=pop_default)] = Field(default=5)
+        # f: Annotated[int, Field(default=6)] = Field(json_schema_extra=pop_default)


I think it would be possible to make this and the previous case work (i.e., e and f), but I tried and it ended up being a significantly bigger change and I wasn't able to see it all the way through and get all tests passing. So rather than wasting time on it for now, I just added a note here. Happy to create an issue for this if there isn't one already. (CC @adriangb)

While it might be tempting to try to make this an error, for various reasons, I think it would be preferable if assigning a FieldInfo was equivalent to adding it as the last annotation on the field (and that does work, as you can see in the handling of attributes c and d).

I do think we should make it work at some point. Is it not working now? I thought it was.

It works for most things, it just doesn't work for json_schema_extra and default specifically, annoyingly.

Also, even on main, this doesn't work:

from pydantic import BaseModel, Field from typing_extensions import Annotated class Model(BaseModel): a: Annotated[int, Field(default=1)] = Field() Model()

pydantic_core._pydantic_core.ValidationError: 1 validation error for Model a Field required [type=missing, input_value={}, input_type=dict]

I was able to fix this through an appropriate change in FieldInfo.from_annotated_attribute 💪

adriangb · 2023-07-22T10:29:04Z

Can’t this go in

pydantic/pydantic/json_schema.py

Line 2174 in 6ef959f

json_schema: JsonSchemaValue | None

? To avoid making Field even more complex.

samuelcolvin

Looks fine, I'm a bit lost on a few things but if you're confident, let's merge.

samuelcolvin · 2023-07-24T19:36:51Z

pydantic/_internal/_generate_schema.py

+        if isinstance(field_info.json_schema_extra, dict):
+            json_schema_updates.update(field_info.json_schema_extra)


why don't we call field_info.json_schema_extra if it's a function here?

okay so, in the update func (def json_schema_update_func below), we have:

json_schema = {**handler(schema), **json_schema_updates}

and handler(schema) may result in a different schema than just the updates; json_schema_updates is not the whole json schema. In particular, getting stuff like "type": "object" (or even other types for RootModel subclasses), you probably want that to be accessible in the json_schema_extra call, which is why that needs to happen after the call to handler in the json_schema_update_func below.

samuelcolvin · 2023-07-24T19:38:09Z

pydantic/_internal/_generate_schema.py

+            if callable(field_info.json_schema_extra):
+                field_info.json_schema_extra(json_schema)


again, I'm a bit lost why we don't update do json_schema.update(field_info.json_schema_extra) here if it's a dict?

If you sure these are fine, great.

I think we can clean this up, I'll push a commit doing that shortly

…ds and dataclasses

Make it so callable JSON schema extra works

7798274

dmontagu mentioned this pull request Jul 21, 2023

✨ Add SkipJsonSchema annotation #6653

Merged

dmontagu added 2 commits July 21, 2023 14:38

Merge branch 'main' into field-json-schema-extra-callable

d94a494

Add test for removing default from JSON schema

bfd0722

dmontagu mentioned this pull request Jul 21, 2023

anyOf causing problems with Swagger/ReDoc on primitive types #6647

Closed

1 task

pydantic-hooky bot added the ready for review label Jul 21, 2023

pydantic-hooky bot assigned samuelcolvin Jul 21, 2023

dmontagu commented Jul 21, 2023

View reviewed changes

samuelcolvin approved these changes Jul 24, 2023

View reviewed changes

Simplify json_schema_extra application, and fix some issues with fiel…

6745019

…ds and dataclasses

dmontagu merged commit 2181d12 into main Jul 24, 2023
48 checks passed

dmontagu deleted the field-json-schema-extra-callable branch July 24, 2023 20:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make it so callable JSON schema extra works #6798

Make it so callable JSON schema extra works #6798

dmontagu commented Jul 21, 2023 •

edited

cloudflare-pages bot commented Jul 21, 2023 •

edited

dmontagu commented Jul 21, 2023

dmontagu Jul 21, 2023

adriangb Jul 22, 2023

dmontagu Jul 21, 2023 •

edited

dmontagu Jul 21, 2023

adriangb Jul 22, 2023

dmontagu Jul 24, 2023

dmontagu Jul 21, 2023 •

edited

adriangb Jul 22, 2023

dmontagu Jul 24, 2023

dmontagu Jul 24, 2023

dmontagu Jul 24, 2023

adriangb commented Jul 22, 2023

samuelcolvin left a comment

samuelcolvin Jul 24, 2023

dmontagu Jul 24, 2023 •

edited

samuelcolvin Jul 24, 2023

dmontagu Jul 24, 2023

		if isinstance(field_info.json_schema_extra, dict):
		json_schema_updates.update(field_info.json_schema_extra)

		if callable(field_info.json_schema_extra):
		field_info.json_schema_extra(json_schema)

Make it so callable JSON schema extra works #6798

Make it so callable JSON schema extra works #6798

Conversation

dmontagu commented Jul 21, 2023 • edited

cloudflare-pages bot commented Jul 21, 2023 • edited

Deploying with Cloudflare Pages

dmontagu commented Jul 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmontagu Jul 21, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmontagu Jul 21, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriangb commented Jul 22, 2023

samuelcolvin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmontagu Jul 24, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmontagu commented Jul 21, 2023 •

edited

cloudflare-pages bot commented Jul 21, 2023 •

edited

dmontagu Jul 21, 2023 •

edited

dmontagu Jul 21, 2023 •

edited

dmontagu Jul 24, 2023 •

edited