support custom root types #628

koxudaxi · 2019-06-29T19:26:02Z

Change Summary

support custom root types that support a model that does not have any field exclude __root__.

example:

from typing import List
import json
from pydantic import BaseModel
from pydantic.schema import schema

class Pets(BaseModel):
    __root__: List[str]

print(Pets(__root__=['dog', 'cat']))
# > Pets __root__=['dog', 'cat']

print(Pets(*['dog', 'cat']))
# > Pets __root__=['dog', 'cat']

print(Pets(*['dog', 'cat']).schema())
# > {'title': 'Pets', 'type': 'array', 'items': {'type': 'string'}}

pets_schema = schema([Pets])
print(json.dumps(pets_schema, indent=2))

# {
#  "definitions": {
#    "Pets": {
#      "title": "Pets",
#      "type": "array",
#      ...

Related issue number

#507

Checklist

Unit tests for the changes exist
Tests pass on CI and coverage remains at 100%
Documentation reflects the changes where applicable
HISTORY.rst has been updated
- this pull request number #628
- include github username @koxudaxi

codecov · 2019-06-29T19:27:41Z

Codecov Report

Merging #628 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master   #628   +/-   ##
=====================================
  Coverage     100%   100%           
=====================================
  Files          15     15           
  Lines        2634   2651   +17     
  Branches      518    524    +6     
=====================================
+ Hits         2634   2651   +17

samuelcolvin

Otherwise the idea looks good, I'll review more once it's complete.

samuelcolvin · 2019-07-01T08:48:25Z

docs/index.rst

@@ -466,6 +466,18 @@ Outputs:
 .. literalinclude:: examples/schema3.json


+Pydantic models can be set custom root types to support a model that does not have any field exclude ``__root__``.


Can you make this a separate section of the docs. This is not just about schema generation.

I see. I have moved the content to around types sections.

samuelcolvin · 2019-07-01T08:48:25Z

pydantic/main.py

@@ -186,7 +186,7 @@ def __new__(mcs, name, bases, namespace):
            for ann_name, ann_type in annotations.items():
                if is_classvar(ann_type):
                    class_vars.add(ann_name)
-                elif not ann_name.startswith('_') and ann_name not in namespace:
+                elif (not ann_name.startswith('_') or '__root__' == ann_name) and ann_name not in namespace:


we do this at least three times, can we have a function for it.

I have created is_valid_field function for checking a field name.

samuelcolvin · 2019-07-01T08:48:25Z

pydantic/main.py

        _json_encoder: Callable[[Any], Any] = lambda x: x
        _schema_cache: 'DictAny' = {}

    Config = BaseConfig
    __slots__ = ('__values__', '__fields_set__')

-    def __init__(self, **data: Any) -> None:
+    def __init__(self, *__root__: Any, **data: Any) -> None:


Humm, I've very unwilling to change the signature of __init__.

Would it be possible to accept __root__ directly to parse_obj only?

I have reverted the __init__ method to master.
And I add a __root__ argument in parse_obj.
However, We can pass __root__ object to __init__ as a keyword argument.

Model(__root__='a') # or Model.parse_obj(__root__='a')

add a keyword argument of "__root__" in parse_obj fix documents create a method for cheking valid field name

samuelcolvin

otherwise looking good.

docs/index.rst

samuelcolvin · 2019-07-02T10:22:03Z

pydantic/main.py

@@ -233,6 +241,7 @@ class BaseModel(metaclass=MetaModel):
        __fields__: Dict[str, Field] = {}
        __validators__: Dict[str, AnyCallable] = {}
        __config__: Type[BaseConfig] = BaseConfig
+        __root__: List[Any] = []


I thought __root__ doesn't have to be a list?

Sorry, it's my mistake. I have changed List[Any] to Any.

samuelcolvin · 2019-07-02T10:22:03Z

pydantic/main.py

@@ -211,6 +217,8 @@ def __new__(mcs, name, bases, namespace):
                        config=config,
                    )

+        if '__root__' in fields and len(fields) > 1:
+            raise ValueError('__root__ cannot be mixed with other fields')


add _custom_root_type: bool to new_namespace below.

I have added it.

samuelcolvin · 2019-07-02T10:22:03Z

pydantic/main.py

@@ -324,9 +334,17 @@ def json(
        )

    @classmethod
-    def parse_obj(cls: Type['Model'], obj: Mapping[Any, Any]) -> 'Model':
+    def parse_obj(
+        cls: Type['Model'], obj: Optional[Mapping[Any, Any]] = None, __root__: Optional[Any] = None


simplify this to

def parse_obj(cls: Type['Model'], obj: Any) -> 'Model'

ok, I just changed it.

samuelcolvin · 2019-07-02T10:22:03Z

pydantic/main.py

        if not isinstance(obj, dict):
            try:
+                if obj is None:


change this to:

if not instanstance(obj, dict): if cls._custom_root_type: obj = {'__root__': obj} else: ...

I think we should also raise an error if the type of __root__ is a dict.

I have changed the condition and added to raise an exception.

samuelcolvin · 2019-07-02T10:22:03Z

tests/test_main.py

+    class MyModel(BaseModel):
+        __root__: str
+
+    m = MyModel(__root__='a')


please test with parse_obj too.

I had made tests for parse_obj in tests/tset_parse.py
Would you please check it?
https://github.com/samuelcolvin/pydantic/pull/628/files#diff-7b34c8bd665994bbdcf056c03622d4a8R45

Co-Authored-By: Samuel Colvin <samcolvin@gmail.com>

samuelcolvin

sorry, reviewed this days ago and somehow forgot to press submit.

docs/index.rst

samuelcolvin · 2019-07-05T16:02:21Z

pydantic/main.py

+    def parse_obj(cls: Type['Model'], obj: Any) -> 'Model':
+        if isinstance(obj, dict):
+            if cls._custom_root_type:
+                raise TypeError('custom root type cannot allow dict')


this test should be moved to model creation, not validation.

OK, I have moved it to __new__ in MetaModel.
Also, I change the validation type from dict to Mapping. If it's wrong, then please tell me correct.

… custom_root_types

Co-Authored-By: Samuel Colvin <samcolvin@gmail.com>

koxudaxi · 2019-07-06T06:49:45Z

sorry, reviewed this days ago and somehow forgot to press submit.

No problem, Thank you for reviewing my code.

mdsrosa · 2020-05-21T14:19:35Z

Hey @koxudaxi @samuelcolvin. I have a question that is related to this PR.

Given the following use case:

from typing import List
from datetime import datetime
from pydantic import BaseModel

class LotteryResults(BaseModel):
    __root__: List[int]

class Lottery(BaseModel):
    when: datetime
    numbers: LotteryResults

numbers = LotteryResults.parse_obj([10, 20, 30, 12 , 34]
lottery = Lottery(when=datetime.now(), numbers=numbers)

Running that code as is, when I call the .json() on lottery I get the following result:

In [7]: lottery.json()
Out[7]: '{"when": "2020-05-21T14:09:14.652009", "numbers": {"__root__": [10, 20, 30, 12, 34]}}'

Is it possible to have the LotterResults return as the list of numbers? That would be much easier to consume without any logic that depends on a __root__ field existing. I would like to get the following result instead:

In [7]: lottery.json()
Out[7]: '{"when": "2020-05-21T14:09:14.652009", "numbers": [10, 20, 30, 12, 34]}'

By the way, Pydantic is awesome!

koxudaxi · 2020-05-21T15:47:50Z

@mdsrosa

I paste a workaround.
You can use the Custom JSON serializer.
https://pydantic-docs.helpmanual.io/usage/exporting_models/#custom-json-deserialisation

Btw, I think you should create a new issue for this question. 📝

import json
from typing import Any
from typing import List
from datetime import datetime
from pydantic import BaseModel
from pydantic.class_validators import ROOT_KEY


def unpack_custom_root(value: Any):
    if isinstance(value, dict):
        if ROOT_KEY in value:
            return value[ROOT_KEY]
        return {k: unpack_custom_root(v) for k, v in value.items()}
    return value


def custom_root_json_dumps(value, *, default):
    return json.dumps(unpack_custom_root(value), default=default)


class LotteryResults(BaseModel):
    __root__: List[int]


class Lottery(BaseModel):
    when: datetime
    numbers: LotteryResults

    class Config:
        json_dumps = custom_root_json_dumps


numbers = LotteryResults.parse_obj([10, 20, 30, 12, 34])
lottery = Lottery(when=datetime.now(), numbers=numbers)

print(lottery.json())
#  {"when": "2020-05-22T00:32:10.811364", "numbers": [10, 20, 30, 12, 34]}

koxudaxi added 2 commits Jun 29, 2019

support custom root types

f718e62

update HISTORY.rst for costom root types

55364bb

fix invalid arguments case

8922961

samuelcolvin reviewed Jul 1, 2019

View changes

remove the keyword argument of "__root__" in __init__

1ae8319

add a keyword argument of "__root__" in parse_obj fix documents create a method for cheking valid field name

samuelcolvin reviewed Jul 2, 2019

View changes

koxudaxi and others added 4 commits Jul 2, 2019

fix parse_obj

e68f944

Merge remote-tracking branch 'upstream/master' into custom_root_types

f06dd99

Update docs/index.rst

f4fe9f4

Co-Authored-By: Samuel Colvin <samcolvin@gmail.com>

Apply suggestions from code review

5dfd71c

Co-Authored-By: Samuel Colvin <samcolvin@gmail.com>

samuelcolvin reviewed Jul 5, 2019

View changes

koxudaxi and others added 3 commits Jul 6, 2019

add type cheking for custom root type

028ef49

Merge branch 'custom_root_types' of github.com:koxudaxi/pydantic into…

301f34e

… custom_root_types

Update docs/index.rst

d8d66cf

Co-Authored-By: Samuel Colvin <samcolvin@gmail.com>

koxudaxi mentioned this pull request Jul 6, 2019

Generate pydantic model python code from jsonschema #643

Closed

Update index.rst

288d4ca

samuelcolvin merged commit e4b285a into pydantic:master Jul 6, 2019
5 of 7 checks passed

dmontagu mentioned this pull request Jul 10, 2019

[QUESTION] One or more objects in request body? tiangolo/fastapi#379

Closed

spolloni mentioned this pull request Jan 27, 2021

Custom root types #507

Closed

samuelcolvin mentioned this pull request Dec 19, 2021

Remove __root__ from BaseModel typing-only attrs #3540

Merged

5 tasks

support custom root types #628

support custom root types #628

koxudaxi commented Jun 29, 2019 •

edited

codecov bot commented Jun 29, 2019 •

edited

samuelcolvin left a comment

samuelcolvin Jul 1, 2019

koxudaxi Jul 1, 2019

samuelcolvin Jul 1, 2019

koxudaxi Jul 1, 2019

samuelcolvin Jul 1, 2019

koxudaxi Jul 1, 2019

samuelcolvin left a comment

samuelcolvin Jul 2, 2019

koxudaxi Jul 2, 2019

samuelcolvin Jul 2, 2019

koxudaxi Jul 2, 2019

samuelcolvin Jul 2, 2019

koxudaxi Jul 2, 2019

samuelcolvin Jul 2, 2019

samuelcolvin Jul 2, 2019

koxudaxi Jul 2, 2019

samuelcolvin Jul 2, 2019

koxudaxi Jul 2, 2019

samuelcolvin left a comment

samuelcolvin Jul 5, 2019

koxudaxi Jul 6, 2019

koxudaxi commented Jul 6, 2019

mdsrosa commented May 21, 2020

koxudaxi commented May 21, 2020 •

edited

		@@ -466,6 +466,18 @@ Outputs:
		.. literalinclude:: examples/schema3.json


		Pydantic models can be set custom root types to support a model that does not have any field exclude ``__root__``.

support custom root types #628

support custom root types #628

Conversation

koxudaxi commented Jun 29, 2019 • edited

Change Summary

Related issue number

Checklist

codecov bot commented Jun 29, 2019 • edited

Codecov Report

samuelcolvin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samuelcolvin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samuelcolvin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koxudaxi commented Jul 6, 2019

mdsrosa commented May 21, 2020

koxudaxi commented May 21, 2020 • edited

koxudaxi commented Jun 29, 2019 •

edited

codecov bot commented Jun 29, 2019 •

edited

koxudaxi commented May 21, 2020 •

edited