Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support custom root types #628

Merged
merged 12 commits into from Jul 6, 2019
Merged

Conversation

koxudaxi
Copy link
Contributor

@koxudaxi koxudaxi commented Jun 29, 2019

Change Summary

support custom root types that support a model that does not have any field exclude __root__.

example:

from typing import List
import json
from pydantic import BaseModel
from pydantic.schema import schema

class Pets(BaseModel):
    __root__: List[str]

print(Pets(__root__=['dog', 'cat']))
# > Pets __root__=['dog', 'cat']

print(Pets(*['dog', 'cat']))
# > Pets __root__=['dog', 'cat']

print(Pets(*['dog', 'cat']).schema())
# > {'title': 'Pets', 'type': 'array', 'items': {'type': 'string'}}

pets_schema = schema([Pets])
print(json.dumps(pets_schema, indent=2))

# {
#  "definitions": {
#    "Pets": {
#      "title": "Pets",
#      "type": "array",
#      ...

Related issue number

#507

Checklist

  • Unit tests for the changes exist
  • Tests pass on CI and coverage remains at 100%
  • Documentation reflects the changes where applicable
  • HISTORY.rst has been updated
    • this pull request number #628
    • include github username @koxudaxi

@codecov
Copy link

@codecov codecov bot commented Jun 29, 2019

Codecov Report

Merging #628 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master   #628   +/-   ##
=====================================
  Coverage     100%   100%           
=====================================
  Files          15     15           
  Lines        2634   2651   +17     
  Branches      518    524    +6     
=====================================
+ Hits         2634   2651   +17

Copy link
Collaborator

@samuelcolvin samuelcolvin left a comment

Otherwise the idea looks good, I'll review more once it's complete.

docs/index.rst Outdated
@@ -466,6 +466,18 @@ Outputs:
.. literalinclude:: examples/schema3.json


Pydantic models can be set custom root types to support a model that does not have any field exclude ``__root__``.
Copy link
Collaborator

@samuelcolvin samuelcolvin Jul 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make this a separate section of the docs. This is not just about schema generation.

Copy link
Contributor Author

@koxudaxi koxudaxi Jul 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. I have moved the content to around types sections.

pydantic/main.py Outdated
@@ -186,7 +186,7 @@ def __new__(mcs, name, bases, namespace):
for ann_name, ann_type in annotations.items():
if is_classvar(ann_type):
class_vars.add(ann_name)
elif not ann_name.startswith('_') and ann_name not in namespace:
elif (not ann_name.startswith('_') or '__root__' == ann_name) and ann_name not in namespace:
Copy link
Collaborator

@samuelcolvin samuelcolvin Jul 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do this at least three times, can we have a function for it.

Copy link
Contributor Author

@koxudaxi koxudaxi Jul 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have created is_valid_field function for checking a field name.

pydantic/main.py Outdated
_json_encoder: Callable[[Any], Any] = lambda x: x
_schema_cache: 'DictAny' = {}

Config = BaseConfig
__slots__ = ('__values__', '__fields_set__')

def __init__(self, **data: Any) -> None:
def __init__(self, *__root__: Any, **data: Any) -> None:
Copy link
Collaborator

@samuelcolvin samuelcolvin Jul 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Humm, I've very unwilling to change the signature of __init__.

Would it be possible to accept __root__ directly to parse_obj only?

Copy link
Contributor Author

@koxudaxi koxudaxi Jul 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reverted the __init__ method to master.
And I add a __root__ argument in parse_obj.
However, We can pass __root__ object to __init__ as a keyword argument.

Model(__root__='a')
# or
Model.parse_obj(__root__='a')

add a keyword argument of "__root__" in parse_obj
fix documents
create a method for cheking valid field name
Copy link
Collaborator

@samuelcolvin samuelcolvin left a comment

otherwise looking good.

docs/index.rst Outdated Show resolved Hide resolved
docs/index.rst Outdated Show resolved Hide resolved
docs/index.rst Outdated Show resolved Hide resolved
pydantic/main.py Outdated
@@ -233,6 +241,7 @@ class BaseModel(metaclass=MetaModel):
__fields__: Dict[str, Field] = {}
__validators__: Dict[str, AnyCallable] = {}
__config__: Type[BaseConfig] = BaseConfig
__root__: List[Any] = []
Copy link
Collaborator

@samuelcolvin samuelcolvin Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought __root__ doesn't have to be a list?

Copy link
Contributor Author

@koxudaxi koxudaxi Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, it's my mistake. I have changed List[Any] to Any.

pydantic/main.py Outdated
@@ -211,6 +217,8 @@ def __new__(mcs, name, bases, namespace):
config=config,
)

if '__root__' in fields and len(fields) > 1:
raise ValueError('__root__ cannot be mixed with other fields')
Copy link
Collaborator

@samuelcolvin samuelcolvin Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add _custom_root_type: bool to new_namespace below.

Copy link
Contributor Author

@koxudaxi koxudaxi Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added it.

pydantic/main.py Outdated
@@ -324,9 +334,17 @@ def json(
)

@classmethod
def parse_obj(cls: Type['Model'], obj: Mapping[Any, Any]) -> 'Model':
def parse_obj(
cls: Type['Model'], obj: Optional[Mapping[Any, Any]] = None, __root__: Optional[Any] = None
Copy link
Collaborator

@samuelcolvin samuelcolvin Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

simplify this to

def parse_obj(cls: Type['Model'], obj: Any) -> 'Model'

Copy link
Contributor Author

@koxudaxi koxudaxi Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I just changed it.

pydantic/main.py Outdated
if not isinstance(obj, dict):
try:
if obj is None:
Copy link
Collaborator

@samuelcolvin samuelcolvin Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change this to:

if not instanstance(obj, dict):
    if cls._custom_root_type:
        obj = {'__root__': obj}
    else:
        ...

Copy link
Collaborator

@samuelcolvin samuelcolvin Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should also raise an error if the type of __root__ is a dict.

Copy link
Contributor Author

@koxudaxi koxudaxi Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have changed the condition and added to raise an exception.

class MyModel(BaseModel):
__root__: str

m = MyModel(__root__='a')
Copy link
Collaborator

@samuelcolvin samuelcolvin Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please test with parse_obj too.

Copy link
Contributor Author

@koxudaxi koxudaxi Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had made tests for parse_obj in tests/tset_parse.py
Would you please check it?
https://github.com/samuelcolvin/pydantic/pull/628/files#diff-7b34c8bd665994bbdcf056c03622d4a8R45

koxudaxi and others added 4 commits Jul 2, 2019
Co-Authored-By: Samuel Colvin <samcolvin@gmail.com>
Co-Authored-By: Samuel Colvin <samcolvin@gmail.com>
Copy link
Collaborator

@samuelcolvin samuelcolvin left a comment

sorry, reviewed this days ago and somehow forgot to press submit.

docs/index.rst Outdated Show resolved Hide resolved
pydantic/main.py Outdated
def parse_obj(cls: Type['Model'], obj: Any) -> 'Model':
if isinstance(obj, dict):
if cls._custom_root_type:
raise TypeError('custom root type cannot allow dict')
Copy link
Collaborator

@samuelcolvin samuelcolvin Jul 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test should be moved to model creation, not validation.

Copy link
Contributor Author

@koxudaxi koxudaxi Jul 6, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I have moved it to __new__ in MetaModel.
Also, I change the validation type from dict to Mapping. If it's wrong, then please tell me correct.

@koxudaxi
Copy link
Contributor Author

@koxudaxi koxudaxi commented Jul 6, 2019

sorry, reviewed this days ago and somehow forgot to press submit.

No problem, Thank you for reviewing my code.

@samuelcolvin samuelcolvin merged commit e4b285a into pydantic:master Jul 6, 2019
5 of 7 checks passed
@mdsrosa
Copy link

@mdsrosa mdsrosa commented May 21, 2020

Hey @koxudaxi @samuelcolvin. I have a question that is related to this PR.

Given the following use case:

from typing import List
from datetime import datetime
from pydantic import BaseModel

class LotteryResults(BaseModel):
    __root__: List[int]

class Lottery(BaseModel):
    when: datetime
    numbers: LotteryResults

numbers = LotteryResults.parse_obj([10, 20, 30, 12 , 34]
lottery = Lottery(when=datetime.now(), numbers=numbers)

Running that code as is, when I call the .json() on lottery I get the following result:

In [7]: lottery.json()
Out[7]: '{"when": "2020-05-21T14:09:14.652009", "numbers": {"__root__": [10, 20, 30, 12, 34]}}'

Is it possible to have the LotterResults return as the list of numbers? That would be much easier to consume without any logic that depends on a __root__ field existing. I would like to get the following result instead:

In [7]: lottery.json()
Out[7]: '{"when": "2020-05-21T14:09:14.652009", "numbers": [10, 20, 30, 12, 34]}'

By the way, Pydantic is awesome!

@koxudaxi
Copy link
Contributor Author

@koxudaxi koxudaxi commented May 21, 2020

@mdsrosa

I paste a workaround.
You can use the Custom JSON serializer.
https://pydantic-docs.helpmanual.io/usage/exporting_models/#custom-json-deserialisation

Btw, I think you should create a new issue for this question. 📝

import json
from typing import Any
from typing import List
from datetime import datetime
from pydantic import BaseModel
from pydantic.class_validators import ROOT_KEY


def unpack_custom_root(value: Any):
    if isinstance(value, dict):
        if ROOT_KEY in value:
            return value[ROOT_KEY]
        return {k: unpack_custom_root(v) for k, v in value.items()}
    return value


def custom_root_json_dumps(value, *, default):
    return json.dumps(unpack_custom_root(value), default=default)


class LotteryResults(BaseModel):
    __root__: List[int]


class Lottery(BaseModel):
    when: datetime
    numbers: LotteryResults

    class Config:
        json_dumps = custom_root_json_dumps


numbers = LotteryResults.parse_obj([10, 20, 30, 12, 34])
lottery = Lottery(when=datetime.now(), numbers=numbers)

print(lottery.json())
#  {"when": "2020-05-22T00:32:10.811364", "numbers": [10, 20, 30, 12, 34]}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants