Validators for dataclasses #454

primal100 · 2019-04-02T18:10:59Z

Change Summary

Custom validator methods work with dataclasses

Related issue number

#415: Validator on dataclasses

Also remove a limitation referred to in the docs

Checklist

Unit tests for the changes exist
Tests pass on CI and coverage remains at 100% 99.3% some tests skipped
Documentation reflects the changes where applicable
HISTORY.rst has been updated
- if this is the first change since a release, please add a new section
- include the issue number or this pull request number #<number>
- include your github username @<whomever>

There are probably a few decisions to be made with regard to the exact implementation. Maybe some of this can be improved?

The validators are discovered using a new utils function, gather_validators, which looks for validator methods in the dataclass and parent classes
The create_model function now accepts a validators keyword argument to pass in a mapping of method names and methods. These are then added to the namespace by the create_model function.
The validate_model now accepts an additional cls keyword argument. The pydantic_model already gets passed in from the dataclass, so the additional cls argument is also added so that the correct class (the dataclass) is passed to the validator classmethods.
The example datetime.py was renamed to datetime_example.py as the examples which import datetime were failing to run due to the name clash

Ronserruya · 2019-04-04T12:16:07Z

This would be great! Found this library today and couldn't understand why validators didn't work for dataclasses.

Was this written in the docs and I missed this?

primal100 · 2019-04-04T12:47:54Z

This would be great! Found this library today and couldn't understand why validators didn't work for dataclasses.

Was this written in the docs and I missed this?

The limitation is referred to in the docs:

"Currently validators don’t work with dataclasses, if it’s something you want please create an issue on github".

samuelcolvin · 2019-04-04T13:05:48Z

thanks @primal100, looks broadly good. I'll go through it in detail when I get a chance... manic week this week.

samuelcolvin

Looking good but hard to fully review until the merge errors are resoved.

If you're not clear how to resolve this, you might be better off doing reset --soft and create a new PR.

HISTORY.rst

samuelcolvin · 2019-04-08T17:48:45Z

pydantic/fields.py

+        values: Dict[str, Any],
+        *,
+        loc: 'LocType',
+        cls: Optional[Type[Union['BaseModel', 'DataclassType']]] = None,


perhaps you could use a named type Optional[Type[Union['BaseModel', 'DataclassType']] as it's reused in quite a few places.

Created a new ModelType in types.py is that ok?

samuelcolvin · 2019-04-08T17:49:32Z

pydantic/main.py

    model_name: str,
    *,
    __config__: Type[BaseConfig] = None,
    __base__: Type[BaseModel] = None,
    __module__: Optional[str] = None,
+    __validators__: Mapping[str, classmethod] = None,


can't we just use Dict here?

Changed main.py

samuelcolvin · 2019-04-08T17:51:52Z

pydantic/utils.py

+    return type(v) == type(ClassVar) and (sys.version_info < (3, 7) or getattr(v, '_name', None) == 'ClassVar')
+
+
+def is_classvar(ann_type: AnyType) -> bool:


again this seems to be messed up due to a rebasing problem.

Should be ok now

tests/test_validators_dataclass.py

samuelcolvin · 2019-04-08T17:57:15Z

pydantic/utils.py

+    return _check_classvar(ann_type) or _check_classvar(getattr(ann_type, '__origin__', None))
+
+
+def gather_validators(type_: AnyType) -> Dict[str, classmethod]:


this should be in class_validators, also can we use the same logic for models and dataclasses, without making it too complicated?

Moved to class_validators.py now.

The same logic for models (extract_validators function ) is still used by dataclasses when the __pydantic_model__ is created. The new gather_validators function just gets the validators in a format to where they can be added to the namespace in the create_model function. Then the same logic as for BaseModel is used when the Model is created.

Both extract_validators and gather_validators contain a check for the __validator_config attribute so there is a slight duplication there. One alternative would be to add the entire namespace (__dict__) of the dataclass to create_model (instead of just validators) and then let the BaseModel __new__ method filter the namespace for validators as normal. What do you think?

codecov · 2019-04-10T16:56:02Z

Codecov Report

Merging #454 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master   #454   +/-   ##
=====================================
  Coverage     100%   100%           
=====================================
  Files          14     14           
  Lines        2230   2238    +8     
  Branches      437    440    +3     
=====================================
+ Hits         2230   2238    +8

samuelcolvin

Looking good, just a few small things and questions.

samuelcolvin · 2019-04-11T14:59:12Z

pydantic/main.py

+    if __validators__:
+        try:
+            if any(not hasattr(v, '__validator_config') for k, v in __validators__.items()):
+                raise ConfigError('Validator methods must be decorated with @validator')


Is this really required?

We don't inspect all arguments to all functions, if we were doing that we might as well right rust.

I think better to remove, but if it's really required for some reason, it'll need tests.

Agreed, especially with type checking it's not needed. Removed these lines.

pydantic/main.py

samuelcolvin · 2019-04-11T15:02:37Z

pydantic/types.py

    from .utils import AnyCallable

    CallableGenerator = Generator[AnyCallable, None, None]
+    ModelType = Optional[Type[Union['BaseModel', 'DataclassType']]]


This name is deceptive, better to have

ModelOrDc = Type[Union['BaseModel', 'DataclassType']]

So the name is clear.

Done, but set to:

ModelOrDc = Optional[Type[Union['BaseModel', 'DataclassType']]]

Maybe better to create without Optional? And then just write Optional[ModelOrDc] where it is used? What do you think? I can easily make the change if required.

Yes please, then it's clear.

samuelcolvin · 2019-04-11T15:03:53Z

HISTORY.rst

@@ -14,6 +14,7 @@ v0.23 (2019-04-04)
 * Support specialized ``ClassVars``, #455 by @tyrylu
 * fix JSON serialization for ``ipaddress`` types, #333 by @pilosus
 * add ``SecretStr`` and ``SecretBytes`` types, #452 by @atheuz
+* Support validators in dataclasses, #454 by @primal100 


wrong place version is released, Please start a new section.

Added to latest section in scolvin:master

samuelcolvin · 2019-04-11T15:06:34Z

tests/test_validators_dataclass.py

+    class Child(Parent):
+        pass
+
+    assert Parent(a='this is foobar good').a == 'this is foobar good'


more concise to have rename check_a to change_a, and return v + 'changed', then you can have just these two lines.

Can you also add a test with a validator having the same name as in the parent, does it overwrite, call both, or raise an error?

Done. The child validator overwrites the parent(same behavior as BaseModel)

primal100 · 2019-04-11T21:15:01Z

Getting an error when running "make":

warning: check: Duplicate explicit target name: "@atheuz".

Also gives an error on the Docs build:

/opt/build/repo/docs/.TMP_HISTORY.rst:4:Duplicate explicit target name: "@atheuz".

However, I made a clone of the samuelcolvin:master branch just to check and see the same error when I run make

samuelcolvin · 2019-04-11T21:28:00Z

Getting an error when running "make":

warning: check: Duplicate explicit target name: "@atheuz".

Also gives an error on the Docs build:

/opt/build/repo/docs/.TMP_HISTORY.rst:4:Duplicate explicit target name: "@atheuz".

However, I made a clone of the samuelcolvin:master branch just to check and see the same error when I run make

My fault, I made a mistake when adding a line to HISTORY earlier, should be fixed now.

Fix with origin

samuelcolvin · 2019-04-11T22:14:06Z

awesome, thank you so much.

…antic#454)

samuelcolvin reviewed Apr 8, 2019

View reviewed changes

samuelcolvin mentioned this pull request Apr 8, 2019

Create model validators #467

Closed

4 tasks

samuelcolvin reviewed Apr 11, 2019

View reviewed changes

Paul Martin and others added 23 commits April 11, 2019 20:30

Added validators for dataclass

fd3ac79

Added dataclass validators

26ea3d4

Added dataclass validators

233254a

Updated docs for added validating to dataclass

6da6345

Updated docs for added validating to dataclass

9016dbd

Fixed line endings

29224d1

Set __validators__ type to Mapping instead of Dict

8ef9938

Update History

43021b2

Use __mro__ instead of __bases__ for gather_validators

4e0ff0e

Fix PR number

81ad64d

Fix issue.rst header underline

4e4dc18

Fix HISTORY.rst merge conflict

9e73f4b

Fix utils.py merge conflict

7df7176

fix utils.py

b13a70e

Rebase and other fixes

f5a07eb

Fix rebase and other issues

37e28ba

Change history

9a41736

Remove unnecessary lines in main.py

ecf40be

Rebase

4e10879

Update history

d9ca13d

Rename ModelType to ModelOrDc

9388d91

Added inheritance replace test

cca0ecb

More consiste dataclass validator tests

3509edb

Merge branch 'master' into master

32fc164

fix history.

1ecf59d

Paul Martin added 3 commits April 11, 2019 22:35

Remove Optional ModelOrDc Type

dc07d50

Merge branch 'master' of https://github.com/primal100/pydantic

4eba77d

Fix with origin

Fix ModelOrDc

d08608c

samuelcolvin merged commit 50fd2c5 into pydantic:master Apr 11, 2019

alexdrydew pushed a commit to alexdrydew/pydantic that referenced this pull request Dec 23, 2023

Set info.data and info.field_name for dataclass field validators (pyd…

3f96b09

…antic#454)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Validators for dataclasses #454

Validators for dataclasses #454

primal100 commented Apr 2, 2019 •

edited by samuelcolvin

Ronserruya commented Apr 4, 2019

primal100 commented Apr 4, 2019

samuelcolvin commented Apr 4, 2019

samuelcolvin left a comment

samuelcolvin Apr 8, 2019

primal100 Apr 10, 2019

samuelcolvin Apr 8, 2019

primal100 Apr 10, 2019

samuelcolvin Apr 8, 2019

primal100 Apr 10, 2019

samuelcolvin Apr 8, 2019

primal100 Apr 10, 2019 •

edited

codecov bot commented Apr 10, 2019 •

edited

samuelcolvin left a comment

samuelcolvin Apr 11, 2019

primal100 Apr 11, 2019

samuelcolvin Apr 11, 2019

primal100 Apr 11, 2019

primal100 Apr 11, 2019

samuelcolvin Apr 11, 2019

primal100 Apr 11, 2019

samuelcolvin Apr 11, 2019

primal100 Apr 11, 2019

samuelcolvin Apr 11, 2019

primal100 Apr 11, 2019

primal100 commented Apr 11, 2019 •

edited

samuelcolvin commented Apr 11, 2019

samuelcolvin commented Apr 11, 2019

		return type(v) == type(ClassVar) and (sys.version_info < (3, 7) or getattr(v, '_name', None) == 'ClassVar')


		def is_classvar(ann_type: AnyType) -> bool:

		return _check_classvar(ann_type) or _check_classvar(getattr(ann_type, '__origin__', None))


		def gather_validators(type_: AnyType) -> Dict[str, classmethod]:

Validators for dataclasses #454

Validators for dataclasses #454

Conversation

primal100 commented Apr 2, 2019 • edited by samuelcolvin

Change Summary

Related issue number

Checklist

Ronserruya commented Apr 4, 2019

primal100 commented Apr 4, 2019

samuelcolvin commented Apr 4, 2019

samuelcolvin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

primal100 Apr 10, 2019 • edited

Choose a reason for hiding this comment

codecov bot commented Apr 10, 2019 • edited

Codecov Report

samuelcolvin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

primal100 commented Apr 11, 2019 • edited

samuelcolvin commented Apr 11, 2019

samuelcolvin commented Apr 11, 2019

primal100 commented Apr 2, 2019 •

edited by samuelcolvin

primal100 Apr 10, 2019 •

edited

codecov bot commented Apr 10, 2019 •

edited

primal100 commented Apr 11, 2019 •

edited