Root validator called in superclass even if overridden in a subclass without calling super

[jkozlowicz]

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.6.1
            pydantic compiled: True
                 install path: /Users/jakub/.virtualenvs/server/lib/python3.8/site-packages/pydantic
               python version: 3.8.1 (default, Mar 13 2020, 20:31:03)  [Clang 11.0.0 (clang-1100.0.33.17)]
                     platform: macOS-10.15.5-x86_64-i386-64bit
     optional deps. installed: ['typing-extensions']

Methods decorated with root_validator that are defined in a subclass do not override the method defined in the superclass. For methods decorated with validator they do override the method defined in the superclass properly.

Context:
I want to have a model that declares a bunch of fields and some default validators for these fields, which should be applied in every subclass, unless overridden. That works for fields decorated with validator, but for fields decorated with root_validator the validator in superclass gets called first.

I couldn't find anything in the docs to disable calling the superclass validator when using the root_validator decorator. Also, I looked into the generic models (it seems that it would fit composition better than inheritance) and reusing validators (doesn't seem to be a good candidate to validate field contents because in my case the validation depends on the exercise_type and I would still have to define something like _normalized_choices in every subclass).

from __future__ import annotations

from pydantic import BaseModel, root_validator, validator
from typing import Union, List, Optional


class ExerciseBase(BaseModel):

    text: Optional[str] = None
    choices: Optional[List[List[str]]] = None

    @root_validator
    def validate_choices(cls, values):
        print("ExerciseBase validate_choices")
        choices = values.get('choices')
        assert choices is None, 'Choices not allowed for this exercise type'
        return values


class FillBlanksExercise(ExerciseBase):
    @root_validator
    def validate_choices(cls, values):
        print("FillBlanksExercise validate_choices")
        choices = values.get('choices')
        if choices is not None:
            text = values.get('text')
            number_of_blanks = text.count('____')
            assert {len(choice) for choice in choices} == {number_of_blanks}, 'Each choice must match number of blanks'
        return values

EXERCISE_TYPES_MAPPING = {
    'FillBlanksExercise': FillBlanksExercise,
    'ChoiceExercise': ChoiceExercise,
}

exercise_data = {
    'text': 'Python is a type of ____?',
    'exercise_type': 'FillBlanksExercise',
    'choices': [['snake'], ['bird'], ['fish'], ['dinosaur']],
}

exercise = FillBlanksExercise(**exercise_data)


# ExerciseBase validate_choices
# FillBlanksExercise validate_choices
# Traceback (most recent call last):
#   File "/Users/jakub/Library/Application Support/JetBrains/PyCharm2020.2/scratches/scratch_2.py", line 85, in <module>
#     exercise = ExerciseBase.build(**exercise_data)
#   File "/Users/jakub/Library/Application Support/JetBrains/PyCharm2020.2/scratches/scratch_2.py", line 22, in build
#     return _class(**kwargs)
#   File "pydantic/main.py", line 346, in pydantic.main.BaseModel.__init__
# pydantic.error_wrappers.ValidationError: 1 validation error for FillBlanksExercise
# __root__
#   Choices not allowed for this exercise type (type=assertion_error)

[alcrene]

I recently ran into this as well, and it certainly was unexpected to me. I ended up writing a small patch to the metaclass which removes overridden validators post-hoc. I post it below in case it is useful to others.

from pydantic import BaseModel
from pydantic.main import ModelMetaclass

def remove_overridden_validators(model: BaseModel) -> BaseModel:
    """
    Currently a Pydantic bug prevents subclasses from overriding root validators.
    (see https://github.com/samuelcolvin/pydantic/issues/1895)
    This function inspects a Pydantic model and removes overriden
    root validators based of their `__name__`.
    Assumes that the latest entries in `__pre_root_validators__` and
    `__post_root_validators__` are earliest in the MRO, which seems to be
    the case.
    """
    model.__pre_root_validators__ = list(
        {validator.__name__: validator
         for validator in model.__pre_root_validators__
         }.values())
    model.__post_root_validators__ = list(
        {validator.__name__: (skip_on_failure, validator)
         for skip_on_failure, validator in model.__post_root_validators__
         }.values())
    return model

class PatchedModelMetaclass(ModelMetaclass):
    def __new__(*args, **kwargs):
        model = ModelMetaclass.__new__(*args, **kwargs)
        return remove_overridden_validators(model)

The following bit of code tests that it works as expected:

from pydantic import BaseModel, root_validator

class A(BaseModel):
# class A(BaseModel, metaclass=PatchedModelMetaclass):
    a: int
    @root_validator(pre=True)
    def pre_root(cls, values):
        print("pre rootA")
        return values
    @root_validator(pre=False)
    def post_root(cls, values):
        print("post rootA")
        return values

class B(A):
    @root_validator(pre=True)
    def pre_root(cls, values):
        print("pre rootB")
        return values
    @root_validator(pre=False)
    def post_root(cls, values):
        print("post rootB")
        return values

# This prints only from the validators in B if PatchedModelMetaclass is used
B(a=1)

[PrettyWood]

Thanks for reporting! I opened #2670 to fix this. Feedback welcome :)

[alcrene]

Great! I can confirm that this solves my original use case than ran into the issue, as well as the test case I posted.

[m0wer]

I recently ran into this as well, and it certainly was unexpected to me. I ended up writing a small patch to the metaclass which removes overridden validators post-hoc. I post it below in case it is useful to others.

from pydantic import BaseModel
from pydantic.main import ModelMetaclass

def remove_overridden_validators(model: BaseModel) -> BaseModel:
    """
    Currently a Pydantic bug prevents subclasses from overriding root validators.
    (see https://github.com/samuelcolvin/pydantic/issues/1895)
    This function inspects a Pydantic model and removes overriden
    root validators based of their `__name__`.
    Assumes that the latest entries in `__pre_root_validators__` and
    `__post_root_validators__` are earliest in the MRO, which seems to be
    the case.
    """
    model.__pre_root_validators__ = list(
        {validator.__name__: validator
         for validator in model.__pre_root_validators__
         }.values())
    model.__post_root_validators__ = list(
        {validator.__name__: (skip_on_failure, validator)
         for skip_on_failure, validator in model.__post_root_validators__
         }.values())
    return model

class PatchedModelMetaclass(ModelMetaclass):
    def __new__(*args, **kwargs):
        model = ModelMetaclass.__new__(*args, **kwargs)
        return remove_overridden_validators(model)

The following bit of code tests that it works as expected:

from pydantic import BaseModel, root_validator

class A(BaseModel):
# class A(BaseModel, metaclass=PatchedModelMetaclass):
    a: int
    @root_validator(pre=True)
    def pre_root(cls, values):
        print("pre rootA")
        return values
    @root_validator(pre=False)
    def post_root(cls, values):
        print("post rootA")
        return values

class B(A):
    @root_validator(pre=True)
    def pre_root(cls, values):
        print("pre rootB")
        return values
    @root_validator(pre=False)
    def post_root(cls, values):
        print("post rootB")
        return values

# This prints only from the validators in B if PatchedModelMetaclass is used
B(a=1)

Does anyone know about a similar fix but for a pydantic dataclass?