Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-use validators? #940

Closed
varunchopra opened this issue Oct 25, 2019 · 19 comments · Fixed by #941
Closed

Re-use validators? #940

varunchopra opened this issue Oct 25, 2019 · 19 comments · Fixed by #941
Labels

Comments

@varunchopra
Copy link

@varunchopra varunchopra commented Oct 25, 2019

Is it possible to create a validator and import it everywhere else it's used?

I have validators for certain fields that are present in multiple classes that I'd like to call simply without defining them.

@samuelcolvin
Copy link
Owner

@samuelcolvin samuelcolvin commented Oct 25, 2019

I'm on my mobile, but should be as simple as doing something like:

class MyModel(BaseModel):
    check_foo = validator('foo')(reused_validator)

Have a try and let me know if that works or doesn't work or I have the syntax slightly wrong.

@dmontagu
Copy link
Collaborator

@dmontagu dmontagu commented Oct 25, 2019

@samuelcolvin I don't think that works due to this check:

        if not in_ipython():  # pragma: no branch
            ref = f.__module__ + '.' + f.__qualname__
            if ref in _FUNCS:
                raise ConfigError(f'duplicate validator function "{ref}"')
            _FUNCS.add(ref)

I think we can probably fix this though.

@dmontagu
Copy link
Collaborator

@dmontagu dmontagu commented Oct 25, 2019

Here's a self contained demonstration that it doesn't currently work.

from pydantic import BaseModel, validator


def double_validator(cls, value):
    return value * 2


class A(BaseModel):
    x: int

    double = validator('x')(double_validator)


class B(BaseModel):
    x: int

    double = validator('x')(double_validator)

# pydantic.errors.ConfigError: duplicate validator function "__main__.double_validator"

@samuelcolvin
Copy link
Owner

@samuelcolvin samuelcolvin commented Oct 25, 2019

Humm, possibly. We could either drop that check or make it optional.

Or you could setup a validator which just calls the reused validator.

Maybe allow_name_reuse as another argument on a validator?

@samuelcolvin samuelcolvin reopened this Oct 25, 2019
@dmontagu
Copy link
Collaborator

@dmontagu dmontagu commented Oct 25, 2019

Here's a hack that works for now:

from copy import deepcopy
from typing import Callable

from pydantic import BaseModel, validator
from pydantic.typing import AnyCallable


def reused_validator(
    *fields: str,
    pre: bool = False,
    each_item: bool = False,
    always: bool = False,
    check_fields: bool = True,
    whole: bool = None,
) -> Callable[[AnyCallable], classmethod]:
    def dec(f: AnyCallable) -> classmethod:
        f = deepcopy(f)
        f.__qualname__ = f"{f.__qualname__}{reused_validator.count}"
        reused_validator.count += 1
        return validator(*fields, pre=pre, whole=whole, always=always, check_fields=check_fields)(f)

    return dec


reused_validator.count = 0


def double_validator(cls, value):
    return value * 2


class A(BaseModel):
    x: int
    double = reused_validator('x')(double_validator)


class B(BaseModel):
    x: int
    double = reused_validator('x')(double_validator)


print(A(x=1))
print(B(x=1))
# x=2
# x=2

This is using v1 imports, but I also checked that if you fix the imports it works in v0.32.2.

@samuelcolvin related -- are models supposed to print like that now? On master I'm not seeing the model class name when I print the model.

@dmontagu
Copy link
Collaborator

@dmontagu dmontagu commented Oct 25, 2019

@samuelcolvin I think it makes sense to keep that error in many cases because the most common situation is that you've accidentally duplicated the name of the validator inside the class, which would cause problems if the logic changed (and is presumably why you added the error in the first place).

I was thinking it could make sense to just look for a . in the __qualname__ before raising the error. If the function was defined globally in a module, you almost certainly meant to reuse it.

Separately, I also think it would be good to add allow_name_reuse as a keyword argument so that you could do the same thing with a function defined at the class level.

Also, we could add a check for whether something is already a classmethod when building the validator so that you could define it as such inside the class and not have a misleading signature.

I can start putting together a PR for this.

@samuelcolvin
Copy link
Owner

@samuelcolvin samuelcolvin commented Oct 25, 2019

are models supposed to print like that now? On master I'm not seeing the model class name when I print the model

Yes, that's standard for __str__ methods, I think you reviewed the PR. Bit late to change now.

@dmontagu
Copy link
Collaborator

@dmontagu dmontagu commented Oct 25, 2019

Yeah, I remember it now 😅; I'm fine with it since we are calling the repr on nested values. I just haven't seen it in the wild much yet (still using 0.32.2 with fastapi :/), so it caught me by surprise.

@varunchopra
Copy link
Author

@varunchopra varunchopra commented Oct 25, 2019

Thanks for the quick response on this. I wasn't even aware we could call validators using check_foo = validator('foo')(reused_validator).

@dmontagu
Copy link
Collaborator

@dmontagu dmontagu commented Oct 25, 2019

@chopraaa that's basically all a python decorator does behind the scenes

@skewty
Copy link
Contributor

@skewty skewty commented Nov 6, 2019

I came across this issue this morning. Thanks for the temporary work around idea.

@dmontagu unless I am failing to see / understand something, I don't see where/how count ever gets incremented. That said it seems to compile and run. This fact almost interested me more :)

def reused_validator(
    *fields: str,
    pre: bool = False,
    each_item: bool = False,
    always: bool = False,
    check_fields: bool = True,
    whole: bool = None,
) -> Callable[[AnyCallable], classmethod]:
    def dec(f: AnyCallable) -> classmethod:
        f = deepcopy(f)
        f.__qualname__ = f"{f.__qualname__}{reused_validator.count + 1}"
        return validator(*fields, pre=pre, whole=whole, always=always, check_fields=check_fields)(f)

    return dec


reused_validator.count = 0

so I added

print(f"Function name will be: {f.__qualname__}")

and this is what came out:

Function name will be: partition_range_check1
Function name will be: partition_range_check11

which explains why it works. That said count is perhaps a poor name given the reason it works :) (unless we are counting notches on a stick)

I wasn't sure if this was by design or by mistake. I Just wanted to share for others who might also be struggling to understand.

@dmontagu
Copy link
Collaborator

@dmontagu dmontagu commented Nov 6, 2019

That was just a mistake -- I meant to actually increment the value. It's now fixed above!

(Clearly there are some edge cases this wouldn't handle well, e.g., if you end the validator name with a number. This is only intended as an admittedly hacky interim solution.)

@pmav99
Copy link
Contributor

@pmav99 pmav99 commented Dec 30, 2019

The issue was closed by #941 but allow_reuse seems not to have been documented yet. If anyone is interested, it can be used like this:

import typing
import pydantic

def normalize(value: str) -> str:
    return value.lower()

class Goo(pydantic.BaseModel):
    a: str
    b: str

    # validators
    _ensure_a_is_normalized: classmethod = pydantic.validator("a", allow_reuse=True)(normalize)
    _ensure_b_is_normalized: classmethod = pydantic.validator("b", allow_reuse=True)(normalize)

g = Goo(a="A", b="BB")
assert g.a == "a"
assert g.b == "bb"

We can further reduce repetition in the models by defining a help function:

import typing
import pydantic

def normalize(value: str) -> str:
    return value.lower()

def normalizing_validator(field: str) -> classmethod:
    decorator = pydantic.validator(field, allow_reuse=True)
    validator = decorator(normalize)
    return validator

class Hoo(pydantic.BaseModel):
    a: str
    b: str

    # validators
    _ensure_a_is_normalized: classmethod = normalizing_validator("a")
    _ensure_b_is_normalized: classmethod = normalizing_validator("b")

h = Hoo(a="A", b="BB")
assert h.a == "a"
assert h.b == "bb"

@samuelcolvin
Copy link
Owner

@samuelcolvin samuelcolvin commented Dec 30, 2019

@pmav99: Would be great if you could submit a PR with documentation for this?

@samuelcolvin
Copy link
Owner

@samuelcolvin samuelcolvin commented Dec 30, 2019

(thanks for reminding us it's not documented)

@pmav99
Copy link
Contributor

@pmav99 pmav99 commented Dec 30, 2019

Sure.

  1. Do you want to have both version in the docs? The first one is simpler and somewhat easier to understand, while the second one is a bit more realistic.
  2. Which section? Maybe between "validate always" and "root-validators"? https://pydantic-docs.helpmanual.io/usage/validators/#root-validators
  3. I should remove types for the docs, right?

@samuelcolvin
Copy link
Owner

@samuelcolvin samuelcolvin commented Dec 30, 2019

Really up to you on all three, do what you think is clearest and we can discuss on the PR.

Thanks so much.

@romain20100
Copy link

@romain20100 romain20100 commented Jan 19, 2021

Sorry to dig up this old topic, but I am curious to know why this check for duplicate validators even is a thing.

Considering the discussion above, it doesn't seem to be a critical problem?

I am asking because I find having to add allow_reuse=True everywhere a bit annoying.
And the alternative is even worse: declaring custom Models with the validator class methods and all is like heavy artillery.

@berzi
Copy link

@berzi berzi commented Feb 24, 2022

I have the same question as @romain20100 and in addition: can one not simply make a mixin with the validators and inherit from it in all models that should use them? If so, then why is this not recommended (or even mentioned) in the docs, since I believe it would be the cleanest and most pythonic (read: you'd understand what is going on right away by reading the code, without having to know how Pydantic validators work in detail)? Is there some hidden gotcha?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants