Multiple validation errors for union between type and dictionary of the same type

[hadavand-rms]

Initial Checks

• I have searched GitHub for a duplicate issue and I'm sure this is something new
• I have searched Google & StackOverflow for a solution and couldn't find anything
• I have read and followed the docs and still think this is a bug
• I am confident that the issue is with pydantic (not my code, or another library in the ecosystem like FastAPI or mypy)

Description

Consider the code block below where the Transaction model can accept one single user or a dictionary of users. The initialization works with no problem

from pydantic import BaseModel, Field, SecretStr
from typing import Union, Dict

class User(BaseModel):
    id: int
    username: str = "dsds"
    password: SecretStr
    
    class Config:
        """Pydantic configuration"""

        arbitrary_types_allowed = True
        validate_all = True  # Validate the default parameters too
        validate_assignment = False
        extra = "forbid"  # Forbid passing keyword arguments that are not registered


class Transaction(BaseModel):
    id: str
    user: Union[Dict[str, User], User]
    value: int
    
Transaction(id='1234567890',
    user={"user_1": dict(
        id=45,
        username='JohnDoe',
        password='hashedpassword'
    )},
    value=9876543210)

>>> Transaction(id='1234567890', user={'user_1': User(id=45, username='JohnDoe', password=SecretStr('**********'))}, value=9876543210)

But, if there is a validation issue, then multiple errors are generated as it check over the union of types

Transaction(id='1234567890',
    user={"user_1": dict(
        id="foo",
        username='JohnDoe',
        password='hashedpassword'
    )},
    value=9876543210)

ValidationError                           Traceback (most recent call last)
<ipython-input-37-f42c1f766053> in <module>
     27         password='hashedpassword'
     28     )},
---> 29     value=9876543210)
     30 
     31 # Transaction(id='1234567890',

~\RMS-Python\envs\bhp3.7\lib\site-packages\pydantic\main.cp37-win_amd64.pyd in pydantic.main.BaseModel.__init__()

ValidationError: 4 validation errors for Transaction
user -> user_1 -> id
  value is not a valid integer (type=type_error.integer)
user -> id
  field required (type=value_error.missing)
user -> password
  field required (type=value_error.missing)
user -> user_1
  extra fields not permitted (type=value_error.extra)

Not sure if this is something that can be handled by discriminator?

pydantic version info

# Name                    Version                   Build  Channel
pydantic                  1.10.2           py37h51bd9d9_0    conda-forge

Example Code

No response

Python, Pydantic & OS Version

pydantic version: 1.10.2\n            pydantic compiled: True\n                 install path: C:\\Users\\hadav\\RMS-Python\\envs\\bhp3.7\\Lib\\site-packages\\pydantic\n               python version: 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 05:37:49) [MSC v.1916 64 bit (AMD64)]\n                     platform: Windows-10-10.0.22621-SP0\n     optional deps. installed: ['dotenv', 'email-validator', 'typing-extensions']

Affected Components

• Compatibility between releases
• Data validation/parsing
• Data serialization - .dict() and .json()
• JSON Schema
• Dataclasses
• Model Config
• Field Types - adding or changing a particular data type
• Function validation decorator
• Generic Models
• Other Model behaviour - construct(), pickling, private attributes, ORM mode
• Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

[hadavand-rms]

A workaround to make the validations work is to override __init__ and change the field.sub_fields and rest it back using a try finally.

[hramezani]

Thanks @hadavand-rms for reporting this and sorry for the late response 🙏

I think the behavior is correct as Pydantic tries to validate data against all types in the union.

This is almost the same in Pydantic V2.

[dmontagu]

I'll note that, while this particular case may be difficult to handle cleanly, #6915 if merged might provide a way to improve the error messages here, though you'd have to have a clear way to distinguish whether a value was meant to be a dict or the model itself.

[hadavand-rms]

Thanks for looking into this 👍. The main logic from my understanding is that a Union can support multiple type validation scenarios but once the initialization is done and it is decided which type from the Union has been used, then the error message should correspond to what that has been set.

What I ended up doing is to check the data type prior to pydnatic initialization and switch the subfields and I wonder if that can ne handled by pydantic.init()

subfields = self.__fields__["parameters"].sub_fields
try:
   if data_type == type1:
       self.__fields__["parameters"].sub_fields = [subfields[0]]
   else:
       self.__fields__["parameters"].sub_fields = [subfields[1]]
   super().__init__(**kwargs)
finally:
   self.__fields__["parameters"].sub_fields = subfields