improve error handling in pydantic validation

[gpetretto]

Pydantic 2.5 introduced some changes that lead to issues in case of unions of MSONable object and dict. See for example materialsproject/atomate2#616 and materialsproject/emmet#889.

Consider the following code

from monty.json import MSONable
from pydantic import BaseModel


class Mson(MSONable):
    def __init__(self, a):
        self.a = a


class Model(BaseModel):
    m: Mson | dict


mson_dict = Mson(5).as_dict()
print(Model.model_validate({"m": mson_dict}))

mson_dict["b"] = 6
print(Model.model_validate({"m": mson_dict}))

With pydantic 2.4.2 this gives the output:

m=<__main__.Mson object at 0x11bf52dd0>
m={'@module': '__main__', '@class': 'Mson', '@version': None, 'a': 5, 'b': 6}

while it failes for pydantic 2.5.1:

m=<__main__.Mson object at 0x150e98610>
Traceback (most recent call last):
  File "/python/tests/packages/pydantic/bug_v2.5/test_bug.py", line 20, in <module>
    print(Model.model_validate({"m": mson_dict}))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python3.11/site-packages/pydantic/main.py", line 503, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python3.11/site-packages/monty/json.py", line 267, in validate_monty_v2
    return cls._validate_monty(__input_value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python3.11/site-packages/monty/json.py", line 242, in _validate_monty
    new_obj = MontyDecoder().process_decoded(__input_value)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/site-packages/monty/json.py", line 486, in process_decoded
    return cls_.from_dict(data)
           ^^^^^^^^^^^^^^^^^^^^
  File "/python3.11/site-packages/monty/json.py", line 200, in from_dict
    return cls(**decoded)
           ^^^^^^^^^^^^^^
TypeError: Mson.__init__() got an unexpected keyword argument 'b'

From what I have seen, the problem is that now pydantic does not simply accept all the exceptions coming from the validation and when TypeError is raised it stops entirely, instead of trying with the next type in the union.

This PR tries to address the issue by ensuring that a ValueError is always returned if a problem arises during the deserialization. Tests has been added that should cover this use case.
Note that this does not address issues like: materialsproject/atomate2#607

Pinging a few people that may be interested or can provide feedback on this: @Andrew-S-Rosen @utf @munrojm @janosh

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (44340e7) 78.88% compared to head (a43a19c) 79.15%.
Report is 8 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #597      +/-   ##
==========================================
+ Coverage   78.88%   79.15%   +0.27%     
==========================================
  Files          27       27              
  Lines        1397     1401       +4     
  Branches      299      299              
==========================================
+ Hits         1102     1109       +7     
+ Misses        239      237       -2     
+ Partials       56       55       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gpetretto]

Coming back to this PR, is there anything needed to go forward? Any comment?

[shyuep]

You need to remove the three changes to the test files in your PR. Otherwise it will cause the tests to fail.

[gpetretto]

Thanks for pointing that out. It looks like those changes were added by this automatic commit 8c69096.
It should be fine now.