Can't parse_raw Optional[Json] generated from Pydantic

[volker48]

Bug

Please complete:

• OS: any
• Python version import sys; print(sys.version):3.7.3 (default, Mar 27 2019, 09:23:15)
  [Clang 10.0.1 (clang-1001.0.46.3)]
• Pydantic version import pydantic; print(pydantic.VERSION): 1.2

Where possible please include a self contained code snippet describing your bug:

from typing import Optional
from pydantic import Json, BaseModel

class Test(BaseModel):
    name: str
    some_obj: Optional[Json]

t = Test(name="hi")
serialized = t.json()
print(serialized)
t_deserialized = Test.parse_raw(serialized)

The above will fail with:

pydantic.error_wrappers.ValidationError: 1 validation error for Test
some_obj
  JSON object must be str, bytes or bytearray (type=type_error.json)

Because serialized is JSON that takes the following form: {"name": "hi", "some_obj": null}. If the some_obj key is omitted entirely everything works fine.

I would expect parse_raw() to work with the output of json().

The crux of the issue seems to be that validation will fail if an Optional[Json] field is present, but has a value of None. This will also fail Test(name='foo', some_obj=None).

This example works fine in pydantic < 1.

[samuelcolvin]

Thanks for reporting, shouldn't be too hard to fix. PR welcome to fix it, otherwise I'll try to fix before the next release.

[volker48]

I can fix it. I've added a failing test case in my fork, but where do you think the best place to fix the issue is? I'm not super familiar with the code base so a hint in the right direction would be greatly appreciated.

[samuelcolvin]

https://github.com/samuelcolvin/pydantic/blob/890589a6106ba6d8dcd0408278c79a910e18068f/pydantic/fields.py#L454-L455

Hopefully as simple as allowing validate_json pass None straight through, then later logic will cause an exception if the field is not allowed to be None.

Simplest approach would be to add that test too and see what happens.

[volker48]

So fixing the original issue is as straight forward as you thought, but I think I've uncovered another issue related to Json.
The following test fails:

def test_json_required():
    class JsonRequired(BaseModel):
        json_obj: Json

    with pytest.raises(ValidationError):
        JsonRequired()

Failed: DID NOT RAISE <class 'pydantic.error_wrappers.ValidationError'>

This happens before I even make any changes to pass None through.

I can get the expected behavior if I do:

def test_json_required():
    class JsonRequired(BaseModel):
        json_obj: Json = ...

    with pytest.raises(ValidationError):
        JsonRequired()

I have an PR here #1080 with the failing test in there, but I wanted to get the PR up to discuss how to fix the code causing the test failure (or possibly remove the failing test and fix the other issue under a separate PR).

[whippiii]

@volker48 Maybe this is related?: If required is not provided (in this case the ...), the field is set to allow None and not being required:
https://github.com/samuelcolvin/pydantic/blob/890589a6106ba6d8dcd0408278c79a910e18068f/pydantic/fields.py#L340-L343
after having received Any as the new type:
https://github.com/samuelcolvin/pydantic/blob/890589a6106ba6d8dcd0408278c79a910e18068f/pydantic/fields.py#L329-L330

[samuelcolvin]

maybe related to #1031

[volker48]

I removed the failing test as this seems like a more involved fix and is only tangentially related to this issue. My PR now passes all checks in CI.

[samuelcolvin]

So fixing the original issue is as straight forward as you thought, but I think I've uncovered another issue related to Json.
The following test fails:

def test_json_required():
    class JsonRequired(BaseModel):
        json_obj: Json

    with pytest.raises(ValidationError):
        JsonRequired()

Failed: DID NOT RAISE <class 'pydantic.error_wrappers.ValidationError'>

...

This is correct since v1.2, see here.