Unwanted and incorrect type coercion in list

[JakkuSakura]

Checks

• I added a descriptive title to this issue
• I have searched (google, github) for similar issues and couldn't find anything
• I have read and followed the docs and still think this is a bug

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.8.2
            pydantic compiled: True
                 install path: /usr/local/lib/python3.9/site-packages/pydantic
               python version: 3.9.1 (default, Jan  8 2021, 17:17:43)  [Clang 12.0.0 (clang-1200.0.32.28)]
                     platform: macOS-11.2.3-x86_64-i386-64bit
     optional deps. installed: ['typing-extensions']

Basically, I have list[Foo], but I then pass [Foo(), Bar('something'), Foo()] and got [Foo(), BaseFoo(), Foo()]. The validation fails silently and did a wanted coercion. This behavior took me a long time to catch caused bug.

from pydantic import BaseModel

class SourceNode(BaseModel):
    pass

class BulkNode(SourceNode):
    sources: list[str] = []

class RustAstNode(BaseModel):
    pass

class RustStatementNode(RustAstNode):
    nodes: list[RustAstNode] = []

class RustRawNode(RustAstNode):
    raw: str

if __name__ == '__main__':
    input_list = [RustRawNode(raw='hi'), BulkNode(sources=['very import information']), RustRawNode(raw='hi')]
    node = RustStatementNode(nodes=input_list)
    # validate manually
    for inp, n in zip(input_list, node.nodes):
        assert type(inp) == type(n), 'Unwanted type coercion {} != {}'.format(type(inp), type(n))

Traceback (most recent call last):
  File "/Users/jack/Dev/python_test/src/test.py", line 30, in <module>
    assert type(inp) == type(n), 'Unwanted type coercion {} != {}'.format(type(inp), type(n))
AssertionError: Unwanted type coercion <class '__main__.BulkNode'> != <class '__main__.RustAstNode'>

Here is a relative question from stackoverflow
https://stackoverflow.com/questions/66019853/pydantic-prevent-conversion-of-incorrect-type

#1423
#1098

[christopherpole-cs]

Use a validator to check each list item - remember pydantic is a parsing library first and a validation library second; i.e. its aim is to guarantee output data.

Try this:

class RustStatementNode(RustAstNode):
    nodes: t.List[RustAstNode] = Field(default_factory=list)

    @validator('nodes', each_item=True)
    def check_node_type(cls, v):
        assert isinstance(v, RustAstNode)
        return v

adjust the validator as necessary if you want to silently ignore non-conforming types.

[JakkuSakura]

Would you accept a pull request if I change the behavior so that it raises an exception here?

[samuelcolvin]

I don't see a bug here, or even unexpected behaviour BulkNode does not inherit from RustAstNode (which is the constraint for nodes), so you wouldn't expect a BulkNode to survive validation. RustAstNode has no fields, and doesn't have extra = 'forbid' so it will accept virtually anything.

RustRawNode should be downcast (or upcast, I can never remember which) to a RustAstNode. I can't remember if this is how pydantic v1.x works, but it's definitely fixed in v2.

[JakkuSakura]

you wouldn't expect a BulkNode to survive validation

BulkNode shouldn’t survive validation, but I expect an Exception, not a silent conversion.

…

On 21 Apr 2022, 10:40 PM +0800, samuelcolvin/pydantic ***@***.***>, wrote: you wouldn't expect a BulkNode to survive validation

[samuelcolvin]

Use validators.

Pydantic looks at attributes primarily, not attributes.