Exclude with nested "__all__" does not work as expected

[xspirus]

Bug

Output of python -c "import pydantic.utils; print(pydantic.utils.version_info())":

             pydantic version: 1.5.1
            pydantic compiled: False
                 install path: /home/spirus/projects/pydantic/pydantic
               python version: 3.8.3 (default, May 17 2020, 18:15:42)  [GCC 10.1.0]
                     platform: Linux-5.6.14-arch1-1-x86_64-with-glibc2.2.5
     optional deps. installed: ['typing-extensions', 'email-validator', 'devtools']

As I was experimenting with FastAPI and pydantic, I thought that it would be nice to use exclude in order to avoid some Schema duplication (or complex inheritances). Ultimately, I came across a weird bug when trying to use exclude on nested sequences. Below is a minimal example for reproducing the bug.

from typing import List

from pydantic import BaseModel


class A(BaseModel):
    a: int = 1
    b: int = 2
    c: int = 3


class B(BaseModel):
    a: List[A]


class C(BaseModel):
    b: List[B]


def test():
    c = C(b=[B(a=[A()])])
    print(c.dict(exclude={"b": {0: {"a": {"__all__": {"c"}}}}}))
    print(c.dict(exclude={"b": {"__all__": {"a": {"__all__": {"c"}}}}}))
    assert c.dict(exclude={"b": {0: {"a": {"__all__": {"c"}}}}}) == {"b": [{"a": [{"a": 1, "b": 2}]}]}
    assert c.dict(exclude={"b": {"__all__": {"a": {"__all__": {"c"}}}}}) == {"b": [{"a": [{"a": 1, "b": 2}]}]}

Running the above function with pytest the output is:

Test session starts (platform: linux, Python 3.8.3, pytest 5.3.5, pytest-sugar 0.9.2)
rootdir: /home/spirus/projects/pydantic, inifile: setup.cfg
plugins: cov-2.8.1, mock-3.1.0, sugar-0.9.2
collecting ... 

―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

    def test():
        c = C(b=[B(a=[A()])])
        print(c.dict(exclude={"b": {0: {"a": {"__all__": {"c"}}}}}))
        print(c.dict(exclude={"b": {"__all__": {"a": {"__all__": {"c"}}}}}))
        assert c.dict(exclude={"b": {0: {"a": {"__all__": {"c"}}}}}) == {"b": [{"a": [{"a": 1, "b": 2}]}]}
>       assert c.dict(exclude={"b": {"__all__": {"a": {"__all__": {"c"}}}}}) == {"b": [{"a": [{"a": 1, "b": 2}]}]}
E       AssertionError: assert {'b': [{}]} == {'b': [{'a': ...1, 'b': 2}]}]}
E         Differing items:
E         {'b': [{}]} != {'b': [{'a': [{'a': 1, 'b': 2}]}]}
E         Use -v to get the full diff

example.py:25: AssertionError
------------------------------------------------------------ Captured stdout call ------------------------------------------------------------
{'b': [{'a': [{'a': 1, 'b': 2}]}]}
{'b': [{}]}

 example.py ⨯                                                                                                                  100% ██████████

Results (0.05s):
       1 failed
         - example.py:20 test

I inspected the codebase a little bit, and found out that the bug is produced because of how _normalize_indexes, as it assumes that when the items is a dict containing the __all__ key, then the value is considered by default to be a set. So calling the update, information may be missed.

I am not certain if my usage violates how the __all__ key should be used, but if this should have been correct, I can create a PR with a solution.

[samuelcolvin]

your naming is very confusing 😕, do you think you could clarify with cases than don't reuse names like "b"?

But I think you're right.

[xspirus]

@samuelcolvin you are right. Here is a more non-confusing example I hope :D

from typing import List

from pydantic import BaseModel


class Alpha(BaseModel):
    id: int = 1
    name: str = "Alpha"


class Beta(BaseModel):
    alphas: List[Alpha]


class Model(BaseModel):
    betas: List[Beta]


def test():
    model = Model(
        betas=[Beta(alphas=[Alpha(), Alpha(id=2)]), Beta(alphas=[Alpha(id=3)])]
    )
    assert model.dict() == {
        "betas": [
            {
                "alphas": [
                    {"id": 1, "name": "Alpha"},
                    {"id": 2, "name": "Alpha"},
                ]
            },
            {"alphas": [{"id": 3, "name": "Alpha"}]},
        ]
    }
    assert model.dict(
        exclude={
            "betas": {
                0: {"alphas": {"__all__": {"name"}}},
                1: {"alphas": {"__all__": {"name"}}},
            }
        }
    ) == {
        "betas": [{"alphas": [{"id": 1}, {"id": 2}]}, {"alphas": [{"id": 3}]}]
    }
    assert model.dict(
        exclude={"betas": {"__all__": {"alphas": {"__all__": {"name"}}}}}
    ) == {
        "betas": [{"alphas": [{"id": 1}, {"id": 2}]}, {"alphas": [{"id": 3}]}]
    }

Output using pytest is:

Test session starts (platform: linux, Python 3.8.3, pytest 5.4.2, pytest-sugar 0.9.3)
rootdir: /home/spirus/projects/local/pydantic, inifile: setup.cfg
plugins: cov-2.8.1, sugar-0.9.3, mock-3.1.0
collecting ...

―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― test ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

    def test():
        model = Model(
            betas=[Beta(alphas=[Alpha(), Alpha(id=2)]), Beta(alphas=[Alpha(id=3)])]
        )
        assert model.dict() == {
            "betas": [
                {
                    "alphas": [
                        {"id": 1, "name": "Alpha"},
                        {"id": 2, "name": "Alpha"},
                    ]
                },
                {"alphas": [{"id": 3, "name": "Alpha"}]},
            ]
        }
        assert model.dict(
            exclude={
                "betas": {
                    0: {"alphas": {"__all__": {"name"}}},
                    1: {"alphas": {"__all__": {"name"}}},
                }
            }
        ) == {
            "betas": [{"alphas": [{"id": 1}, {"id": 2}]}, {"alphas": [{"id": 3}]}]
        }
>       assert model.dict(
            exclude={"betas": {"__all__": {"alphas": {"__all__": {"name"}}}}}
        ) == {
            "betas": [{"alphas": [{"id": 1}, {"id": 2}]}, {"alphas": [{"id": 3}]}]
        }
E       AssertionError: assert {'betas': [{}, {}]} == {'betas': [{'...[{'id': 3}]}]}
E         Differing items:
E         {'betas': [{}, {}]} != {'betas': [{'alphas': [{'id': 1}, {'id': 2}]}, {'alphas': [{'id': 3}]}]}
E         Use -v to get the full diff

example.py:44: AssertionError

 example.py ⨯                                                                                                                                                                  100% ██████████
================================================================================== short test summary info ===================================================================================
FAILED example.py::test - AssertionError: assert {'betas': [{}, {}]} == {'betas': [{'...[{'id': 3}]}]}

Results (0.12s):
       1 failed
         - example.py:19 test

As I mentioned, the problem seems to occur when using __all__ as the index. I think I have found the solution and can submit a PR, once I've written tests to cover these as well.

[pacific0009]

from typing import List

from pydantic import BaseModel


class Option(BaseModel):
    text: str
   data: str


class Question(BaseModel):
    options: List[Option]


class Item(BaseModel):
    questions: List[Question]
item = Item(questions =[Question(options=[Option(text='x', data='y')])]

item.dict(exclude={'questions': {'0': {'options': {'__all__': {'data'}}}}})

above is working as expected but

item.dict(exclude={'questions': {'__all__': {'options': {'__all__': {'data'}}}}})

excludes entire option key for each entry in questions instead of excluding only correctAnswer key from each option