`TypeAliasType` ref `KeyError` #8320

samuelcolvin · 2023-12-07T08:49:18Z

See https://github.com/samuelcolvin/FastUI/compare/pydantic-ref-bug?expand=1

To reproduce:

check that out
install deps
run python -c 'import fastui'

Exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/samuel/code/fastui/src/python-fastui/fastui/__init__.py", line 11, in <module>
    class FastUI(pydantic.RootModel):
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 182, in __new__
    complete_model_class(
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 501, in complete_model_class
    schema = gen_schema.clean_schema(schema)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 405, in clean_schema
    schema = _discriminated_union.apply_discriminators(schema)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_discriminated_union.py", line 60, in apply_discriminators
    return simplify_schema_references(_core_utils.walk_core_schema(schema, inner))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py", line 480, in simplify_schema_references
    schema = walk_core_schema(schema, count_refs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py", line 431, in walk_core_schema
    return f(schema.copy(), _dispatch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py", line 476, in count_refs
    recurse(definitions[ref], count_refs)
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py", line 207, in walk
    return f(schema, self._walk)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py", line 464, in count_refs
    return recurse(s, count_refs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py", line 210, in _walk
    schema = self._schema_type_to_method[schema['type']](schema.copy(), f)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py", line 219, in _handle_other_schemas
    schema['schema'] = self.walk(sub_schema, f)  # type: ignore
                       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py", line 207, in walk
    return f(schema, self._walk)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py", line 464, in count_refs
    return recurse(s, count_refs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py", line 210, in _walk
    schema = self._schema_type_to_method[schema['type']](schema.copy(), f)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py", line 254, in handle_list_schema
    schema['items_schema'] = self.walk(items_schema, f)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py", line 207, in walk
    return f(schema, self._walk)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/samuel/code/fastui/env311/lib/python3.11/site-packages/pydantic/_internal/_core_utils.py", line 476, in count_refs
    recurse(definitions[ref], count_refs)
            ~~~~~~~~~~~^^^^^
KeyError: 'fastui.components.AnyComponent:4401377552'

The text was updated successfully, but these errors were encountered:

dmontagu · 2023-12-07T09:26:34Z

Looks to be an issue with recursive type aliases that are (or contain as union cases) BaseModels.

More minimal reproductions:

import typing_extensions as _te

from pydantic import BaseModel, TypeAdapter


class Div(BaseModel):
    components: 'AnyComponent'


AnyComponent = _te.TypeAliasType('AnyComponent', Div)

TypeAdapter(AnyComponent)

pydantic_core._pydantic_core.SchemaError: Invalid Schema:
definitions.definitions.1.definition-ref.ref
  Extra inputs are not permitted [type=extra_forbidden, input_value='__main__.AnyComponent:4346368048', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/extra_forbidden

import typing_extensions as _te

from pydantic import BaseModel, TypeAdapter, Field


class Div(BaseModel):
    type: _te.Literal['Div'] = 'Div'
    components: 'AnyComponent'


AnyComponent = _te.TypeAliasType('AnyComponent', _te.Annotated[Div, Field(discriminator='type')])

TypeAdapter(AnyComponent)

  File "/Users/davidmontague/Programming/pydantic/pydantic/pydantic/_internal/_core_utils.py", line 476, in count_refs
    recurse(definitions[ref], count_refs)
            ~~~~~~~~~~~^^^^^
KeyError: '__main__.AnyComponent:4337067376'

(Note: if you want to actually be able to instantiate the types, you'd want to change 'AnyComponent' to 'AnyComponent | None' above, it just makes the core schema building a bit more complex.)

dmontagu · 2023-12-07T09:38:11Z

Changing pydantic._internal._core_utils.validate_core_schema to

from pydantic_core import SchemaError

def validate_core_schema(schema: CoreSchema) -> CoreSchema:
    if 'PYDANTIC_SKIP_VALIDATING_CORE_SCHEMAS' in os.environ:
        return schema
    try:
        return _validate_core_schema(schema)
    except SchemaError as e:
        pretty_print_core_schema(schema)
        raise

You can see:

{
    'type': 'definitions',
    'schema': {
        'type': 'definition-ref',
        'schema_ref': '__main__.AnyComponent:4307210288'
    },
    'definitions': [
        {
            'type': 'model',
            'cls': <class '__main__.Div'>,
            'schema': {
                'type': 'model-fields',
                'fields': {
                    'components': {
                        'type': 'model-field',
                        'schema': {
                            'type': 'definition-ref',
                            'schema_ref': '__main__.AnyComponent:4307210288'
                        }
                    }
                },
                'model_name': 'Div'
            },
            'ref': '__main__.Div:4313912816'
        },
        {
            'type': 'definition-ref',
            'schema_ref': '__main__.Div:4313912816',
            'ref': '__main__.AnyComponent:4307210288'
        }
    ]
}

The issue is:

        {
            'type': 'definition-ref',
            'schema_ref': '__main__.Div:4313912816',
            'ref': '__main__.AnyComponent:4307210288'
        }

Right now, in pydantic-core, we don't allow definition-ref schemas to have refs, but I don't currently see any reason this should be a problem in principle. I'm not sure if it would address everything or just reveal other issues but it seems reasonable to me to support.

samuelcolvin · 2023-12-18T12:40:55Z

I'm getting the same while trying to use a ref for AnyEvent:

AnyEvent = TypeAliasType('AnyEvent', Annotated[Union[PageEvent, GoToEvent, BackEvent, AuthEvent], Field(discriminator='type')])

here gives me the same error.

dmontagu · 2023-12-19T00:53:39Z

@samuelcolvin I've pushed branches called dmontagu/support-indirect-definition-refs to both pydantic and pydantic-core:

I didn't test extensively, but using both of those branches together seems to resolve this issue, at least it prevents the error on import and some basic validation seemed to work.

I didn't check the JSON schema, etc., and I won't be surprised if we need to do a bit more work, but let me know if this seems to be on the right track.

samuelcolvin added the bug V2 Bug related to Pydantic V2 label Dec 7, 2023

samuelcolvin mentioned this issue Dec 7, 2023

JSON schema tests pydantic/FastUI#70

Merged

samuelcolvin mentioned this issue Dec 8, 2023

Expanding the components in /docs hang the server pydantic/FastUI#58

Open

dmontagu mentioned this issue Jan 9, 2024

Fix schema-building bug with TypeAliasType for types with refs #8526

Merged

sydney-runkle closed this as completed in #8526 Jan 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`TypeAliasType` ref `KeyError` #8320

`TypeAliasType` ref `KeyError` #8320

samuelcolvin commented Dec 7, 2023

dmontagu commented Dec 7, 2023 •

edited

dmontagu commented Dec 7, 2023

samuelcolvin commented Dec 18, 2023

dmontagu commented Dec 19, 2023

TypeAliasType ref KeyError #8320

TypeAliasType ref KeyError #8320

Comments

samuelcolvin commented Dec 7, 2023

dmontagu commented Dec 7, 2023 • edited

dmontagu commented Dec 7, 2023

samuelcolvin commented Dec 18, 2023

dmontagu commented Dec 19, 2023

`TypeAliasType` ref `KeyError` #8320

`TypeAliasType` ref `KeyError` #8320

dmontagu commented Dec 7, 2023 •

edited