Pydantic fails with Python 3.10 new UnionType #3300
Comments
|
Nevermind, already found in master, that you did exactly what I thought to do 😄 I will wait for the new release, thank you for your work! |
DrJackilD
changed the title
|
I'll try to get a new release out asap. |
|
@samuelcolvin any ETA? |
|
+1 on this |
|
I'll reopen the issue just to keep it visible until it gets released |
|
Here's a workaround until it gets released. from types import UnionType
from typing import Union
from pydantic.main import ModelMetaclass, BaseModel
class MyModelMetaclass(ModelMetaclass):
def __new__(cls, name, bases, namespace, **kwargs):
annotations = namespace.get("__annotations__", {})
for annotation_name, annotation_type in annotations.items():
if isinstance(annotation_type, UnionType):
annotations[annotation_name] = Union.__getitem__(annotation_type.__args__)
return super().__new__(cls, name, bases, namespace, **kwargs)
class MyBaseModel(BaseModel, metaclass=MyModelMetaclass):
...
class Python310(MyBaseModel):
a: str | None
print(repr(Python310(a="3.10")))Same approach code also works for SQLModel, but instead of inheriting from Update: rather fragile addition that also supports ForwardRef'sfrom types import UnionType
from typing import Union
from pydantic.main import ModelMetaclass, BaseModel
class MyModelMetaclass(ModelMetaclass):
def __new__(cls, name, bases, namespace, **kwargs):
annotations = namespace.get("__annotations__", {})
for annotation_name, annotation_type in annotations.items():
if isinstance(annotation_type, UnionType):
annotations[annotation_name] = Union.__getitem__(
annotation_type.__args__
)
elif isinstance(annotation_type, str) and "|" in annotation_type:
# it's rather naive to think that having | can only mean that it's a Union
# It will work in most cases, but you need to be aware that it's not always the case
annotations[
annotation_name
] = f'Union[{", ".join(map(str.strip, annotation_type.split("|")))}]'
return super().__new__(cls, name, bases, namespace, **kwargs)
class MyBaseModel(BaseModel, metaclass=MyModelMetaclass):
...
class Python310(MyBaseModel):
a: str | None
b: "WhatIsThat | int | None"
class WhatIsThat(BaseModel):
foo = "1"
Python310.update_forward_refs()
print(repr(Python310(a="3.10", b=None))) |
|
I think there's a related bug that, based on the existing fix in master, I think would also be fixed by this. I wanted to share the trace here in case it helps someone googling, but didn't think it was worthy of a separate issue. If you disagree let me know and I'll raise one. Given: One gets: The JSON error confused me when googling this initially after updating a FastAPI project of mine to 3.10 and making no other changes. I believe it happens because the lack of support for |
|
+1 on this |
|
+1 |
If I may, I'd like to discourage people from posting comments like "+1".The issue is already confirmed and acknowledged by maintainer, fix will be available with the next release: #3300 (comment). These comments are noisy for everyone subscribed to this issue, while providing no value at all. You can subscribe to this issue if you wish to be notified when it will be fixed. In the meanwhile you can use this as a workaround: #3300 (comment). |
|
@Bobronium sorry for the noise then. For me personally, "+1" comments are having the value of noticing the maintainer there are actually people interested in the fix (and I see this practice is very common in open-source projects, so I wasn't aware this can be bad), especially considering the #3300 (comment) was actually from a few months ago and there's still no release yet. But you're arguments are perfectly valid and if you see the alternative solution for that (do you see if and how many people subscribe to the issue?) then I'm happy to oblige 😅 |
|
@Toreno96, no worries. Sorry if my comment read too grumpy 🙆🏻. Let me elaborate on why I personally think that "+1" comments are usually a bad idea. Let's start with better alternative first: reactions. While I understand your concern and desire to give a notice to the maintainer, I don't think "+1" would be the best approach for that. I believe maintainers are already very aware of this, and many other issues. I think some form of discussion here would be more applicable if, for example, So I'm not discouraging any conversations, I just believe that "+1" is not a great start for one, especially in given situation :) |
|
Hi guys, I believe the fix suggested in #3300 (comment) will not be working for the following case: |
You can see in the changes, that I alternate between the `typing.Optional` and new union operator `|`, which introduces inconsistency. This is because: 1. It seems the union operator `|` will be the preferred way to annotate the optionals: <https://twitter.com/raymondh/status/1472426960349548549?s=20> I think it makes sense, and tbh I like this syntax, so I decided to incorporate this preference into the project. 1. Unfortunately, Pydantic does not support the new union operator `|` at all yet: <pydantic/pydantic#3300>. I've tried the [suggested workaround](pydantic/pydantic#3300 (comment)), but it didn't work. As such, I decided to use `typing.Optional` in the case of Pydantic models, as an exception to the consistency. --- * Forbid extra fields in the Pydantic model * Store `label` and `issuer` parameters
|
I remember running into this problem when testing 3.10 with Pydantic and found this topic. It appears like it's no longer a problem with the release of 1.9 (#2885), so the problem is resolved. |
|
Sorry for reopening the issue, and sorry if that's not the best place to ask this question, but IMHO it's pretty related: Python 3.7 also supports the new UnionType if $ python
Python 3.7.12 (default, Oct 13 2021, 06:51:32)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> foo: str | None = None
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
>>> from __future__ import annotations
>>> foo: str | None = None
>>> foo is None
TrueBut it does not work in Pydantic 1.9: from __future__ import annotations
from pydantic import BaseModel
class Foo(BaseModel):
bar: str | None$ python example.py
Traceback (most recent call last):
File "a.py", line 6, in <module>
class Foo(BaseModel):
File "pydantic/main.py", line 187, in pydantic.main.ModelMetaclass.__new__
File "pydantic/typing.py", line 356, in pydantic.typing.resolve_annotations
if self._name == 'Optional':
File "/usr/local/Cellar/python@3.7/3.7.12_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/typing.py", line 263, in _eval_type
return t._evaluate(globalns, localns)
File "/usr/local/Cellar/python@3.7/3.7.12_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/typing.py", line 467, in _evaluate
eval(self.__forward_code__, globalns, localns),
File "<string>", line 1, in <module>
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'Is it something that could be supported in the future, or such backward compatibility is out of scope? |
|
same as above happens for python 3.8 |
|
Hi everyone |
|
@PrettyWood thank you very much for the answer and detailed explanation 🙏 |
This commit will update type annotation syntax for Python 3.10. The project currently also supports Python 3.8 and 3.9, so the annotations are imported with `from __future__ import annotations`. The Python 3.10 union operator (the pipe, like `str | None`) will not be used on pydantic models. If running Python 3.9 or below, pydantic is not compatible with the union operator, even if annotations are imported with `from __future__ import annotations`. https://peps.python.org/pep-0604/ https://docs.python.org/3/whatsnew/3.10.html pydantic/pydantic#2597 (comment) pydantic/pydantic#2609 (comment) pydantic/pydantic#3300 (comment)
|
I'm actually experiencing this on Python results in: Adding the following to results in |
|
@GregHilston, Old postFirst of all, does this work as expected? class PromptData(BaseModel):
image = PIL.Image.Image
class Config:
arbitrary_types_allowed = True What about this? from typing import Optional
class PromptData(BaseModel):
image = Optional[PIL.Image.Image]
class Config:
arbitrary_types_allowed = True I also would appreciate if you'd post complete examples, including required imports and pip packages. Upd.Actually, I'm able to reproduce this as well: from pydantic import BaseModel
class M(BaseModel):
a = int | NoneTraceback (most recent call last):
File "repro.py", line 4, in <module>
class M(BaseModel):
File "pydantic/main.py", line 222, in pydantic.main.ModelMetaclass.__new__
File "pydantic/fields.py", line 506, in pydantic.fields.ModelField.infer
File "pydantic/fields.py", line 436, in pydantic.fields.ModelField.__init__
File "pydantic/fields.py", line 557, in pydantic.fields.ModelField.prepare
File "pydantic/fields.py", line 831, in pydantic.fields.ModelField.populate_validators
File "pydantic/validators.py", line 752, in find_validators
RuntimeError: no validator found for <class 'types.UnionType'>, see `arbitrary_types_allowed` in ConfigOk, this is strange: from typing import Optional
from pydantic import BaseModel
class M(BaseModel):
a = Optional[str]gives me Makes me feel like my setup is broken Full reproduction log: ~/temp/test
$ python -m venv venv
~/temp/test
$ . venv/bin/activate
~/temp/test (venv)
$ pip install pydantic
Collecting pydantic
Downloading pydantic-1.10.1-cp310-cp310-macosx_11_0_arm64.whl (2.6 MB)
|████████████████████████████████| 2.6 MB 1.6 MB/s
Collecting typing-extensions>=4.1.0
Using cached typing_extensions-4.3.0-py3-none-any.whl (25 kB)
Installing collected packages: typing-extensions, pydantic
Successfully installed pydantic-1.10.1 typing-extensions-4.3.0
WARNING: You are using pip version 21.2.3; however, version 22.2.2 is available.
You should consider upgrading via the '/Users/rocky/temp/test/venv/bin/python -m pip install --upgrade pip' command.
~/temp/test (venv)
$ pbpaste > repro.py
~/temp/test (venv)
$ python repro.py
Traceback (most recent call last):
File "/Users/rocky/temp/test/repro.py", line 4, in <module>
class M(BaseModel):
File "pydantic/main.py", line 222, in pydantic.main.ModelMetaclass.__new__
File "pydantic/fields.py", line 506, in pydantic.fields.ModelField.infer
File "pydantic/fields.py", line 436, in pydantic.fields.ModelField.__init__
File "pydantic/fields.py", line 557, in pydantic.fields.ModelField.prepare
File "pydantic/fields.py", line 831, in pydantic.fields.ModelField.populate_validators
File "pydantic/validators.py", line 752, in find_validators
RuntimeError: no validator found for <class 'typing._UnionGenericAlias'>, see `arbitrary_types_allowed` in Config
~/temp/test (venv)
$ python -c "from pydantic import version; print(version.version_info())"
pydantic version: 1.10.1
pydantic compiled: True
install path: /Users/rocky/temp/test/venv/lib/python3.10/site-packages/pydantic
python version: 3.10.0 (default, Oct 21 2021, 22:41:19) [Clang 12.0.5 (clang-1205.0.22.11)]
platform: macOS-12.3.1-arm64-arm-64bit
optional deps. installed: ['typing-extensions'] |
|
@Bobronium My apologies on the weak minimum code example. Thanks for doing this leg work to better understand what I was saying :) |
|
shouldn't we replace |
|
Ah, of course... Thank you so much! I thought I started to go crazy. |
|
@hramezani Oh my god. That's hilarious that I totally didn't see that... I'm embarrassed ha |
Checks
Bug
Pydantic fails with a new-style unios, presented in Python 3.10. The reason is straightforward - it simply doesn't know about UnionType.
The problem is here:
pydantic.fields.py:529where we check theoriginfor typeUnion.Since the new
UnionTypeis compatible withtyping.Union(according to documentation) it's probably should be easy to fix. I could do this, but I need to understand, what could be affected by this. Is it will be enough to check all usages of Union type and fix it carefully? Also, it's probably should be some kind of a functionis_unionto make it work for versions before 3.10.Additional info
Output of
python -c "import pydantic.utils; print(pydantic.utils.version_info())":Raises exception:
The text was updated successfully, but these errors were encountered: