[PYD-143] Cannot get mypy to work with both pydantic v1.x and pydantic.v1 from v2.x

[jwodder]

Initial Checks

• I confirm that I'm using Pydantic V2 installed directly from the main branch, or equivalent

Description

(Originally posted as Discussion question #5971, where I was encouraged to repost as an issue)

We have several inter-dependent packages that depend on pydantic in ways that will break under v2, and we've decided to stagger the updates to these packages, including having transitional releases that import from either pydantic.v1 (on pydantic 2.x) or pydantic (on pydantic 1.x). Our code runs correctly with this strategy, but when we try to type-check it with mypy, things break, and I am unsure how to fix this breakage.

Example Code

An example (available in full at https://github.com/jwodder/pydantic-v1v2-test): This module:

from __future__ import annotations
from pathlib import Path
from typing import Literal

try:
    import pydantic.v1 as pydantic
except ImportError:
    import pydantic  # type: ignore[no-redef]


class Foobar(pydantic.BaseModel):
    classname: Literal["Foobar"] = pydantic.Field("Foobar", readOnly=True)
    foo: int
    bar: str

    @classmethod
    def load(cls, file: Path) -> Foobar:
        return cls.parse_raw(file.read_text())


if __name__ == "__main__":
    print(Foobar(foo=42, bar="hello"))

with this mypy config:

[mypy]
allow_incomplete_defs = False
allow_untyped_defs = False
ignore_missing_imports = True
no_implicit_optional = True
implicit_reexport = False
local_partial_types = True
pretty = True
show_error_codes = True
show_traceback = True
strict_equality = True
warn_redundant_casts = True
warn_return_any = True
warn_unreachable = True
plugins = pydantic.mypy

[pydantic-mypy]
init_forbid_extra = True
warn_untypes_fields = True

and this tox.ini:

[tox]
envlist = run,devrun,typing,devtyping
skip_missing_interpreters = True
isolated_build = True
minversion = 3.3.0

[testenv:run]
commands = python -m foobar

[testenv:devrun]
commands_pre =
    pip install --pre --upgrade --no-warn-conflicts git+https://github.com/pydantic/pydantic
commands = python -m foobar

[testenv:typing]
deps =
    mypy
commands =
    mypy src

[testenv:devtyping]
deps = {[testenv:typing]deps}
commands_pre = {[testenv:devrun]commands_pre}
commands = {[testenv:typing]commands}

When tested with Python 3.11.3 on macOS 11.7.2 using mypy 1.3.0, using variously pydantic 1.10.8 and dev pydantic commit f6479b8, the run and devrun environments succeed, but the typing and devtyping do not. The typing environment fails with:

src/foobar.py:24: error: Returning Any from function declared to return
"Foobar"  [no-any-return]
            return cls.parse_raw(file.read_text())
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Found 1 error in 1 file (checked 1 source file)

and the devtyping environment fails with:

src/foobar.py:28: error: Missing named argument "classname" for "Foobar" 
[call-arg]
        print(Foobar(foo=42, bar="hello"))
              ^~~~~~~~~~~~~~~~~~~~~~~~~~~
Found 1 error in 1 file (checked 1 source file)

Python, Pydantic & OS Version

pydantic version: 2.0b2
        pydantic-core version: 0.38.0 release build profile
                 install path: /Users/jwodder/work/dev/tmp/pydantic-v1v2-test/.tox/devtyping/lib/python3.11/site-packages/pydantic
               python version: 3.11.3 (main, Apr  7 2023, 19:30:05) [Clang 13.0.0 (clang-1300.0.29.30)]
                     platform: macOS-11.7.2-x86_64-i386-64bit
     optional deps. installed: ['typing-extensions']

_PYD-143

[yarikoptic]

I would deeply appreciate if someone looks at this issue since it blocks migration to support pydantic v2 for us.

[dmontagu]

For what it's worth, you should be able to fix the src/foobar.py:28: error: Missing named argument "classname" for "Foobar" issue by putting default="Foobar" instead of just "Foobar" as the first argument to Field. But this is a sign that the mypy plugin is not working correctly, unsurprisingly. I think we may be able to make that work by adding handling of pydantic.v1 items in the mypy plugin, it should be fairly straightforward, (though not very fun to test, probably).

I'm not quite sure what is going on in the first one (with the Any), it would be convenient if you could explain exactly what I need to do to reproduce the issue. It seems clone the repo and then run something with tox? But if you could explain what commands I should run to set up the environment and how to run things using tox (I don't have experience using tox) then I can look into it further.

[jwodder]

@dmontagu To reproduce with tox:

• Clone https://github.com/jwodder/pydantic-v1v2-test
• Install tox via pip
• Inside the clone, run tox -e typing. This will create a virtualenv, install the local project in it (including pydantic v1), install mypy, and run mypy src.

As to why this error is happening, I've noticed in the past that mypy doesn't like code of the form try: import package1; except ImportError: import package2, and it often proceeds as though only package1 was imported, even if it's not installed in the environment. Since pydantic v1 doesn't have a pydantic.v1 module, mypy is presumably acting as though some un-annotated module unrelated to pydantic was imported, and so it doesn't know what the return type of parse_raw() is supposed to be. One way that pydantic could address this would be to provide a pydantic.v1 module in v1 as well, with support in the v1 mypy plugin.

[yarikoptic]

@dmontagu do you see how we could be of any help to help this issue to get addressed?

[randomed]

Same issue with alias:

from typing_extensions import Annotated

from pydantic.v1 import BaseModel as BaseModel1
from pydantic.v1 import Field as Field1
from pydantic import BaseModel as BaseModel2
from pydantic import Field as Field2


class ModelV1(BaseModel1):
    class Config:
        allow_population_by_field_name = True

    foo: str = Field1(alias="bar")

# no mypy error in 1.x, but produces a mypy error in 2.x
m1 = ModelV1(foo="param")  # error: Unexpected keyword argument "foo" for "ModelV1"  [call-arg]

print(m1.json(by_alias=True)) # {"bar": "param"}

# equivalent V2 model here for comparison
class ModelV2(BaseModel2):
    foo: Annotated[str, Field2(serialization_alias="bar")]

m2 = ModelV2(foo="param")
print(m2.model_dump_json(by_alias=True)) # {"bar": "param"}

Update:
Ended up working around it by casting the Field type:

from pydantic import Field
from pydantic.v1 import BaseModel
from pydantic.v1 import Field as V1Field
from pydantic.v1.fields import Undefined
from pydantic.v1.typing import NoArgAnyCallable


def v1_field(default: Any = Undefined, *, alias: Optional[str] = None, default_factory: Optional[NoArgAnyCallable] = None) -> Field:  # type: ignore[valid-type]
    """
    Pydantic's mypy plugin has a bug in 2.x that causes 1.x's `Field` objects to not be typed correctly:
    https://github.com/pydantic/pydantic/issues/6022

    This method will cast fields into the 2.x version until the bug is fixed.
    """
    return V1Field(default=default, alias=alias, default_factory=default_factory)  # type: ignore[no-any-return]

class Foo(BaseModel):
   bar: str = v1_field("default", alias="alias")

[yarikoptic]

I've noticed in the past that mypy doesn't like code of the form try: import package1; except ImportError: import package2, and it often proceeds as though only package1 was imported, even if it's not installed in the environment

@jwodder Do you think it might be something to check with mypy folks, so may be it could be addressed at that level?

[jwodder]

@yarikoptic python/mypy#1393 seems to be the closest pre-existing mypy issue.

[yarikoptic]

digging through linked issues there, ran into python/mypy#1153 (comment) suggesting code pattern like

if TYPE_CHECKING:
    import json
else:
    try:
        import simplejson as json
    except ImportError:
        import json

and thus making it specific for what interface to use during TYPE_CHECKING. Would it help us here?

[jwodder]

@yarikoptic That would only work here if the version of pydantic installed in the type-checking environment was fixed to either 1.x or 2.x (as the pydantic version installed determines whether to import pydantic or pydantic.v1). While that would be doable for type-checking a library that directly uses pydantic, any other project that depended on that library would have to use the same major version of pydantic when type-checking, yet the whole point of the "staggered pydantic update" design is to avoid forcing early upgrades to pydantic dependencies.

[jenshnielsen]

Ran into the same issue trying to port code from v1 to v2 and for now have followed @yarikoptic's suggestion and am only type checking with v1. It seems like this could be done cleaner if we had one final release of the 1.x branch that ships with the v1 namespace as an alias to the top-level namespace such that we could pin pydantic to >= 1.10.x and use from pydantic.v1 imports everywhere until we are ready to start supporting v2