Mypy finds errors in the first example from the documentation

[andreymal]

This example

from datetime import datetime
from typing import List
from pydantic import BaseModel

class User(BaseModel):
    id: int
    name = 'John Doe'
    signup_ts: datetime = None
    friends: List[int] = []

external_data = {
    'id': '123',
    'signup_ts': '2019-06-01 12:22',
    'friends': [1, 2, '3']
}
user = User(**external_data)
print(user.id)
print(repr(user.signup_ts))
print(user.friends)
print(user.dict())

$ mypy --strict pydantic_example.py 
pydantic_example.py:3: error: Module 'pydantic' has no attribute 'BaseModel'
pydantic_example.py:5: error: Class cannot subclass 'BaseModel' (has type 'Any')
pydantic_example.py:8: error: Incompatible types in assignment (expression has type "None", variable has type "datetime")

This is not false positive. Pydantic __init__.py uses implicit reexport (this is an error, and I don't want to ignore it using --no-implicit-reexport), and this example uses None for a non-optional field (this is also an error).

In general, I can't trust a library that suggests putting None in a non-optional field in the very first example. This violates the mypy type system.

[dmontagu]

I agree that the implicit reexport is a nuisance; I think it is worth addressing. @samuelcolvin any ideas?

---

While mypy doesn’t like the following: signup_ts: datetime = None, there is an awkward inconsistency with mypy that this would be allowed as function arguments, and would imply the argument was optional. Given that it raises a mypy error, if you want to pass mypy, of course you should “correct” it. It is valid pydantic though if you choose not to care.

For what it’s worth, there may be times when you explicitly choose to override the annotations and have mypy assume the type is str, but allow it to be None for pydantic (eg, if you know a validator will always make it non-None). Obviously you would want to add # type: ignore but this doesn’t necessarily need to be a part of the documentation.

That said, I’m also not opposed to changing the docs here, PR welcome.

---

In general, I can't trust a library that suggests putting None in a non-optional field in the very first example.

Well, if you can’t trust the library, you are absolutely free to not use it.

This kind of language is not appreciated by the people who put much of their spare time into this project, and are offering it up to you for free. If you care enough to create an issue, please be courteous and try to phrase your misgivings in a less incendiary way.

[samuelcolvin]

Can we fix the implicit re-export in __init__ with __all__? It would be a little ugly but very explicit, we could add a test by extending the mypy tests. But very hard to implement a test that makes sure every new thing available to import from __init__.py gets added to __all__.

I'm less bothered by the Optional/None thing. Every python developer knows what this means, if they don't know what it means, adding optional won't help. Adding this would:

1. Add more fluff to the first example which tries to be as minimal as possible.
2. Perhaps confuse users as to whether Optional is required by pydantic. Since pydantic is very unusual in using annotations at run-time we have to be careful about this.

---

Couldn't have said it better myself.

When I see questions like this I'm always left asking myself the same question:

Why create the issue? - If you don't like the look of the library (I see lots of libraries that I instantly decide I don't want to use), why create the issue? If you do like it the library, surely you know starting off like that won't help get your problem fixed.

I guess the answer is either, to vent at an unjust world, or to demonstrate some new-found piece of knowledge by pointing out what you reprieve to be an error.

😴

[andreymal]

Every python developer knows what this means

Yes, every python developer and every mypy user knows this is an obvious error.

print(user.signup_ts.year)

Mypy will think this code is correct, but actually it will raise AttributeError: 'NoneType' object has no attribute 'year' when signup_ts is not set. This is a false negative because of the incorrect type hint.

Using # type: ignore is more awful.

[samuelcolvin]

python is not mypy. The standard library doesn't use mypy, many widely popular python packages don't use mypy. In fact a tiny percentage of python written or executed in the world everyday has passed mypy.

Using Optional[datetime] doesn't prevent the potential bug in python; just allows mypy (or other static analysis libraries) to warn about it more easily. Note, it would in theory be possible for static analysis tools to find this potential bug without the Optional[] decoration, as mypy does for keyword arguments.

I still think the example in the documentation is okay. If you'd like to create a PR to add a note linking from the first example to how to use pydantic with mypy I'd be happy to review it.

[andreymal]

I understand it, but it's strange that you don't respect the mypy type system. If you think that print(user.signup_ts.year) false negative is fine, it implicitly means that pydantic is not compatible with mypy and other static analysis libraries by design, and this means that python developers can't use mypy and pydantic together in their projects and libraries (they can use type:ignore, but it's equivalent to lack of mypy).

[samuelcolvin]

pydantic works absolutely fine with mypy indeed,

• @dmontagu put a massive amount of effort into building a mypy extension just for pydantic, see Bump flake8-quotes from 2.0.1 to 2.1.0 #708
• I put a fairly large amount of effort implement full type hint coverage in pydantic in add type hints #373
• We have two sections of the docs dedicated to mypy

That still doesn't mean our first example has to be fully type hinted. Just as most examples in django, requests, or the python standard library aren't type hinted.

[dmontagu]

it implicitly means that pydantic is not compatible with mypy and other static analysis libraries by design

It absolutely does not mean this, and it feels like you are being purposely obtuse for the sake of being contrarian.

All it means is that this one documentation example is not valid with mypy. You absolutely do not need to make use of any mypy-violating patterns in your use of pydantic, and we have done our best to make sure that pydantic is completely compatible with complete mypy coverage.

I personally use the same mypy config that pydantic uses in all of my projects, and while it is not --strict, it is also far from lenient. (And I think we should absolutely strive toward better supporting --strict mode.)

Yes, there are some patterns that work with pydantic that are not valid with mypy. There is also an enormous amount of non-pydantic python code that is valid python and not valid with mypy.

[euri10]

it implicitly means that pydantic is not compatible with mypy and other static analysis libraries by design

It absolutely does not mean this, and it feels like you are being purposely obtuse for the sake of being contrarian.

@dmontagu and @samuelcolvin it seems like you're both playing a new troll game 🤣

[samuelcolvin]

@euri10 you've confused me and my entire office 😕 😖. Shouldn't it be "is compatible"?

[euri10]

@euri10 you've confused me and my entire office . Shouldn't it be "is compatible"?

just for you ;)

[andreymal]

there are some patterns that work with pydantic that are not valid with mypy.

This is a problem. And you can't change my mind

django, requests, or the python standard library aren't type hinted.

This is also a problem, but I hope most libraries will be type hinted in the future. Note that typeshed provides type hints for the standard library and some other libraries. Although the standard library itself is not checked with mypy (it's probably also a problem but I'm not sure because the standard library has a lot of unit tests), typeshed allows me to check my code that interacts with the standard library. It doesn't save from bugs in the standard library, but it saves from bugs in my code.

There is also an enormous amount of non-pydantic python code that is valid python and not valid with mypy.

I prefer to use a python subset that is valid with mypy wherever possible. (Sometimes I use Any because I'm very lazy, but this is also a big problem in my discipline that I should fix someday.)

[dmontagu]

This is a problem.

All right, well once your PEP to change mypy errors to syntax errors is accepted, we’ll get right on converting the behavior of every edge case in the use of this library.

[samuelcolvin]

django, requests, or the python standard library aren't type hinted.

You've miss quoted me, I said

Just as most examples in django, requests, or the python standard library aren't type hinted.

Which is quite different. pydantic is type hinted.

I was talking about examples in documentation where it's common (and I would argue good practice) to partially omit type hints to keep the code as succinct and easy to read as possible.