Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IMPORTANT: PEP 563, PEP 649 and the future of pydantic #2678

Closed
samuelcolvin opened this issue Apr 15, 2021 · 30 comments
Closed

IMPORTANT: PEP 563, PEP 649 and the future of pydantic #2678

samuelcolvin opened this issue Apr 15, 2021 · 30 comments
Labels

Comments

@samuelcolvin
Copy link
Owner

@samuelcolvin samuelcolvin commented Apr 15, 2021

Update: see below this has been resolved by a changes in python 3.10 from the python steering council.

Thanks everyone for your thoughts, patience and hard work.


PEP 563 "Postponed Evaluation of Annotations" was introduced in python 3.7 behind the from __future__ import annotations switch, it basically meant all annotations are strings, not python objects.

Pydantic has tried to supported Postponed Annotations since v0.18.0, see #348.

The problem however is that trying to evaluate those strings to get the real annotation objects is really hard, perhaps impossible to always do correctly, see #248 #234 #397 #415 #486 #531 #545 #635 #655 #704 #1298 #1332 #1345 #1370 #1668 #1736 #1873 #1895 #2112 #2314 #2323 #2411

The reasons are complicated but basically typing.get_type_hints() doesn't work all the time and neither do the numerous hacks we've introduced to try and get fix it. Even if typing.get_type_hints() was faultless, it would still be massively slower than the current semantics or PEP 649 (see below).

In short - pydantic doesn't work very well with postponed annotations, perhaps it never will.

The Problem

The problem is that postponed annotations are set to be come default in 3.10, features of which will be frozen in about three week.

Even worse, there's no way to switch back to the current behaviour.

The Solution

The solution to this is PEP 649 developed by Larry Hastings. This basically means annotations evaluation is lazy - the work of building __annotations__ isn't done until you access __annotations__.

As far as I can tell this is the best of both worlds.

The Second Problem

The sad reality however is that it seems very possible that PEP 649 will get rejected and when python 3.10 is released it will break much of pydantic, and thereby FastAPI and all the other libraries that rely on pydantic (as well as other libraries that use annotations at runtime like enforce and typer presumably).

See this very long discussion of the issue on python-dev about whether PEP 649 is a good idea.


The Point

So why am I saying all this apart from whinging about something I don't agree with?

1. This is fair warning

That pydantic might break in a big way if python's core developers continue value principle over pragmatism.

2. You can help

Thousands of developers and organisations big and small rely on pydantic. Type hints might have been conceived to help readers and static type checkers, but they're now used to great effect at runtime by lots and lots of people - they make python better.

In the end I don't think python's core developers and steering council want to make your experience of python worse. They just haven't realised how important this is. (Even Larry hadn't heard of pydantic or FastAPI until yesterday when he emailed me, I guess they don't read the python developer survey 😉)

If you value pydantic or FastAPI or other libraries that use annotations at runtime, please (constructively and respectfully) let the python steering council know that you would like PEP 649 to be accepted. Please don't contact members of the python community, they're aware of this issue (see below) and are taking it seriously.

I understand the decision on PEP 649 will be made over the next few days, so if you're going to do anything do it today.

@samuelcolvin samuelcolvin pinned this issue Apr 15, 2021
@mathematicalmichael
Copy link

@mathematicalmichael mathematicalmichael commented Apr 15, 2021

Could you please provide some suggestions for the best way to accomplish this ask?:

If you value pydantic or FastAPI or other libraries that use annotations at runtime, please (constructively and respectfully) let the python steering council know that you would like PEP 649 to be accepted.

Is there an email I can write to? Some forum to post on? What's the most impactful course of action?

@samuelcolvin
Copy link
Owner Author

@samuelcolvin samuelcolvin commented Apr 15, 2021

Not really, it's precisely because I don't know how to make my concerns clear that I created this issue.

I've posted to python-dev supporting the PEP, but I suspect that more people chiming in to just support what I've said might do more harm than good.

I've you've spoken to core devs in the past and they know you, perhaps prompting them might be the best thing to do.

@notatallshaw
Copy link

@notatallshaw notatallshaw commented Apr 15, 2021

As a long time reader of the Python Dev mailing list I would say that it is not necessary to contribute to the mailing list beyond what @samuelcolvin has already posted, unless there are specific technical questions that you are highly qualified to answer.

The mail is already extremely forthright and I have no doubt it will generate a lot of discussion. One thing I would say though @samuelcolvin is it will probably help your case a lot to go in to more details about why typing.get_type_hints() doesn't work for pydantic, either because of specific issues, real world performance examples, or if there is some fundamental flaw in the approach itself.

There's a lot of smart people in the Python core Dev team, so it makes sense to state the problem not the solution. It may be that typing.get_type_hints() can be fixed or some other method could be implemented that would breach the gap.

My 2 cents anyway, hope it helps.

@samuelcolvin
Copy link
Owner Author

@samuelcolvin samuelcolvin commented Apr 15, 2021

I take your point, but the problems with typing.get_type_hints() and PEP 563 are well described in PEP 649 and debated at length in the other thread, I didn't want to bring all that up again.

@notatallshaw
Copy link

@notatallshaw notatallshaw commented Apr 15, 2021

I'm going to be honest, I skimmed over many dozens of mails for this one.

But to me it seemed the discussions were talking about problems in the abstract, without a real motivation or need for them to be fixed. Whereas specifics on actual problems implementing it, with an actual library, that's used by lots of users, is a more much compelling position.

I would guess some core devs are going to ask questions about this though anyway, so responses to those questions will likely fill the gap.

Best of luck! Don't use this tool on a day to day basis, but it's been great when I have 😄

@gaborbernat
Copy link

@gaborbernat gaborbernat commented Apr 15, 2021

I think you should explain better why it's impossible to implement lazy evaluation of type hints. The long list of issues doesn't help get a better understanding of the problem in achieving this, and in many it's never explained why it's hard to do.

@methane
Copy link

@methane methane commented Apr 15, 2021

I take your point, but the problems with typing.get_type_hints() and PEP 563 are well described in PEP 649 and debated at length in the other thread, I didn't want to bring all that up again.

Even there is a long thread, I don't know what point is actually breaking pydantic.

@brettcannon
Copy link

@brettcannon brettcannon commented Apr 15, 2021

Do note that as a steering council member it is more important to me to hear from you directly, @samuelcolvin , in PEP 649 itself on how its acceptance/rejection will impact Pydantic as a maintainer of a popular package using runtime annotations. But inundating us with a flood of emails or trying to get a high upvote count is actually a negative as it implicitly tries to put the SC in a position of "us versus them" for this decision (on top of having to sift through more emails with what limited time I have for SC topics).

I am personally already well aware of the popularity of Pydantic, FastAPI, and Typer, so there's no need to prove that to the SC.

@pablogsal
Copy link

@pablogsal pablogsal commented Apr 16, 2021

My recommendation here is to gather as much detailed, objective and technical information on how PEP 563 is impacting pydantic in a way it cannot adapt without compromising the core of the package. As a member of the steering council, I normally spend many hours parsing the mailing lists, https://discuss.python.org/ and other sources to gather a complete and holistic view of all the pros, cons and different views and how different parties should be impacted in order to make the best decision possible. The fact that this is bringing to our attention very close to beta freeze gives us very limited time to react in a way that keeps out standards, so it would be very very helpful to get this information directly from you without needing to be extracted from tons of messages.

Also, as a release manager of Python 3.10, it makes me sad that the first issues that are mentioned here:

#248 #234 #397 #415 #486 #531 #545 #635 #655 #704 #1298 #1332 #1345 #1370 #1668 #1736 #1873 #1895 #2112 #2314 #2323 #2411

go back to 2018 but we are hearing about all these problems and how they impact pydantic critically dangerously close to beta freeze. My priority as a release manager is to veil for the highest stability for the release. A considerable amount of features are merged days before feature freeze and this already has a considerable amount of chaos and work from the release management team and the buildbot team and the potential of reverting or including such big changes on top of all this with too little time close to feature freeze makes my job much harder as you can imagine.

In any case, for us, making sure that all our user base is taken into account is a very serious matter so you can be sure that we will take this into account when discussing the overall issue.

I do understand that everyone is very nervous and there is a lot of people worried and all of this is augmented with time limitations, but we are all volunteers that want the best for our users and the community so let's all be empathetic to each other, and try to work together towards the best solution.

Thanks for your patience and for your understanding.

P.S.

I guess they don't read the python developer survey 😉

We certainly do (at least from the Steering Council 😄 ).

@willingc
Copy link

@willingc willingc commented Apr 16, 2021

Hi folks.

Let me state up front that I'm a very satisfied user of pydantic and FastAPI, and I'm very thankful for the work and contributions the maintainers and community around them have put in. ☀️

I also appreciate the volunteer efforts of the Python core team, Steering Council, and many contributors to Python.

I'm optimistic that we can find a win-win for pydantic / FastAPI and Python. I believe this is possible if we try not to polarize the solution prematurely to "all or nothing" or "accept or reject 649". To accomplish that we need to look at this through the lens of "what is possible", balance the tradeoffs, and work toward a "good but perhaps not ideal" solution.

Thank you and I'm looking forward to seeing us move forward together.

@tirkarthi
Copy link

@tirkarthi tirkarthi commented Apr 16, 2021

There is an API proposal over adding typing.get_annotations(object) that could help and seems to have similar motivation : https://bugs.python.org/issue43817

Not sure if it helps but an issue were KeyError was raised for typing.get_type_hints was resolved : https://bugs.python.org/issue41515

@FelixTheC
Copy link

@FelixTheC FelixTheC commented Apr 16, 2021

I will have the same issue in the future.
I wrote my own implementation which works in that way that all my tests are passing.

https://github.com/FelixTheC/strongtyping/blob/%2347_in_py10_globals_from_func_must_be_known/strongtyping/utils.py

Feel free to adapt and/or change it for your needs.

@samuelcolvin
Copy link
Owner Author

@samuelcolvin samuelcolvin commented Apr 16, 2021

First of all, I really appreciate what the python core developers in general, and the steering council in particular do for us all.

I only heard about the debate about PEP 649 on Wednesday night, and given the extremely narrow window to get my point heard, I banged the drum in every way I could think of. The lesson here is "careful what you wish for", my message has been amplified more than I expected.

I'm sorry if that has wasted anyone's time, even more so if it reduces the chance my request gets a positive reception.

You're also right that i should have engaged with python-dev long ago about this, I'm sorry I did not. Lesson (hopefully) learnt.

I never meant to generate an "us vs. them" atmosphere, and re-reading what I wrote, I don't think that's what I did. I think the most contentious think i said was that python's core might value "principle over pragmatism" - that would be a perfectly justifiable thing for you do in some scenarios, even though I disagree in this case.

A few points in my defence:

  • I had already (before Larry emailed me about PEP 649) requested to talk about this at the python developer summit in May, it was on my radar and I didn't realise how little time we had
  • The situation has changed in the last week as Guido himself points out "... the hypothetical future where annotations are not always syntactically expressions (which did not even exist before this week)..." - this change really would be a problem for pydantic and other such libraries
  • I specifically avoided telling people to email a particular address or contact particular people to avoid individuals being spammed, I also specifically suggested people didn't reply to the python-dev thread with "me too"s.

overall: Thank you for taking this seriously and I'm sorry to have caused a fuss.


I'm now going to spend some time doing my best to produce a coherent explanation of the short-comings of PEP 563 for pydantic to post both here and on python-dev (I assume just posting a link to here on the mailing list will annoy people?)

@cheesycod
Copy link

@cheesycod cheesycod commented Apr 16, 2021

Would it be possible to use eval() to bypass this in some way?

Like eval what you can and make the rest internally a ForwardRef or something

@PrettyWood
Copy link
Collaborator

@PrettyWood PrettyWood commented Apr 16, 2021

@cheesycod get_type_hints() uses eval() behind the scene with forward references. Unfortunately eval() is slow, not always available and may not have the necessary context to work. For example if you define a model or a type inside a function, there is no easy way to retrieve it and make it work.
I tried some time ago to test pydantic with 3.10.0a4 in local and had to play with the stack trace on top of get_type_hints() but couldn't make it work all the time. But I thought it was just a matter of time before get_type_hints() was improved or anything else was added to ensure this would still be doable with the new annotation system.
I didn't pay enough attention to the PEPs and since python 3.10 was still in alpha I didn't panic...

@henryiii
Copy link

@henryiii henryiii commented Apr 16, 2021

How important is performance here? I hear "eval is slow", but when looking at that PEP, it looks like while PEP 649 is a little better than Python 3.9, Python 3.10/PEP 563 is much better, as it makes all annotation strings, pretty much leaving you with 0 cost for type annotations - which is affects all code, not just these ~three libraries. If pedantic is a few percent slower but all other code is half a percent faster, you might even win for pydantic libraries, and a big win for all other libraries. There might be some caching tricks to be used here (most of the PEP 563 improvements are for startup / memory usage)?

Also, being able to put invalid objects or usage into type annotations is huge(*); it's the main reason using Python 3.7 is so much better than 3.6 for typed code, since you can use Python 3.9 and 3.10 constructs in your type annotations, even though they had not been developed yet, you just need a recent MyPy or other type checker. To me it looks like PEP 649 nullifies some of that benefit, as accessing an annotation can crash if invalid objects are used, and invalid objects (which might be used eventually, like Guido's --flag example) might also open up new library ideas. Also, under "other uses" in the description, you have "documentation" - that usage is certainly not better with PEP 649 than PEP 563!

I'm not very familiar with pydantic, just a bit with typer; to me pydantic is a better dataclasses/attrs with serialization (I actually just heard about it a couple of days ago before this tweet/issue from a happy user). A MWE for something hard to do would be helpful. "Good" usages of runtime attributes still look like typing, and Pydantic falls into that, so seeing what is hard might really help. Is making a local type a good idea? Is it for a loop or something like that? This seems like it might be very specific to Pydantic, as I can't think of a good reason to make a local type for Typer.

(PS: I'm sort of assuming that PEP 649 requires the object to exist, since the point of this is to capture a reference to the local object. That actually may not be the case...)

@cheesycod
Copy link

@cheesycod cheesycod commented Apr 16, 2021

It appears that there are ways to make eval faster if you are using that route using lambdas: https://stackoverflow.com/questions/12467570/python-way-to-speed-up-a-repeatedly-executed-eval-statement

Could also use a optional C module to speed things up as well in performance

@samuelcolvin
Copy link
Owner Author

@samuelcolvin samuelcolvin commented Apr 16, 2021

I'm now going to spend some time doing my best to produce a coherent explanation of the short-comings of PEP 563 for pydantic to post both here and on python-dev

Well luckily, I was saved the effort. Łukasz Langa (the instigator of PEP 563) has explained it very clearly in another thread on python-dev "PEP 563 in light of PEP 649".

There's another very interesting description of the subtle changes in scope associated with PEP 563, in PEP 649.

I won't go over everything here, but I'll just give one taster of challenges presented by PEP 563:

This works fine:

from __future__ import annotations
from pydantic import BaseModel, PositiveInt

class TestModel(BaseModel):
    foo: PositiveInt

debug(TestModel(foo=1))

But this fails:

from __future__ import annotations

def main():
    from pydantic import BaseModel, PositiveInt

    class TestModel(BaseModel):
        foo: PositiveInt

    debug(TestModel(foo=1))

main()

Why? Because PositiveInt is not defined in the module scope.

Even adding TestModel.update_forward_refs() doesn't help, you need to add TestModel.update_forward_refs(PositiveInt=PositiveInt) (update_forwards_refs() uses logic similar to get_type_hints(), though it doesn't call it).

Note: this isn't a limitation of pydantic or even a true "bug" in python, it is defined (albeit in passing) in PEP 563 "Annotations can only use names present in the module scope..."

This means if pydantic implemented some magic to get the second example to work, it would be contradicting PEP 563.

You might say "well imports like this should always be module level anyway", perhaps true, but what about custom data types which are extensively used with pydantic? There are many good reasons why you might want to use non-module level custom types, currently with python 3.10 that would require complex .update_forward_refs(...) calls.

@WhyNotHugo
Copy link

@WhyNotHugo WhyNotHugo commented Apr 16, 2021

If you value pydantic or FastAPI or other libraries that use annotations at runtime, please (constructively and respectfully) let the python steering council know that you would like PEP 649 to be accepted.

Considering that PEP 563 is from around four years ago, and that the plan to change this on Python 3.10 has been set in stone for so long, it doesn't see right to unleash a mass of complaints to the core Python developers three weeks before the release happens.

The plan to implement and release these changes on Python were clear even before FastAPI was even written. It was clear from that moment that it was not going to be future-proof. Yet here we are, with a last minute campaign trying to push back on this.

@jayqi
Copy link

@jayqi jayqi commented Apr 17, 2021

@samuelcolvin I think it would be helpful for you to update the root post for this issue thread to reflect the current state of things. For example, it seems like the Pydantic project has gotten sufficient attention and engagement from the Python Steering Council and that efforts are in motion to figure it out (that's my perception anyways), so the stuff in "You can help" is probably no longer needed. That way, nobody will inadvertently make unnecessary noise for the SC, and people who are worried about that noise won't keep commenting about it.

Strikethrough (~text~) might be a good way to indicate an update to that information but also preserving its history for transparency.

@samuelcolvin
Copy link
Owner Author

@samuelcolvin samuelcolvin commented Apr 17, 2021

Thanks @jayqi, done.

I'll update the issue again when we hear from the steering council.

@alexmojaki
Copy link

@alexmojaki alexmojaki commented Apr 19, 2021

Note: this isn't a limitation of pydantic or even a true "bug" in python, it is defined (albeit in passing) in PEP 563 "Annotations can only use names present in the module scope..."

This means if pydantic implemented some magic to get the second example to work, it would be contradicting PEP 563.

Is the reasoning behind this bit of the PEP explained somewhere in more detail than "postponed evaluation using local names is not reliable"? What would be wrong with pydantic getting frame.f_locals from where the class is defined (main in your example) and passing that to get_type_hints or update_forward_refs?

You might say "well imports like this should always be module level anyway", perhaps true, but what about custom data types which are extensively used with pydantic? There are many good reasons why you might want to use non-module level custom types, currently with python 3.10 that would require complex .update_forward_refs(...) calls.

I think using locally defined types (including imports) is perfectly fine and useful and allowing it is important. However I can't see how your custom data types link supports your argument, what are you referring to?

@samuelcolvin
Copy link
Owner Author

@samuelcolvin samuelcolvin commented Apr 19, 2021

Is the reasoning behind this bit of the PEP explained somewhere in more detail than "postponed evaluation using local names is not reliable"?

The best explanation I can find is here in PEP 649, if you want more detail you would probably need to dig into the PR or python-dev mailing from the time.

However I can't see how your custom data types link supports your argument, what are you referring to?

Because custom types (class definitions which are used as annotations) as well as nested models, might often be defined outside the module scope.

@alexmojaki
Copy link

@alexmojaki alexmojaki commented Apr 19, 2021

Note: this isn't a limitation of pydantic or even a true "bug" in python, it is defined (albeit in passing) in PEP 563 "Annotations can only use names present in the module scope..."

This means if pydantic implemented some magic to get the second example to work, it would be contradicting PEP 563.

I'm still not sure about this. Again, here's what it says:

Annotations can only use names present in the module scope as postponed evaluation using local names is not reliable (with the sole exception of class-level names resolved by typing.get_type_hints()).

This is under the "Implementation" section. To me this means that implementations are not required to handle "postponed evaluation using local names" automatically for the user. Which makes sense, it'd be impossible, get_type_hints can't guess where to look and by the time it's called the scope may not exist any more. But it still allows passing a local namespace if you want. So I don't think the PEP is forbidding using local names or magically resolving them, just saying that the Python implementation is not required to resolve them for you. Besides, given the confusion about scopes when PEP 563 was accepted, is this still binding?

For pydantic, resolving these is more achievable than for get_type_hints, essentially by resolving the names ASAP (before the scope is deleted) either at class creation time or when update_forward_refs is called. Getting the correct frame isn't trivial but it's not too hard. Users should only notice problems if they write their own helpers wrapping pydantic, especially model class creation, and you can provide an API for such helpers. Here's how I'd solve this:

from __future__ import annotations
import inspect
from pydantic import BaseModel

no_locals_codes = set()

def no_locals_here(f):
    """
    Decorate a function with @no_locals_here
    if it's a wrapper around model class creation
    or update_forward_refs to indicate that the function
    is not the place to look for local type names.
    """
    no_locals_codes.add(f.__code__)
    return f

class AutoBaseModelMeta(type(BaseModel)):
    @no_locals_here
    def __new__(cls, *args, **kwargs):
        result = super().__new__(cls, *args, **kwargs)
        try:
            result.update_forward_refs()
        except NameError:
            pass  # user will have to call update_forward_refs themselves
        return result

class AutoBaseModel(BaseModel, metaclass=AutoBaseModelMeta):
    @classmethod
    def update_forward_refs(cls, **localns):
        frame = inspect.currentframe().f_back
        while frame.f_code in no_locals_codes:
            frame = frame.f_back
        localns = {**frame.f_locals, **localns}
        super().update_forward_refs(**localns)

def main():
    from pydantic import PositiveInt

    class TestModel(AutoBaseModel):
        # An actual forward ref, will raise a NameError in
        # AutoBaseModelMeta.__new__ when calling update_forward_refs,
        # so user still needs to call update_forward_refs
        # but they don't have to specify Sub=Sub
        sub: Sub = None

    class Sub(AutoBaseModel):
        # update_forward_refs fixes this field automatically
        # when the class is defined
        x: PositiveInt

    TestModel.update_forward_refs()

    print(repr(TestModel(sub=Sub(x=2))))

main()

It's not ideal, I prefer PEP 649, but it's a start and it'd be helpful even for current users.

@willingc
Copy link

@willingc willingc commented Apr 20, 2021

@samuelcolvin @tiangolo and others following this thread,

Thanks for your patience. The Python Steering Council has posted the plans for 3.10 on python-dev mailing list.

@PrettyWood
Copy link
Collaborator

@PrettyWood PrettyWood commented Apr 20, 2021

Thanks a lot to the SC! ❤️ And thank you @samuelcolvin @tiangolo and everyone else for warning the SC just in time! I'm sorry some people felt wronged with this matter and that some extra work is needed to roll back PEP 563 as default. But for what it's worth I'm really glad it turned out well. Postponing the breaking change seems indeed like the most reasonable solution for now.

@tiangolo
Copy link
Collaborator

@tiangolo tiangolo commented Apr 20, 2021

This is amazing news! 🎉

On behalf of the FastAPI community, thank you @willingc, the steering council, and everyone involved! 🙇

I understand the big effort and commitment the steering council is making with this decision to support the pydantic and FastAPI communities (especially with our short notice), and the last-minute extra work involved in reverting the default for 3.10 (sorry @pablogsal 😬😅). Thanks for that.

This will give us all the time to figure out how to handle it taking into account pydantic/FastAPI and the use cases that would benefit from PEP 563 and PEP 649. 🚀

@pablogsal
Copy link

@pablogsal pablogsal commented Apr 20, 2021

and the last-minute extra work involved in reverting the default for 3.10 (sorry @pablogsal 😬😅). Thanks for that.

Well, I voted for it so I cannot complain 😉 Sometimes being in the SC and being the release manager at the same time ends with me voting to give myself more work 😅

But please, next time make sure to voice your concerns and thoughts in time so we have enough window to operate and consider all aspects of change.

@samuelcolvin
Copy link
Owner Author

@samuelcolvin samuelcolvin commented Apr 20, 2021

Thank you all so much for listening to us and making a hard decision! 🙇

I'm really sorry we wasted so much of your time @pablogsal by not bringing this up earlier. 🙏

This has been a really positive experience for me and I feel much more positive about talking to the python-dev community.

In future I will engage with the release process and treat it like something I'm a (very small) part of, rather than something that happens to me.

@boweeb
Copy link

@boweeb boweeb commented Apr 26, 2021

From linke above, python-dev mailing list (emphasis mine):

In the Steering Council’s unanimous opinion, rolling back the default flip
for stringified annotations in Python 3.10 is the least disruptive of all
the options.

I choked when I read "unanimous". Others' well-wishes have already been better articulated than I could, but I wanted to highlight this.

My sincere thanks to everyone involved!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet