Use __getattr__ for all package imports, improve import time

[samuelcolvin]

Change Summary

This means pydantic/__init__.py is now lazy about all imports using __getattr__.

Import performance changes:

Before

code/pydantic 0 ➤ pdm run python -m timeit -n 1 -r 1 "import pydantic"
1 loop, best of 1: 29.7 msec per loop

code/pydantic 0 ➤ pdm run python -m timeit -n 1 -r 1 "from pydantic import BaseModel"
1 loop, best of 1: 30.4 msec per loop

After

code/pydantic 0 ➤ pdm run python -m timeit -n 1 -r 1 "import pydantic"
1 loop, best of 1: 4.8 msec per loop

(env3112) code/pydantic 0 ➤ pdm run python -m timeit -n 1 -r 1 "from pydantic import BaseModel"           
1 loop, best of 1: 18.7 msec per loop

Tuna output:

Related issue number

As promised in #7423 (comment).

Checklist

• The pull request title is a good summary of the changes - it will be used in the changelog
• Unit tests for the changes exist
• Tests pass on CI
• Documentation reflects the changes where applicable
• My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

Selected Reviewer: @sydney-runkle

[cloudflare-pages]

Deploying with    Cloudflare Pages

Latest commit:	9733589
Status:	✅  Deploy successful!
Preview URL:	https://827630e5.pydantic-docs2.pages.dev
Branch Preview URL:	https://defer-init-imports.pydantic-docs2.pages.dev

View logs

[ofek]

Awesome thank you, definitely a step in the right direction!

❯ pip install -Uq pydantic
❯ python -m timeit -n 1 -r 1 "import pydantic"
1 loop, best of 1: 83.4 msec per loop
❯ python -m timeit -n 1 -r 1 "from pydantic import BaseModel"
1 loop, best of 1: 89.6 msec per loop
❯ pip install -Iq "pydantic @ git+https://github.com/pydantic/pydantic.git@defer-init-imports"
❯ python -m timeit -n 1 -r 1 "import pydantic"
1 loop, best of 1: 15.7 msec per loop
❯ python -m timeit -n 1 -r 1 "from pydantic import BaseModel"
1 loop, best of 1: 79 msec per loop

Do you see a way to reduce the import cost of BaseModel any further? That is what I, and I think most others, are optimizing for because that is the main interface that classes in one's code will inherit at import time.

[ofek]

That last commit helped a little!

❯ python -m timeit -n 1 -r 1 "from pydantic import BaseModel"
1 loop, best of 1: 63.9 msec per loop

[samuelcolvin]

@ofek that's great, I was able to avoid importing pydantic.fields in that which should have helped a fair bit.

Could you show me the output of importtime using tuna from your system, that might help me know where to look.

[ofek]

[samuelcolvin]

@ofek I've got rid of import time import of annotated-types, with that I see a >38% improvement. I really don't think there's much more I can do from there.

If you see anything else, let me know.

[samuelcolvin]

I think this is ready.

Please review @sydney-runkle

[ofek]

Thank you!

❯ python -m timeit -n 1 -r 1 "from pydantic import BaseModel"
1 loop, best of 1: 56.8 msec per loop

[sydney-runkle]

Nice! LGTM. Solid performance improvements 🏎️

One question - does this merit any update to the docs? I'm guessing not, that a change log entry is enough, but just wanted to check. I'm pretty sure 99% of users aren't going to be digging around for this kind of info.

[samuelcolvin]

I don't think anything in logs, I think it merits a prominent notice in the release notes and a tweet from me now saying we're doing it.

AFAIK this shouldn't affect anyone and I made the change in few steps to see, but the only real answer is to release it and see.

[ofek]

Out of curiosity, how will the practice of being conscious about import time be maintained during continued development?

[samuelcolvin]

I've added unit tests that that those modules aren't imported in specific cases, e.g. from pydantic import BaseModel.

The longer term solution is all down to @art049 - we should use Codspeed for continuous profile of stuff we care about like import time, as we do on pydantic-core.