Add support for pydantic v2 via pydantic.v1 compat module #16332

gotmax23 · 2023-09-17T04:36:59Z

Add support for pydantic v2 via pydantic.v1 compat module

This adds support for pydantic v2 by using the pydantic.v1 compat module
when pydantic v2 is installed and otherwise falling back to the pydantic module.

In order to satisfy mypy, we only import pydantic.v1 during type
checking. This allows proper type checking without needing to clutter
the code with type: ignores.

Once some time is passed, we can entirely drop compatibility for pydantic v1 and
start using the actual pydantic v2 code to reap the additional
performance benefits and other improvements of pydantic v2.

Relates: #15858

(I didn't use Fixes:, as we should track enabling full support for pydantic v2)

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Pull request includes a sign off
Code style is correct
(run the linters)

gotmax23 · 2023-09-17T04:38:11Z

poetry.lock

This was generated with poetry update pydantic so other dependencies weren't touched here.

clokep · 2023-09-18T11:21:55Z

See also #16331.

pyproject.toml

synapse/config/_util.py

gotmax23 · 2023-09-21T06:01:39Z

@clokep, do you mind approving the CI run?

synapse/_pydantic_compat.py

synapse/events/validator.py

clokep · 2023-09-21T12:26:35Z

It looks like one of the CI jobs is failing too: https://github.com/matrix-org/synapse/actions/runs/6230975624/job/16994483499#step:10:32

gotmax23 · 2023-09-21T15:24:09Z

It looks like one of the CI jobs is failing too: https://github.com/matrix-org/synapse/actions/runs/6230975624/job/16994483499#step:10:32

Those tests are running with packaging==16.1 from 2016 😨. Can this be updated to at least version 20 from 2019 that has pypa/packaging@f1b2b22?

This adds support for pydantic v2 by using the `pydantic.v1` compat module when pydantic v2 is installed and otherwise falling back to the `pydantic` module. In order to satisfy mypy, we only import `pydantic.v1` during type checking. This allows proper type checking without needing to clutter the code with `type: ignores`. Once some time is passed, we can entirely drop compatibility for pydantic v1 and start using the actual pydantic v2 code to reap the additional performance benefits and other improvements of pydantic v2. Signed-off-by: Maxwell G <maxwell@gtmx.me>

Old pydantic v1 versions do not define `__version__`. Signed-off-by: Maxwell G <maxwell@gtmx.me>

gotmax23 · 2023-09-21T16:49:22Z

I pushed to address the merge conflicts and feedback. @clokep, can you please approve CI again 🙂?

gotmax23 · 2023-09-21T16:50:01Z

It looks like one of the CI jobs is failing too: https://github.com/matrix-org/synapse/actions/runs/6230975624/job/16994483499#step:10:32

Those tests are running with packaging==16.1 from 2016 😨. Can this be updated to at least version 20 from 2019 that has pypa/packaging@f1b2b22?

I bumped the minimum packaging version.

pyproject.toml

changelog.d/16332.feature

Signed-off-by: Maxwell G <maxwell@gtmx.me>

Apparently, towncrier doesn't like this. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

clokep

This should be fine, just double checking that packaging >= 20 won't cause issues.

6543 · 2023-09-23T08:09:15Z

Alpine packager: ACK

arkamar · 2023-09-23T16:31:50Z

Gentoo: ACK, tested with pydantic 1.10.12 and 2.3.0.

clokep · 2023-09-25T13:10:02Z

Just need CI to churn and this is good to go. 👍

gotmax23 · 2023-09-25T19:39:22Z

Thanks @clokep!

The <dev-python/pydantic-2 restriction was removed as this version supports new releases as well [1]. [1] matrix-org/synapse#16332 Signed-off-by: Petr Vaněk <arkamar@atlas.cz>

The <dev-python/pydantic-2 restriction was removed as this version supports new releases as well [1]. [1] matrix-org/synapse#16332 Signed-off-by: Petr Vaněk <arkamar@atlas.cz> Closes: #33279 Signed-off-by: Sam James <sam@gentoo.org>

@gotmax23

No significant changes since 1.94.0rc1. - Render plain, CSS, CSV, JSON and common image formats in the browser (inline) when requested through the /download endpoint. ([\matrix-org#15988](matrix-org#15988)) - Add experimental support for [MSC4028](matrix-org/matrix-spec-proposals#4028) to push all encrypted events to clients. ([\matrix-org#16361](matrix-org#16361)) - Minor performance improvement when sending presence to federated servers. ([\matrix-org#16385](matrix-org#16385)) - Minor performance improvement by caching server ACL checking. ([\matrix-org#16360](matrix-org#16360)) - Add developer documentation concerning gradual schema migrations with column alterations. ([\matrix-org#15691](matrix-org#15691)) - Improve documentation of the user directory search algorithm. ([\matrix-org#16320](matrix-org#16320)) - Fix rendering of user admin API documentation around deactivation. This was broken in Synapse 1.91.0. ([\matrix-org#16355](matrix-org#16355)) - Update documentation around message retention policies. ([\matrix-org#16382](matrix-org#16382)) - Add note to `federation_domain_whitelist` config option to clarify its usage. ([\matrix-org#16416](matrix-org#16416)) - Improve legacy release notes. ([\matrix-org#16418](matrix-org#16418)) - Remove Python version from `/_synapse/admin/v1/server_version`. ([\matrix-org#16380](matrix-org#16380)) - Avoid running CI steps when the files they check have not been changed. ([\matrix-org#14745](matrix-org#14745), [\matrix-org#16387](matrix-org#16387)) - Improve type hints. ([\matrix-org#14911](matrix-org#14911), [\matrix-org#16350](matrix-org#16350), [\matrix-org#16356](matrix-org#16356), [\matrix-org#16395](matrix-org#16395)) - Added support for pydantic v2 in addition to pydantic v1. Contributed by Maxwell G (@gotmax23). ([\matrix-org#16332](matrix-org#16332)) - Get CI to check PRs have been signed-off. ([\matrix-org#16348](matrix-org#16348)) - Add missing licence header. ([\matrix-org#16359](matrix-org#16359)) - Improve type hints, and bump types-psycopg2 from 2.9.21.11 to 2.9.21.14. ([\matrix-org#16381](matrix-org#16381)) - Improve comments in `StateGroupBackgroundUpdateStore`. ([\matrix-org#16383](matrix-org#16383)) - Update maturin configuration. ([\matrix-org#16394](matrix-org#16394)) - Downgrade replication stream time out error log lines to warning. ([\matrix-org#16401](matrix-org#16401)) * Bump actions/checkout from 3 to 4. ([\matrix-org#16250](matrix-org#16250)) * Bump cryptography from 41.0.3 to 41.0.4. ([\matrix-org#16362](matrix-org#16362)) * Bump dawidd6/action-download-artifact from 2.27.0 to 2.28.0. ([\matrix-org#16374](matrix-org#16374)) * Bump docker/setup-buildx-action from 2 to 3. ([\matrix-org#16375](matrix-org#16375)) * Bump gitpython from 3.1.35 to 3.1.37. ([\matrix-org#16376](matrix-org#16376)) * Bump msgpack from 1.0.5 to 1.0.6. ([\matrix-org#16377](matrix-org#16377)) * Bump msgpack from 1.0.6 to 1.0.7. ([\matrix-org#16412](matrix-org#16412)) * Bump phonenumbers from 8.13.19 to 8.13.22. ([\matrix-org#16413](matrix-org#16413)) * Bump psycopg2 from 2.9.7 to 2.9.8. ([\matrix-org#16409](matrix-org#16409)) * Bump pydantic from 2.3.0 to 2.4.2. ([\matrix-org#16410](matrix-org#16410)) * Bump regex from 1.9.5 to 1.9.6. ([\matrix-org#16408](matrix-org#16408)) * Bump sentry-sdk from 1.30.0 to 1.31.0. ([\matrix-org#16378](matrix-org#16378)) * Bump types-netaddr from 0.8.0.9 to 0.9.0.1. ([\matrix-org#16411](matrix-org#16411)) * Bump types-psycopg2 from 2.9.21.11 to 2.9.21.14. ([\matrix-org#16381](matrix-org#16381)) * Bump urllib3 from 1.26.15 to 1.26.17. ([\matrix-org#16422](matrix-org#16422)) # -----BEGIN PGP SIGNATURE----- # # iQIzBAABCAAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmUlINwACgkQLS76LzL7 # 4EdvExAAgjk6+/Fu45MRG7u5kFmFzoZWLOPD10XROANaSeqW1l/pBhFh+XvwR4TZ # l/FdkSfS9YpHnw3aof13TclLu6IVWDM+vqYFuY2HSY/yzbcGvJFHqr26kOccpTTd # 2r9m/AkguyHEBECDW8qJLXb8M7dqNa2SydTBu1+IrKfj6nq+fRxVyQhyAJXrI1Ta # Dnz0XJ4TcwTrMPVk4MYrAcYjID6IV89dtp7ttH4DwXKDeSjMtxM/46EIg4u+VXDz # fzK25JHVFYJA5+/rOn/RslmxjJHQfEIEB6NYxQwLeMeZuGSZooTebKn1odwogvhI # Srtfsytum+twgSHD1s+7KldM+EjTiu7ouKi8VcfOlFuLnuBiROEc5WUljcL5K63F # kVx2bXGU/eNkPp6ntNhYfgswx+yk2rXFqkTjz+xZQIZcOBqehHBDy8VhtwlRkTUw # bzocdKkLMA4nfSlq5fFOAErMqJKsPS8aN9yYPShqEUiSUOKle8eHfA1cTXJuK0MS # K2/YcDDZmJBrwVADyNDk5GKaDx39rR752OSuJb57Sp/edwUg6+H1I6lIN6YTeoJw # FzJwGMzuMCktOQRW2enxQiA6RZjXFCwvD1LoWMjyO4YTXQwXxNCXsb0kLKUqfwsy # qMGphWEl3rdzVSuFapNAgOLF0RfFNYZdhQnk+3fNEwxumxoqgho= # =hx8G # -----END PGP SIGNATURE----- # gpg: Signature made Tue Oct 10 11:01:00 2023 BST # gpg: using RSA key F124520CEEE062448FE1C8442D2EFA2F32FBE047 # gpg: Can't check signature: No public key # Conflicts: # .github/workflows/docker.yml # .github/workflows/docs-pr-netlify.yaml # .github/workflows/docs-pr.yaml # .github/workflows/docs.yaml # .github/workflows/latest_deps.yml # .github/workflows/poetry_lockfile.yaml # .github/workflows/push_complement_image.yml # .github/workflows/release-artifacts.yml # .github/workflows/tests.yml # .github/workflows/twisted_trunk.yml # poetry.lock # rust/src/push/base_rules.rs

gotmax23 requested a review from a team as a code owner September 17, 2023 04:36

gotmax23 commented Sep 17, 2023

View reviewed changes

poetry.lock Outdated

Copy link

Contributor Author

gotmax23 Sep 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was generated with poetry update pydantic so other dependencies weren't touched here.

gotmax23 mentioned this pull request Sep 17, 2023

Synapse does not take advantage of Pydantic 2.0 #15858

Open

clokep mentioned this pull request Sep 18, 2023

from pydantic import v1 as pydantic #16331

Closed

4 tasks

clokep reviewed Sep 18, 2023

View reviewed changes

pyproject.toml Show resolved Hide resolved

clokep reviewed Sep 18, 2023

View reviewed changes

synapse/config/_util.py Show resolved Hide resolved

gotmax23 force-pushed the pydantic-v1-v2 branch from 1dd3295 to 296bbf0 Compare September 19, 2023 04:02

clokep requested a review from a team September 19, 2023 11:21

clokep reviewed Sep 21, 2023

View reviewed changes

synapse/_pydantic_compat.py Show resolved Hide resolved

synapse/events/validator.py Outdated Show resolved Hide resolved

gotmax23 added 2 commits September 21, 2023 16:47

_pydantic_compat: fix HAS_PYDANTIC_V2 version check

319fc2b

Old pydantic v1 versions do not define `__version__`. Signed-off-by: Maxwell G <maxwell@gtmx.me>

gotmax23 force-pushed the pydantic-v1-v2 branch from 296bbf0 to a3f4f59 Compare September 21, 2023 16:48

clokep reviewed Sep 21, 2023

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

clokep reviewed Sep 21, 2023

View reviewed changes

changelog.d/16332.feature Outdated Show resolved Hide resolved

pyproject.toml: bump packaging minimum version to 20.0.

2dcd89f

Signed-off-by: Maxwell G <maxwell@gtmx.me>

gotmax23 force-pushed the pydantic-v1-v2 branch from a3f4f59 to 2dcd89f Compare September 21, 2023 18:36

Remove newline from changelog entry

4707549

Apparently, towncrier doesn't like this. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

clokep approved these changes Sep 22, 2023

View reviewed changes

clokep added 3 commits September 25, 2023 09:04

Merge remote-tracking branch 'origin/develop' into pull/16332/head

0bb7f3d

Update packaging comment.

7f9482e

Newsfragment

ac41f59

clokep enabled auto-merge (squash) September 25, 2023 15:01

clokep merged commit 12611bf into matrix-org:develop Sep 25, 2023
38 checks passed

clokep mentioned this pull request Sep 25, 2023

Update pydantic to >= 2 #15979

Open

4 tasks

gnunicorn mentioned this pull request Nov 14, 2023

Error with pydantic in latest release julianhille/matrix-synapse-testutils#38

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for pydantic v2 via pydantic.v1 compat module #16332

Add support for pydantic v2 via pydantic.v1 compat module #16332

gotmax23 commented Sep 17, 2023

gotmax23 Sep 17, 2023

clokep commented Sep 18, 2023

gotmax23 commented Sep 21, 2023

clokep commented Sep 21, 2023

gotmax23 commented Sep 21, 2023

gotmax23 commented Sep 21, 2023

gotmax23 commented Sep 21, 2023

clokep left a comment •

edited

Loading

6543 commented Sep 23, 2023

arkamar commented Sep 23, 2023

clokep commented Sep 25, 2023

gotmax23 commented Sep 25, 2023

Add support for pydantic v2 via pydantic.v1 compat module #16332

Add support for pydantic v2 via pydantic.v1 compat module #16332

Conversation

gotmax23 commented Sep 17, 2023

Pull Request Checklist

gotmax23 Sep 17, 2023

Choose a reason for hiding this comment

clokep commented Sep 18, 2023

gotmax23 commented Sep 21, 2023

clokep commented Sep 21, 2023

gotmax23 commented Sep 21, 2023

gotmax23 commented Sep 21, 2023

gotmax23 commented Sep 21, 2023

clokep left a comment • edited Loading

Choose a reason for hiding this comment

6543 commented Sep 23, 2023

arkamar commented Sep 23, 2023

clokep commented Sep 25, 2023

gotmax23 commented Sep 25, 2023

clokep left a comment •

edited

Loading