Synapse does not take advantage of Pydantic 2.0

Description

After upgrading the PIP packages, Synapse failed to launch.

Steps to reproduce

install matrix-synapse using the "virtualenv" approach with the most recent dependency versions
start Synapse

Homeserver

another homeserver

Synapse Version

1.87.0rc1

Installation Method

pip (from PyPI)

Database

PostgreSQL

Workers

Single process

Platform

Raspberry Pi OS (Debian)

Configuration

No response

Relevant log output

/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/pydantic/_internal/_config.py:257: UserWarning: Valid config keys have changed in V2:                                                            
* 'allow_mutation' has been removed                                                
  warnings.warn(message, UserWarning)                                              
Traceback (most recent call last):                                                 
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main             
    return _run_code(code, main_globals, None,                                     
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code                        
    exec(code, run_globals)                                                        
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/app/homeserver.py", line 37, in <module>                                                                                         
    from synapse.app import _base                                                  
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/app/_base.py", line 68, in <module>                                                                                              
    from synapse.handlers.auth import load_legacy_password_auth_providers          
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/handlers/auth.py", line 56, in <module>                                                                                          
    from synapse.api.ratelimiting import Ratelimiter                              
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/api/ratelimiting.py", line 21, in <module>                                                                                       
    from synapse.storage.databases.main import DataStore                           
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/__init__.py", line 34, in <module>                                                                                       
    from synapse.storage.databases import Databases                                
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/__init__.py", line 20, in <module>                                                                             
    from synapse.storage.databases.main.events import PersistEventsStore           
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/__init__.py", line 27, in <module>                                                                        
    from synapse.storage.databases.main.stats import UserSortOrder                 
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/stats.py", line 42, in <module>                                                                           
    from synapse.storage.databases.main.events_worker import InvalidEventError     
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/events_worker.py", line 62, in <module>
    from synapse.replication.tcp.streams import BackfillStream, UnPartialStatedEventStream
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/tcp/streams/__init__.py", line 27, in <module>
    from synapse.replication.tcp.streams._base import (
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/tcp/streams/_base.py", line 32, in <module>
    from synapse.replication.http.streams import ReplicationGetStreamUpdates
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/__init__.py", line 18, in <module>
    from synapse.replication.http import (
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/account_data.py", line 21, in <module>
    from synapse.replication.http._base import ReplicationEndpoint
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/http/_base.py", line 31, in <module>
    from synapse.http.servlet import parse_json_object_from_request
  File "/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/http/servlet.py", line 32, in <module>
    from pydantic import BaseModel, MissingError, PydanticValueError, ValidationError
ImportError: cannot import name 'MissingError' from 'pydantic' (/usr/local/opt/venvs/matrix-synapse/lib/python3.9/site-packages/pydantic/__init__.py)
systemd[1]: matrix-synapse.service: Control process exited, code=exited, status=1/FAILURE

Anything else that would be useful to know?

pydantic 2.0 was released yesterday

Workaround: run pip install 'pydantic<2.0' in the virtual environment.

Workaround: run pip install 'pydantic<2.0' in the virtual environment.

Suggest we specify this in pyproject.toml, or change

synapse/pyproject.toml

Lines 209 to 210 in 07fd6d8

    
           # This is the most recent version of Pydantic with available on common distros. 
        
           pydantic = ">=1.7.4"

to ^=1.7.4.

Not sure why I didn't do this when starting to use Pydantic. I think because generally speaking it makes packagers' lives easier if we don't impose upper bounds?

Just encountered this, and in other projects as well on Friday morning. I agree with pinning to <2.0.0 as a workaround; hoping to make a PR for this shortly.

Just created: #15862

net-im/synapse: restrict rdeps to <pillow-10 and <pydantic-2

Closing this as it looks like it is resolved by #15862

net-im/synapse: restrict rdeps to <pillow-10 and <pydantic-2

Hi! We're currently blocked by matrix-synapse in our pydantic 2 rebuild for Arch Linux. Please reopen this ticket so that discussion around how/when/if to switch to pydantic2 can happen.

The change in #15862 is merely a workaround and not the solution to this ticket (which would be "be compatible with pydantic >= 2").

Thanks! :)

Re-opening per the above -- pinning to <= 2.0 isn't supporting Pydantic 2.0.

Pydantic V2 includes V1 built-in to help migration.

We could use the following all over the place as a transitional measure.

try:
    from pydantic import v1 as pydantic
except ImportError:
    import pydantic

Given that V1 and V2 use the same Python package name (⇒ you can't have them both installed at the same time), we sadly can expect some trouble with distro packages that would make switching to using V2 proper difficult for some time.

We could use the following all over the place as a transitional measure.

I'm not sure if that really works as intended. There are some implied breakages AFAIK irt the validators.

I can earliest look at my open PR again after 2023-08-21. If someone manages to fix this earlier and create a new release, that would be much appreciated, as synapse is currently the only package blocking us from moving to pydantic v2 on Arch Linux: https://archlinux.org/todo/python-pydantic-2/

Merge tag 'v1.87.0' into develop

We could use the following all over the place as a transitional measure.

I'm not sure if that really works as intended. There are some implied breakages AFAIK irt the validators.

The pydantic.v1 module is simply a copy of the old pydantic v1 code, so there should not be breakages there.

I opened #16332 to add support for pydantic v2 while maintaining support for v1.

I opened #16332 to add support for pydantic v2 while maintaining support for v1.

I re-titled this since Synapse is now compatible, but not taking advantage of pydantic v2.

Navigation Menu

Repository navigation

Synapse does not take advantage of Pydantic 2.0 #15858

Description

Steps to reproduce

Homeserver

Synapse Version

Installation Method

Database

Workers

Platform

Configuration

Relevant log output

Anything else that would be useful to know?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	# This is the most recent version of Pydantic with available on common distros.
	pydantic = ">=1.7.4"

Synapse does not take advantage of Pydantic 2.0 #15858

Description

Description

Steps to reproduce

Homeserver

Synapse Version

Installation Method

Database

Workers

Platform

Configuration

Relevant log output

Anything else that would be useful to know?

Activity

c7hm4r commented on Jul 1, 2023

DMRobertson commented on Jul 2, 2023

PaarthShah commented on Jul 2, 2023

PaarthShah commented on Jul 2, 2023

H-Shay commented on Jul 5, 2023

dvzrv commented on Jul 23, 2023

clokep commented on Jul 24, 2023

reivilibre commented on Aug 7, 2023

dvzrv commented on Aug 13, 2023

gotmax23 commented on Sep 11, 2023

gotmax23 commented on Sep 17, 2023

clokep commented on Oct 2, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions