Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

synapse_port_db Python type error on rejection_status_changed #15098

Closed
fmielicki opened this issue Feb 19, 2023 · 1 comment · Fixed by #15108
Closed

synapse_port_db Python type error on rejection_status_changed #15098

fmielicki opened this issue Feb 19, 2023 · 1 comment · Fixed by #15108
Labels
A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db A-SQLite Database issues specific to SQLite T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Regression Something broke which worked on a previous release

Comments

@fmielicki
Copy link

Description

When running the synapse_port_db script, I'm getting thrown a type error.

Synapse version: server_version: 1.77.0, python_version: 3.9.2
PostgreSQL version: psql (PostgreSQL) 13.9 (Debian 13.9-0+deb11u1)
OS: Linux debian 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

Command used: sudo synapse_port_db --sqlite-database /home/sysadmin/homeserver.db.backup --postgres-config ./homeserver.yaml --curses

Steps to reproduce

  • Copy homeserver.db to another location
  • Shut down Synapse
  • Change homeserver.yaml to reflect PostgreSQL config
  • Run synapse_port_db script

Homeserver

Personal Homeserver

Synapse Version

{"server_version":"1.77.0","python_version":"3.9.2"}

Installation Method

Other (please mention below)

Database

SQLite, attempting to port to Postgres 13.9 (Debian). Fresh installation with only a couple of messages.

Workers

Single process

Platform

Installed using my install script (https://github.com/ephidrineon/matrix-synapse-easy-install) using Debian packages.

Running on a VPS (1 Core, 1 GB RAM, 25 GB /), Debian 11. Virtualized using KVM

Configuration

No response

Relevant log output

port-synapse.log:

2023-02-19 16:46:55,364 - synapse.config.key - 153 - WARNING - This server is configured to use 'matrix.org' as its trusted key server via the
'trusted_key_servers' config option. 'matrix.org' is a good choice for a key
server since it is long-lived, stable and trusted. However, some admins may
wish to use another server for this purpose.

To suppress this warning and continue using 'matrix.org', admins should set
'suppress_key_server_warning' to 'true' in homeserver.yaml.
--------------------------------------------------------------------------------
2023-02-19 16:46:55,369 - synapse.config.key - 169 - WARNING - Config is missing macaroon_secret_key
2023-02-19 16:46:55,440 - synapse.storage.prepare_database - 115 - INFO - ('main', 'state'): Checking existing schema version
2023-02-19 16:46:55,442 - synapse.storage.prepare_database - 119 - INFO - ('main', 'state'): Existing schema is 73 (+22 deltas)
2023-02-19 16:46:55,442 - synapse.storage.databases.main - 286 - INFO - Checking database for consistency with configuration...
2023-02-19 16:46:55,443 - synapse.storage.prepare_database - 411 - INFO - Applying schema deltas for v73
2023-02-19 16:46:55,445 - synapse.storage.prepare_database - 411 - INFO - Applying schema deltas for v74
2023-02-19 16:46:55,445 - synapse.storage.prepare_database - 546 - INFO - Schema now up to date
2023-02-19 16:46:55,450 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for receipts_linearized(stream_id): 615
2023-02-19 16:46:55,451 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for events(stream_ordering): 577
2023-02-19 16:46:55,451 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for events(stream_ordering): -241
2023-02-19 16:46:55,452 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for ex_outlier_stream(event_stream_ordering): -244
2023-02-19 16:46:55,455 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for un_partial_stated_event_stream(stream_id): 9
2023-02-19 16:46:55,458 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for pushers(id): 46
2023-02-19 16:46:55,458 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for deleted_pushers(stream_id): 1
2023-02-19 16:46:55,459 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for push_rules_stream(stream_id): 1
2023-02-19 16:46:55,460 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for push_rules(id): 1
2023-02-19 16:46:55,460 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for push_rules_enable(id): 1
2023-02-19 16:46:55,460 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for room_account_data(stream_id): 372
2023-02-19 16:46:55,460 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for room_tags_revisions(stream_id): 1
2023-02-19 16:46:55,461 - synapse.storage.databases.main.event_push_actions - 1291 - INFO - Searching for stream ordering 1 month ago
2023-02-19 16:46:55,462 - synapse.storage.databases.main.event_push_actions - 1295 - INFO - Found stream ordering 1 month ago: it's 0
2023-02-19 16:46:55,462 - synapse.storage.databases.main.event_push_actions - 1298 - INFO - Searching for stream ordering 1 day ago
2023-02-19 16:46:55,462 - synapse.storage.databases.main.event_push_actions - 1302 - INFO - Found stream ordering 1 day ago: it's 453
2023-02-19 16:46:55,493 - synapse.storage.prepare_database - 115 - INFO - ('main', 'state'): Checking existing schema version
2023-02-19 16:46:55,495 - synapse.storage.prepare_database - 119 - INFO - ('main', 'state'): Existing schema is 74 (+0 deltas)
2023-02-19 16:46:55,495 - synapse.storage.databases.main - 286 - INFO - Checking database for consistency with configuration...
2023-02-19 16:46:55,496 - synapse.storage.prepare_database - 411 - INFO - Applying schema deltas for v74
2023-02-19 16:46:55,497 - synapse.storage.prepare_database - 546 - INFO - Schema now up to date
2023-02-19 16:46:55,524 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for pushers(id): 46
2023-02-19 16:46:55,525 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for deleted_pushers(stream_id): 1
2023-02-19 16:46:55,525 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for push_rules_stream(stream_id): 1
2023-02-19 16:46:55,526 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for push_rules(id): 1
2023-02-19 16:46:55,527 - synapse.storage.util.id_generators - 92 - INFO - Initialising stream generator for push_rules_enable(id): 1
2023-02-19 16:46:55,533 - synapse.storage.databases.main.event_push_actions - 1291 - INFO - Searching for stream ordering 1 month ago
2023-02-19 16:46:55,537 - synapse.storage.databases.main.event_push_actions - 1295 - INFO - Found stream ordering 1 month ago: it's 0
2023-02-19 16:46:55,537 - synapse.storage.databases.main.event_push_actions - 1298 - INFO - Searching for stream ordering 1 day ago
2023-02-19 16:46:55,542 - synapse.storage.databases.main.event_push_actions - 1302 - INFO - Found stream ordering 1 day ago: it's 453
2023-02-19 16:46:55,786 - synapse_port_db - 759 - INFO - Found 146 tables
2023-02-19 16:46:57,463 - synapse_port_db - 381 - INFO - Table e2e_device_keys_json: 0/2 (rows 1-0) already ported
2023-02-19 16:46:57,464 - synapse_port_db - 381 - INFO - Table users_in_public_rooms: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,465 - synapse_port_db - 381 - INFO - Table deleted_pushers: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,465 - synapse_port_db - 381 - INFO - Table event_to_state_groups: 697/697 (rows 1-697) already ported
2023-02-19 16:46:57,465 - synapse_port_db - 381 - INFO - Table user_ips: 0/6 (rows 1-0) already ported
2023-02-19 16:46:57,466 - synapse_port_db - 381 - INFO - Table pushers: 0/1 (rows 1-0) already ported
2023-02-19 16:46:57,466 - synapse_port_db - 381 - INFO - Table device_federation_outbox: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,466 - synapse_port_db - 381 - INFO - Table application_services_txns: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,467 - synapse_port_db - 381 - INFO - Table room_tags_revisions: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,467 - synapse_port_db - 381 - INFO - Table current_state_events: 0/141 (rows 1-0) already ported
2023-02-19 16:46:57,467 - synapse_port_db - 381 - INFO - Table threepid_guest_access_tokens: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,467 - synapse_port_db - 381 - INFO - Table appservice_room_list: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,467 - synapse_port_db - 381 - INFO - Table user_filters: 0/2 (rows 1-0) already ported
2023-02-19 16:46:57,468 - synapse_port_db - 381 - INFO - Table user_daily_visits: 0/13 (rows 1-0) already ported
2023-02-19 16:46:57,469 - synapse_port_db - 381 - INFO - Table event_backward_extremities: 0/4 (rows 1-0) already ported
2023-02-19 16:46:57,470 - synapse_port_db - 381 - INFO - Table room_stats_state: 0/11 (rows 1-0) already ported
2023-02-19 16:46:57,473 - synapse_port_db - 381 - INFO - Table threepid_validation_token: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,473 - synapse_port_db - 381 - INFO - Table blocked_rooms: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,473 - synapse_port_db - 381 - INFO - Table erased_users: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,473 - synapse_port_db - 381 - INFO - Table room_retention: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,474 - synapse_port_db - 381 - INFO - Table sessions: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,474 - synapse_port_db - 381 - INFO - Table e2e_room_keys: 0/22 (rows 1-0) already ported
2023-02-19 16:46:57,474 - synapse_port_db - 381 - INFO - Table device_lists_remote_cache: 0/20 (rows 1-0) already ported
2023-02-19 16:46:57,475 - synapse_port_db - 381 - INFO - Table insertion_events: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,475 - synapse_port_db - 381 - INFO - Table event_auth: 572/572 (rows 1-572) already ported
2023-02-19 16:46:57,476 - synapse_port_db - 381 - INFO - Table event_auth_chain_links: 0/441 (rows 1-0) already ported
2023-02-19 16:46:57,476 - synapse_port_db - 381 - INFO - Table event_push_actions: 0/4 (rows 1-0) already ported
2023-02-19 16:46:57,476 - synapse_port_db - 381 - INFO - Table room_depth: 0/11 (rows 1-0) already ported
2023-02-19 16:46:57,476 - synapse_port_db - 381 - INFO - Table presence_stream: 6/6 (rows 1-77) already ported
2023-02-19 16:46:57,476 - synapse_port_db - 381 - INFO - Table state_groups: 154/154 (rows 1-154) already ported
2023-02-19 16:46:57,477 - synapse_port_db - 381 - INFO - Table e2e_cross_signing_signatures: 0/2 (rows 1-0) already ported
2023-02-19 16:46:57,477 - synapse_port_db - 381 - INFO - Table redactions: 1/1 (rows 1-1) already ported
2023-02-19 16:46:57,477 - synapse_port_db - 381 - INFO - Table profiles: 0/1 (rows 1-0) already ported
2023-02-19 16:46:57,477 - synapse_port_db - 381 - INFO - Table device_lists_remote_extremeties: 0/6 (rows 1-0) already ported
2023-02-19 16:46:57,477 - synapse_port_db - 381 - INFO - Table room_memberships: 89/89 (rows 1-89) already ported
2023-02-19 16:46:57,478 - synapse_port_db - 381 - INFO - Table local_media_repository_url_cache: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,483 - synapse_port_db - 381 - INFO - Table received_transactions: 1713/1713 (rows 1-1713) already ported
2023-02-19 16:46:57,484 - synapse_port_db - 381 - INFO - Table stream_ordering_to_exterm: 555/555 (rows 1-555) already ported
2023-02-19 16:46:57,484 - synapse_port_db - 381 - INFO - Table user_directory: 0/8 (rows 1-0) already ported
2023-02-19 16:46:57,487 - synapse_port_db - 381 - INFO - Table room_stats_current: 0/11 (rows 1-0) already ported
2023-02-19 16:46:57,487 - synapse_port_db - 381 - INFO - Table refresh_tokens: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,487 - synapse_port_db - 381 - INFO - Table event_json: 765/765 (rows 1-765) already ported
2023-02-19 16:46:57,488 - synapse_port_db - 381 - INFO - Table room_tags: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,488 - synapse_port_db - 381 - INFO - Table user_stats_current: 0/1 (rows 1-0) already ported
2023-02-19 16:46:57,488 - synapse_port_db - 381 - INFO - Table worker_locks: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,488 - synapse_port_db - 381 - INFO - Table event_push_summary_stream_ordering: 0/1 (rows 1-0) already ported
2023-02-19 16:46:57,488 - synapse_port_db - 381 - INFO - Table remote_media_cache: 10/10 (rows 1-10) already ported
2023-02-19 16:46:57,489 - synapse_port_db - 381 - INFO - Table rejections: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,489 - synapse_port_db - 381 - INFO - Table event_auth_chains: 0/187 (rows 1-0) already ported
2023-02-19 16:46:57,489 - synapse_port_db - 381 - INFO - Table rooms: 11/11 (rows 1-11) already ported
2023-02-19 16:46:57,489 - synapse_port_db - 381 - INFO - Table current_state_delta_stream: 0/163 (rows 1-0) already ported
2023-02-19 16:46:57,490 - synapse_port_db - 381 - INFO - Table state_groups_state: 294/294 (rows 1-294) already ported
2023-02-19 16:46:57,490 - synapse_port_db - 381 - INFO - Table users: 1/1 (rows 1-1) already ported
2023-02-19 16:46:57,494 - synapse_port_db - 381 - INFO - Table device_lists_stream: 0/43 (rows 1-0) already ported
2023-02-19 16:46:57,494 - synapse_port_db - 381 - INFO - Table open_id_tokens: 0/2 (rows 1-0) already ported
2023-02-19 16:46:57,494 - synapse_port_db - 381 - INFO - Table events: 765/765 (rows -241-577) already ported
2023-02-19 16:46:57,495 - synapse_port_db - 381 - INFO - Table stats_incremental_position: 0/1 (rows 1-0) already ported
2023-02-19 16:46:57,495 - synapse_port_db - 381 - INFO - Table remote_media_cache_thumbnails: 48/48 (rows 1-48) already ported
2023-02-19 16:46:57,495 - synapse_port_db - 381 - INFO - Table federation_stream_position: 0/2 (rows 1-0) already ported
2023-02-19 16:46:57,495 - synapse_port_db - 381 - INFO - Table device_inbox: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,495 - synapse_port_db - 381 - INFO - Table user_directory_search: 0/8 (rows 1-0) already ported
2023-02-19 16:46:57,495 - synapse_port_db - 381 - INFO - Table applied_module_schemas: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,496 - synapse_port_db - 381 - INFO - Table event_relations: 0/4 (rows 1-0) already ported
2023-02-19 16:46:57,496 - synapse_port_db - 381 - INFO - Table event_auth_chain_to_calculate: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,496 - synapse_port_db - 381 - INFO - Table device_lists_changes_in_room: 0/249 (rows 1-0) already ported
2023-02-19 16:46:57,496 - synapse_port_db - 381 - INFO - Table device_lists_remote_pending: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,496 - synapse_port_db - 381 - INFO - Table server_keys_json: 0/5 (rows 1-0) already ported
2023-02-19 16:46:57,496 - synapse_port_db - 381 - INFO - Table receipts_graph: 0/34 (rows 1-0) already ported
2023-02-19 16:46:57,498 - synapse_port_db - 381 - INFO - Table appservice_stream_position: 0/1 (rows 1-0) already ported
2023-02-19 16:46:57,505 - synapse_port_db - 381 - INFO - Table local_media_repository_thumbnails: 5/5 (rows 1-5) already ported
2023-02-19 16:46:57,505 - synapse_port_db - 381 - INFO - Table state_group_edges: 130/130 (rows 1-130) already ported
2023-02-19 16:46:57,505 - synapse_port_db - 381 - INFO - Table room_stats_earliest_token: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,505 - synapse_port_db - 381 - INFO - Table user_threepid_id_server: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,505 - synapse_port_db - 381 - INFO - Table application_services_state: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,505 - synapse_port_db - 381 - INFO - Table event_search: 96/96 (rows 1-96) already ported
2023-02-19 16:46:57,506 - synapse_port_db - 381 - INFO - Table event_push_actions_staging: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,506 - synapse_port_db - 381 - INFO - Table monthly_active_users: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,506 - synapse_port_db - 381 - INFO - Table state_events: 187/187 (rows 1-187) already ported
2023-02-19 16:46:57,506 - synapse_port_db - 381 - INFO - Table push_rules: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,507 - synapse_port_db - 381 - INFO - Table insertion_event_extremities: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,507 - synapse_port_db - 381 - INFO - Table ignored_users: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,507 - synapse_port_db - 381 - INFO - Table device_lists_outbound_pokes: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,507 - synapse_port_db - 381 - INFO - Table ratelimit_override: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,507 - synapse_port_db - 381 - INFO - Table event_push_summary: 0/6 (rows 1-0) already ported
2023-02-19 16:46:57,508 - synapse_port_db - 381 - INFO - Table server_signature_keys: 0/5 (rows 1-0) already ported
2023-02-19 16:46:57,508 - synapse_port_db - 381 - INFO - Table users_who_share_private_rooms: 0/41 (rows 1-0) already ported
2023-02-19 16:46:57,508 - synapse_port_db - 381 - INFO - Table event_labels: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,509 - synapse_port_db - 381 - INFO - Table destinations: 0/4 (rows 1-0) already ported
2023-02-19 16:46:57,509 - synapse_port_db - 381 - INFO - Table e2e_one_time_keys_json: 0/100 (rows 1-0) already ported
2023-02-19 16:46:57,509 - synapse_port_db - 381 - INFO - Table e2e_cross_signing_keys: 0/18 (rows 1-0) already ported
2023-02-19 16:46:57,509 - synapse_port_db - 381 - INFO - Table e2e_room_keys_versions: 0/2 (rows 1-0) already ported
2023-02-19 16:46:57,510 - synapse_port_db - 381 - INFO - Table event_push_summary_last_receipt_stream_id: 0/1 (rows 1-0) already ported
2023-02-19 16:46:57,510 - synapse_port_db - 381 - INFO - Table insertion_event_edges: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,514 - synapse_port_db - 381 - INFO - Table schema_compat_version: 0/1 (rows 1-0) already ported
2023-02-19 16:46:57,514 - synapse_port_db - 381 - INFO - Table event_reports: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,514 - synapse_port_db - 381 - INFO - Table account_data: 0/13 (rows 1-0) already ported
2023-02-19 16:46:57,514 - synapse_port_db - 381 - INFO - Table e2e_fallback_keys_json: 0/2 (rows 1-0) already ported
2023-02-19 16:46:57,514 - synapse_port_db - 381 - INFO - Table device_federation_inbox: 0/22 (rows 1-0) already ported
2023-02-19 16:46:57,515 - synapse_port_db - 381 - INFO - Table threepid_validation_session: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,515 - synapse_port_db - 381 - INFO - Table device_lists_changes_converted_stream_position: 0/1 (rows 1-0) already ported
2023-02-19 16:46:57,515 - synapse_port_db - 381 - INFO - Table device_auth_providers: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,515 - synapse_port_db - 381 - INFO - Table user_external_ids: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,515 - synapse_port_db - 381 - INFO - Table push_rules_enable: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,515 - synapse_port_db - 381 - INFO - Table ex_outlier_stream: 46/46 (rows 1-46) already ported
2023-02-19 16:46:57,516 - synapse_port_db - 381 - INFO - Table account_validity: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,516 - synapse_port_db - 381 - INFO - Table ui_auth_sessions: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,516 - synapse_port_db - 381 - INFO - Table user_directory_stream_pos: 0/1 (rows 1-0) already ported
2023-02-19 16:46:57,516 - synapse_port_db - 381 - INFO - Table receipts_linearized: 0/34 (rows 1-0) already ported
2023-02-19 16:46:57,521 - synapse_port_db - 381 - INFO - Table batch_events: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,521 - synapse_port_db - 381 - INFO - Table threads: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,521 - synapse_port_db - 381 - INFO - Table room_account_data: 0/7 (rows 1-0) already ported
2023-02-19 16:46:57,521 - synapse_port_db - 381 - INFO - Table room_aliases: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,521 - synapse_port_db - 381 - INFO - Table event_expiry: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,522 - synapse_port_db - 381 - INFO - Table users_pending_deactivation: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,522 - synapse_port_db - 381 - INFO - Table push_rules_stream: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,522 - synapse_port_db - 381 - INFO - Table event_forward_extremities: 0/11 (rows 1-0) already ported
2023-02-19 16:46:57,522 - synapse_port_db - 381 - INFO - Table dehydrated_devices: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,522 - synapse_port_db - 381 - INFO - Table stream_positions: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,522 - synapse_port_db - 381 - INFO - Table registration_tokens: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,522 - synapse_port_db - 381 - INFO - Table user_signature_stream: 0/1 (rows 1-0) already ported
2023-02-19 16:46:57,522 - synapse_port_db - 381 - INFO - Table device_lists_remote_resync: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,522 - synapse_port_db - 381 - INFO - Table login_tokens: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,522 - synapse_port_db - 381 - INFO - Table background_updates: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,523 - synapse_port_db - 381 - INFO - Table device_lists_outbound_last_success: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,523 - synapse_port_db - 381 - INFO - Table pusher_throttle: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,523 - synapse_port_db - 381 - INFO - Table room_alias_servers: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,523 - synapse_port_db - 381 - INFO - Table local_current_membership: 0/11 (rows 1-0) already ported
2023-02-19 16:46:57,523 - synapse_port_db - 381 - INFO - Table devices: 0/5 (rows 1-0) already ported
2023-02-19 16:46:57,523 - synapse_port_db - 381 - INFO - Table federation_inbound_events_staging: 0/0 (rows 1-0) already ported
2023-02-19 16:46:57,523 - synapse_port_db - 381 - INFO - Table local_media_repository: 31/31 (rows 1-31) already ported
2023-02-19 16:46:57,523 - synapse_port_db - 381 - INFO - Table user_threepids: 0/0 (rows 1-0) already ported
2023-02-19 16:46:58,022 - synapse_port_db - 381 - INFO - Table destination_rooms: 0/9 (rows 1-0) already ported
2023-02-19 16:46:58,022 - synapse_port_db - 381 - INFO - Table un_partial_stated_event_stream: 0/8 (rows 1-0) already ported
2023-02-19 16:46:58,022 - synapse_port_db - 381 - INFO - Table ui_auth_sessions_ips: 0/0 (rows 1-0) already ported
2023-02-19 16:46:58,023 - synapse_port_db - 381 - INFO - Table event_failed_pull_attempts: 0/0 (rows 1-0) already ported
2023-02-19 16:46:58,023 - synapse_port_db - 381 - INFO - Table ui_auth_sessions_credentials: 0/0 (rows 1-0) already ported
2023-02-19 16:46:58,023 - synapse_port_db - 381 - INFO - Table event_edges: 757/757 (rows 1-757) already ported
2023-02-19 16:46:58,023 - synapse_port_db - 381 - INFO - Table un_partial_stated_room_stream: 0/8 (rows 1-0) already ported
2023-02-19 16:46:58,023 - synapse_port_db - 381 - INFO - Table access_tokens: 0/2 (rows 1-0) already ported
2023-02-19 16:46:58,023 - synapse_port_db - 381 - INFO - Table users_to_send_full_presence_to: 0/0 (rows 1-0) already ported
2023-02-19 16:46:58,024 - synapse_port_db - 381 - INFO - Table partial_state_rooms: 0/0 (rows 1-0) already ported
2023-02-19 16:46:58,032 - synapse_port_db - 235 - ERROR - Failed to insert: un_partial_stated_event_stream
Traceback (most recent call last):
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py:1792:gotResult
--- <exception caught here> ---
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py:1693:_inlineCallbacks
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/failure.py:518:throwExceptionIntoGenerator
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py:482:handle_table
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:882:runInteraction
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py:1693:_inlineCallbacks
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/failure.py:518:throwExceptionIntoGenerator
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:849:_runInteraction
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:977:runWithConnection
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py:244:inContext
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py:260:<lambda>
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py:117:callWithContext
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py:82:callWithContext
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py:282:_runWithConnection
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:970:inner_func
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:711:new_transaction
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py:470:insert
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py:233:insert_many_txn
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:391:executemany
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:436:_do_execute

Console output of synapse_port_db:

Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py", line 796, in run
    await make_deferred_yieldable(
twisted.internet.defer.FirstError: FirstError[#1, [Failure instance: Traceback: <class 'psycopg2.errors.DatatypeMismatch'>: column "rejection_status_changed" is of type boolean but expression is of type integer
LINE 1: ...'master', '$foQmi7puBLD3_yv4WJKfHnpgi8QApuT8O51mXoKRwqg', 0)
                                                                     ^
HINT:  You will need to rewrite or cast the expression.

/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py:735:errback
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py:798:_startRunCallbacks
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py:892:_runCallbacks
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py:1792:gotResult
--- <exception caught here> ---
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py:1693:_inlineCallbacks
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/failure.py:518:throwExceptionIntoGenerator
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py:482:handle_table
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:882:runInteraction
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py:1693:_inlineCallbacks
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/failure.py:518:throwExceptionIntoGenerator
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:849:_runInteraction
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:977:runWithConnection
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py:244:inContext
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py:260:<lambda>
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py:117:callWithContext
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py:82:callWithContext
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py:282:_runWithConnection
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:970:inner_func
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:711:new_transaction
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py:470:insert
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/_scripts/synapse_port_db.py:233:insert_many_txn
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:391:executemany
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py:436:_do_execute
]]
FirstError[#1, [Failure instance: Traceback: <class 'psycopg2.errors.DatatypeMismatch'>: column "rejection_status_changed" is of type boolean but expression is of type integer
LINE 1: ...'master', '$foQmi7puBLD3_yv4WJKfHnpgi8QApuT8O51mXoKRwqg', 0)
                                                                     ^
HINT:  You will need to rewrite or cast the expression.

/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py:735:errback
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py:798:_startRunCallbacks
/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py:892:_runCallbacks

Anything else that would be useful to know?

No response
@DMRobertson DMRobertson added A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db A-SQLite Database issues specific to SQLite T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Regression Something broke which worked on a previous release labels Feb 20, 2023
@DMRobertson
Copy link
Contributor

We added a boolean column in #14545

synapse/synapse/storage/schema/main/delta/73/22_un_partial_stated_event_stream.sql

Lines 17 to 30 in fb60cb1

CREATE TABLE un_partial_stated_event_stream(
-- Position in the stream
stream_id BIGINT PRIMARY KEY NOT NULL,
-- Which instance wrote this entry.
instance_name TEXT NOT NULL,
-- Which event has been un-partial-stated.
event_id TEXT NOT NULL REFERENCES events(event_id) ON DELETE CASCADE,
-- true iff the `rejected` status of the event changed when it became
-- un-partial-stated.
rejection_status_changed BOOLEAN NOT NULL
);

But didn't mark it as a boolean here:

synapse/synapse/_scripts/synapse_port_db.py

Lines 96 to 117 in 65d0386

BOOLEAN_COLUMNS = {
"events": ["processed", "outlier", "contains_url"],
"rooms": ["is_public", "has_auth_chain_index"],
"event_edges": ["is_state"],
"presence_list": ["accepted"],
"presence_stream": ["currently_active"],
"public_room_list_stream": ["visibility"],
"devices": ["hidden"],
"device_lists_outbound_pokes": ["sent"],
"users_who_share_rooms": ["share_private"],
"e2e_room_keys": ["is_verified"],
"account_validity": ["email_sent"],
"redactions": ["have_censored"],
"room_stats_state": ["is_federatable"],
"local_media_repository": ["safe_from_quarantine"],
"users": ["shadow_banned", "approved"],
"e2e_fallback_keys_json": ["used"],
"access_tokens": ["used"],
"device_lists_changes_in_room": ["converted_to_destinations"],
"pushers": ["enabled"],
}

@DMRobertson DMRobertson mentioned this issue Feb 20, 2023
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db A-SQLite Database issues specific to SQLite T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Regression Something broke which worked on a previous release
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Teach portdb about un_partial_stated_event_stream matrix-org/synapse
2 participants
@DMRobertson @fmielicki