Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Fix push for invites received over federation #15820

Merged
merged 9 commits into from
Jul 12, 2023
Merged

Fix push for invites received over federation #15820

merged 9 commits into from
Jul 12, 2023

Conversation

erikjohnston
Copy link
Member

@erikjohnston erikjohnston commented Jun 23, 2023 •
edited
Loading

Sytests: matrix-org/sytest#1356

Fixes errors like:

RuntimeError: No state group for unknown or outlier event $AAAAAAAAAAAAAAAAAAA
  File "twisted/internet/defer.py", line 1697, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "synapse/storage/databases/main/state.py", line 393, in _get_state_group_for_events
    raise RuntimeError("No state group for unknown or outlier event %s" % e)
KeyError: '$WbaU0awJEynlK-FBflXpfp_AK6EG3LrVfiwIzhPNW6c'
  File "synapse/push/httppusher.py", line 215, in _process
    await self._unsafe_process()
  File "synapse/push/httppusher.py", line 253, in _unsafe_process
    processed = await self._process_one(push_action)
  File "synapse/push/httppusher.py", line 352, in _process_one
    rejected = await self.dispatch_push_event(event, tweaks, badge)
  File "synapse/push/httppusher.py", line 469, in dispatch_push_event
    ctx = await push_tools.get_context_for_event(
  File "synapse/push/push_tools.py", line 54, in get_context_for_event
    room_state_ids = await storage.state.get_state_ids_for_event(ev.event_id)
  File "synapse/logging/opentracing.py", line 916, in _wrapper
    return await func(*args, **kwargs)  # type: ignore[misc]
  File "synapse/logging/opentracing.py", line 916, in _wrapper
    return await func(*args, **kwargs)  # type: ignore[misc]
  File "synapse/storage/controllers/state.py", line 344, in get_state_ids_for_event
    state_map = await self.get_state_ids_for_events(
  File "synapse/logging/opentracing.py", line 916, in _wrapper
    return await func(*args, **kwargs)  # type: ignore[misc]
  File "synapse/logging/opentracing.py", line 916, in _wrapper
    return await func(*args, **kwargs)  # type: ignore[misc]
  File "synapse/storage/controllers/state.py", line 293, in get_state_ids_for_events
    return {event: event_to_state[event] for event in event_ids}
  File "synapse/storage/controllers/state.py", line 293, in <dictcomp>
    return {event: event_to_state[event] for event in event_ids}

@erikjohnston erikjohnston mentioned this pull request Jun 23, 2023
Merged
@erikjohnston erikjohnston marked this pull request as ready for review June 23, 2023 10:10
@erikjohnston erikjohnston requested a review from a team as a code owner June 23, 2023 10:10
MadLittleMods
MadLittleMods reviewed Jun 23, 2023
View reviewed changes
changelog.d/15820.bugfix
@@ -0,0 +1 @@
Fix long-standing bug where remote invites weren't correctly pushed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this fix spurred on by anything? Context?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noticed the error in Sentry when I was looking at other stuff

@MadLittleMods MadLittleMods requested a review from a team June 23, 2023 22:15
MadLittleMods
MadLittleMods reviewed Jun 28, 2023
View reviewed changes
synapse/push/push_tools.py
Comment on lines 50 to 52
async def get_context_for_event(
storage: StorageControllers, ev: EventBase, user_id: str
) -> Dict[str, str]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Historical problem but this function name is confusing because EventContext is also a thing.

And it would be nice to return a dataclass here so it's obvious what keys are available.

MadLittleMods
MadLittleMods reviewed Jun 28, 2023
View reviewed changes
synapse/push/push_tools.py Outdated
Comment on lines 56 to 62
# We don't have state for outliers, so we can't compute the context
# except for invites with `invite_room_state` set..
if (
ev.type != EventTypes.Member
or ev.content.get("membership") != Membership.INVITE
):
return ctx
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Explaining for my own understanding:

Since this is just used to create the push Notification object, it's mostly a best effort thing since it's ok to get a notification that doesn't say what the room name is as it can just fallback to the room_alias or room_id, etc. Which is why it's ok that we return an empty dict without any info (sender_display_name and room_name are optional)

@erikjohnston erikjohnston force-pushed the erikj/fix_invites_over_federation branch from b8ecd5e to 5ac6309 Compare July 11, 2023 09:26
@MadLittleMods MadLittleMods added A-Federation A-Push Issues related to push/notifications A-Invite Inviting users to rooms and accepting invites labels Jul 11, 2023
@erikjohnston erikjohnston enabled auto-merge (squash) July 12, 2023 09:36
@erikjohnston erikjohnston merged commit 36c6b92 into develop Jul 12, 2023
37 checks passed
@erikjohnston erikjohnston deleted the erikj/fix_invites_over_federation branch July 12, 2023 11:02
yingziwu added a commit to yingziwu/synapse that referenced this pull request Aug 2, 2023
@yingziwu
Merge tag 'v1.89.0' into develop
ce16c05 
No significant changes since 1.89.0rc1.

- Add Unix Socket support for HTTP Replication Listeners. [Document and provide usage instructions](https://matrix-org.github.io/synapse/v1.89/usage/configuration/config_documentation.html#listeners) for utilizing Unix sockets in Synapse. Contributed by Jason Little. ([\matrix-org#15708](matrix-org#15708), [\matrix-org#15924](matrix-org#15924))
- Allow `+` in Matrix IDs, per [MSC4009](matrix-org/matrix-spec-proposals#4009). ([\matrix-org#15911](matrix-org#15911))
- Support room version 11 from [MSC3820](matrix-org/matrix-spec-proposals#3820). ([\matrix-org#15912](matrix-org#15912))
- Allow configuring the set of workers to proxy outbound federation traffic through via `outbound_federation_restricted_to`. ([\matrix-org#15913](matrix-org#15913), [\matrix-org#15969](matrix-org#15969))
- Implement [MSC3814](matrix-org/matrix-spec-proposals#3814), dehydrated devices v2/shrivelled sessions and move [MSC2697](matrix-org/matrix-spec-proposals#2697) behind a config flag. Contributed by Nico from Famedly, H-Shay and poljar. ([\matrix-org#15929](matrix-org#15929))

- Fix a long-standing bug where remote invites weren't correctly pushed. ([\matrix-org#15820](matrix-org#15820))
- Fix background schema updates failing over a large upgrade gap. ([\matrix-org#15887](matrix-org#15887))
- Fix a bug introduced in 1.86.0 where Synapse starting with an empty `experimental_features` configuration setting. ([\matrix-org#15925](matrix-org#15925))
- Fixed deploy annotations in the provided Grafana dashboard config, so that it shows for any homeserver and not just matrix.org. Contributed by @wrjlewis. ([\matrix-org#15957](matrix-org#15957))
- Ensure a long state res does not starve CPU by occasionally yielding to the reactor. ([\matrix-org#15960](matrix-org#15960))
- Properly handle redactions of creation events. ([\matrix-org#15973](matrix-org#15973))
- Fix a bug where resyncing stale device lists could block responding to federation transactions, and thus delay receiving new data from the remote server. ([\matrix-org#15975](matrix-org#15975))

- Better clarify how to run a worker instance (pass both configs). ([\matrix-org#15921](matrix-org#15921))
- Improve [the documentation](https://matrix-org.github.io/synapse/v1.89/admin_api/user_admin_api.html#login-as-a-user) for the login as a user admin API. ([\matrix-org#15938](matrix-org#15938))
- Fix broken Arch Linux package link. Contributed by @SnipeXandrej. ([\matrix-org#15981](matrix-org#15981))

- Remove support for calling the `/register` endpoint with an unspecced `user` property for application services. ([\matrix-org#15928](matrix-org#15928))

- Mark `get_user_in_directory` private since it is only used in tests. Also remove the cache from it. ([\matrix-org#15884](matrix-org#15884))
- Document which Python version runs on a given Linux distribution so we can more easily clean up later. ([\matrix-org#15909](matrix-org#15909))
- Add details to warning in log when we fail to fetch an alias. ([\matrix-org#15922](matrix-org#15922))
- Remove unneeded `__init__`. ([\matrix-org#15926](matrix-org#15926))
- Fix bug with read/write lock implementation. This is currently unused so has no observable effects. ([\matrix-org#15933](matrix-org#15933), [\matrix-org#15958](matrix-org#15958))
- Unbreak the nix development environment by pinning the Rust version to 1.70.0. ([\matrix-org#15940](matrix-org#15940))
- Update presence metrics to differentiate remote vs local users. ([\matrix-org#15952](matrix-org#15952))
- Stop reading from column `user_id` of table `profiles`. ([\matrix-org#15955](matrix-org#15955))
- Build packages for Debian Trixie. ([\matrix-org#15961](matrix-org#15961))
- Reduce the amount of state we pull out. ([\matrix-org#15968](matrix-org#15968))
- Speed up updating state in large rooms. ([\matrix-org#15971](matrix-org#15971))

* Bump anyhow from 1.0.71 to 1.0.72. ([\matrix-org#15949](matrix-org#15949))
* Bump click from 8.1.3 to 8.1.6. ([\matrix-org#15984](matrix-org#15984))
* Bump cryptography from 41.0.1 to 41.0.2. ([\matrix-org#15943](matrix-org#15943))
* Bump jsonschema from 4.17.3 to 4.18.3. ([\matrix-org#15948](matrix-org#15948))
* Bump pillow from 9.4.0 to 10.0.0. ([\matrix-org#15986](matrix-org#15986))
* Bump prometheus-client from 0.17.0 to 0.17.1. ([\matrix-org#15945](matrix-org#15945))
* Bump pydantic from 1.10.10 to 1.10.11. ([\matrix-org#15946](matrix-org#15946))
* Bump pygithub from 1.58.2 to 1.59.0. ([\matrix-org#15834](matrix-org#15834))
* Bump pyo3-log from 0.8.2 to 0.8.3. ([\matrix-org#15951](matrix-org#15951))
* Bump sentry-sdk from 1.26.0 to 1.28.1. ([\matrix-org#15985](matrix-org#15985))
* Bump serde_json from 1.0.100 to 1.0.103. ([\matrix-org#15950](matrix-org#15950))
* Bump types-pillow from 9.5.0.4 to 10.0.0.1. ([\matrix-org#15932](matrix-org#15932))
* Bump types-requests from 2.31.0.1 to 2.31.0.2. ([\matrix-org#15983](matrix-org#15983))
* Bump typing-extensions from 4.5.0 to 4.7.1. ([\matrix-org#15947](matrix-org#15947))
Fizzadar added a commit to beeper/synapse-legacy-fork that referenced this pull request Aug 29, 2023
@Fizzadar
Merge tag 'v1.89.0' into merge-1.89
13e9b31 
No significant changes since 1.89.0rc1.

- Add Unix Socket support for HTTP Replication Listeners. [Document and provide usage instructions](https://matrix-org.github.io/synapse/v1.89/usage/configuration/config_documentation.html#listeners) for utilizing Unix sockets in Synapse. Contributed by Jason Little. ([\matrix-org#15708](matrix-org#15708), [\matrix-org#15924](matrix-org#15924))
- Allow `+` in Matrix IDs, per [MSC4009](matrix-org/matrix-spec-proposals#4009). ([\matrix-org#15911](matrix-org#15911))
- Support room version 11 from [MSC3820](matrix-org/matrix-spec-proposals#3820). ([\matrix-org#15912](matrix-org#15912))
- Allow configuring the set of workers to proxy outbound federation traffic through via `outbound_federation_restricted_to`. ([\matrix-org#15913](matrix-org#15913), [\matrix-org#15969](matrix-org#15969))
- Implement [MSC3814](matrix-org/matrix-spec-proposals#3814), dehydrated devices v2/shrivelled sessions and move [MSC2697](matrix-org/matrix-spec-proposals#2697) behind a config flag. Contributed by Nico from Famedly, H-Shay and poljar. ([\matrix-org#15929](matrix-org#15929))

- Fix a long-standing bug where remote invites weren't correctly pushed. ([\matrix-org#15820](matrix-org#15820))
- Fix background schema updates failing over a large upgrade gap. ([\matrix-org#15887](matrix-org#15887))
- Fix a bug introduced in 1.86.0 where Synapse starting with an empty `experimental_features` configuration setting. ([\matrix-org#15925](matrix-org#15925))
- Fixed deploy annotations in the provided Grafana dashboard config, so that it shows for any homeserver and not just matrix.org. Contributed by @wrjlewis. ([\matrix-org#15957](matrix-org#15957))
- Ensure a long state res does not starve CPU by occasionally yielding to the reactor. ([\matrix-org#15960](matrix-org#15960))
- Properly handle redactions of creation events. ([\matrix-org#15973](matrix-org#15973))
- Fix a bug where resyncing stale device lists could block responding to federation transactions, and thus delay receiving new data from the remote server. ([\matrix-org#15975](matrix-org#15975))

- Better clarify how to run a worker instance (pass both configs). ([\matrix-org#15921](matrix-org#15921))
- Improve [the documentation](https://matrix-org.github.io/synapse/v1.89/admin_api/user_admin_api.html#login-as-a-user) for the login as a user admin API. ([\matrix-org#15938](matrix-org#15938))
- Fix broken Arch Linux package link. Contributed by @SnipeXandrej. ([\matrix-org#15981](matrix-org#15981))

- Remove support for calling the `/register` endpoint with an unspecced `user` property for application services. ([\matrix-org#15928](matrix-org#15928))

- Mark `get_user_in_directory` private since it is only used in tests. Also remove the cache from it. ([\matrix-org#15884](matrix-org#15884))
- Document which Python version runs on a given Linux distribution so we can more easily clean up later. ([\matrix-org#15909](matrix-org#15909))
- Add details to warning in log when we fail to fetch an alias. ([\matrix-org#15922](matrix-org#15922))
- Remove unneeded `__init__`. ([\matrix-org#15926](matrix-org#15926))
- Fix bug with read/write lock implementation. This is currently unused so has no observable effects. ([\matrix-org#15933](matrix-org#15933), [\matrix-org#15958](matrix-org#15958))
- Unbreak the nix development environment by pinning the Rust version to 1.70.0. ([\matrix-org#15940](matrix-org#15940))
- Update presence metrics to differentiate remote vs local users. ([\matrix-org#15952](matrix-org#15952))
- Stop reading from column `user_id` of table `profiles`. ([\matrix-org#15955](matrix-org#15955))
- Build packages for Debian Trixie. ([\matrix-org#15961](matrix-org#15961))
- Reduce the amount of state we pull out. ([\matrix-org#15968](matrix-org#15968))
- Speed up updating state in large rooms. ([\matrix-org#15971](matrix-org#15971))

* Bump anyhow from 1.0.71 to 1.0.72. ([\matrix-org#15949](matrix-org#15949))
* Bump click from 8.1.3 to 8.1.6. ([\matrix-org#15984](matrix-org#15984))
* Bump cryptography from 41.0.1 to 41.0.2. ([\matrix-org#15943](matrix-org#15943))
* Bump jsonschema from 4.17.3 to 4.18.3. ([\matrix-org#15948](matrix-org#15948))
* Bump pillow from 9.4.0 to 10.0.0. ([\matrix-org#15986](matrix-org#15986))
* Bump prometheus-client from 0.17.0 to 0.17.1. ([\matrix-org#15945](matrix-org#15945))
* Bump pydantic from 1.10.10 to 1.10.11. ([\matrix-org#15946](matrix-org#15946))
* Bump pygithub from 1.58.2 to 1.59.0. ([\matrix-org#15834](matrix-org#15834))
* Bump pyo3-log from 0.8.2 to 0.8.3. ([\matrix-org#15951](matrix-org#15951))
* Bump sentry-sdk from 1.26.0 to 1.28.1. ([\matrix-org#15985](matrix-org#15985))
* Bump serde_json from 1.0.100 to 1.0.103. ([\matrix-org#15950](matrix-org#15950))
* Bump types-pillow from 9.5.0.4 to 10.0.0.1. ([\matrix-org#15932](matrix-org#15932))
* Bump types-requests from 2.31.0.1 to 2.31.0.2. ([\matrix-org#15983](matrix-org#15983))
* Bump typing-extensions from 4.5.0 to 4.7.1. ([\matrix-org#15947](matrix-org#15947))

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmTI2e4ACgkQkD7OEIo5
# 3t2x1RAAohu1Rmjv0mOqFR4P1YZpA5RFbYajcyq77n/ciDKSM1dqBelONqKOq2A9
# uGbVNm6rC+EFwIl5MF5TrFdsDQHvGcRgW6NpQDZ+uIUOYizjZH1g37BoNPLlGYQx
# fmKG7/XqdWhSc5tHN9HsRHyHKmsndebjXoUCPKmieGZa1GLXvGwrNkWQlEpwd9Qu
# mj3uewJxLFGgIIAOiplJ4UO8FaCbMD+By27hSiWtVsLT6pyav4HC2P8RQD1iv0jW
# OXNHvEWyqfBPlsPOkCD4nQZrmZqa5GWLYfBm8zFgIBxNy+e33C07L4bO+QdCE86v
# /SUKug/0nsp66jSZst1fM/M2ssXvjU+LNO9fqonOCZ4TiJ4i/yoa8AvmcAg5hy7C
# HR9IBp9cMrQ2u1y2/knxF657AGHxgXEltgw0PDvZHowqsqoSb+5HWl0zv1wnVjMa
# 2QYLKWPBk/AdlHkmC3S4/+gfVZVsT2RSBP3JUCbFyOqug9vXFvSGTfH07Lk4PDI3
# o5idBzumvyonsuC2ypkzlj49FAj21l/8DInxEpY9JcHdVncLWvu9gmLd+H7GY7H7
# ODa2gOynrsSGVH7IpOl6dpw/GH6R8ZlfHl87bFslOqVObBxquL/ODIoFOgld+MpT
# YYXp+0tW564mg+AYw3+eo44JTq0lKh7eyENP3SqKN/Z8ssQL97c=
# =Ar/g
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue Aug  1 11:09:50 2023 BST
# gpg:                using RSA key D79D3CA0B61429A8A760525A903ECE108A39DEDD
# gpg: key 903ECE108A39DEDD: new key but contains no user ID - skipped
# gpg: Total number processed: 1
# gpg:           w/o user IDs: 1
# gpg: Can't check signature: No public key

# Conflicts:
#	poetry.lock
#	synapse/http/site.py
#	synapse/storage/databases/main/roommember.py
Fizzadar pushed a commit to beeper/synapse-legacy-fork that referenced this pull request Aug 29, 2023
@erikjohnston @Fizzadar
Fix push for invites received over federation (matrix-org#15820)
2f43d63
Fizzadar pushed a commit to beeper/synapse-legacy-fork that referenced this pull request Aug 31, 2023
@erikjohnston @Fizzadar
Fix push for invites received over federation (matrix-org#15820)
8aedcd1
jcgruenhage pushed a commit to famedly/synapse that referenced this pull request Oct 4, 2023
@erikjohnston @jcgruenhage
Fix push for invites received over federation (matrix-org#15820)
348a8fc
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@MadLittleMods MadLittleMods MadLittleMods approved these changes

Assignees
No one assigned
Labels
A-Federation A-Invite Inviting users to rooms and accepting invites A-Push Issues related to push/notifications
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@erikjohnston @MadLittleMods