Change use of a sync for obtaining the room history to a room initial sync. #132
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Yoric
approved these changes
Sep 17, 2021
There was a problem hiding this comment.
Nit: Could you reword the commit to something along the lines of:
Use `initialSync` instead of `sync` to fetch room history.
See [github issue]: for the time being, this seems to be the only way to avoid requests that take the server several minutes to complete and timeout.
src/utils.ts
Outdated
| filter: JSON.stringify(filter), | ||
| }; | ||
| return client.doRequest("GET", "/_matrix/client/r0/sync", qs); | ||
| // There is no direct replacement for this and doing a sync here |
There was a problem hiding this comment.
This is not a function documentation but implementation comment, so this should go inside the function.
Also, it's not clear what you're commenting. Let me suggest:
/*
Note: `/initialSync` is deprecated. However, there is no replacement for this API for the time being.
While previous versions of this function used `/sync`, experience shows that it can grow extremely
slow (4-5 minutes long) when we need to sync many large rooms, which leads to timeouts and
breakage in Mjolnir, see e.g. [insert issue numbers].
*/
See matrix-org/synapse#10842. For the time being this seems to be the only way to avoid requests that take several minutes for synapse to complete and timeout.
Gnuxie
force-pushed
the
gnuxie/remove-messages-sync
branch
from
453dca0
to
6edf503
Compare
Gnuxie
added a commit
that referenced
this pull request
Oct 14, 2021
Related to #132. The old code would call `/sync` with this filter. If a token was provided in the response of `/sync` for earlier messages, it would then use this same filter to call `/rooms/messages`. However, this filter does not do anything on that endpoint when we know the id of the sender, as it requires a RoomEventFilter and there is no warning or error from synapse about the structure of the filter being wrong. This was not noticed until after the related PR because `/sync` with the filter would usually be able to provide a user's entire history in one room. This is because in most cases a user is banned/redacted shortly after joining a room. In the case that `/rooms/messages` was called for more events, the method would always paginate the timeline up until the limit or the end of the room history, which is only the expected behavior when matching the sender with a "glob".
Gnuxie
added a commit
that referenced
this pull request
Oct 15, 2021
Related to #132. The old code would call `/sync` with this filter. If a token was provided in the response of `/sync` for earlier messages, it would then use this same filter to call `/rooms/messages`. However, this filter does not do anything on that endpoint when we know the id of the sender, as it requires a RoomEventFilter and there is no warning or error from synapse about the structure of the filter being wrong. This was not noticed until after the related PR because `/sync` with the filter would usually be able to provide a user's entire history in one room. This is because in most cases a user is banned/redacted shortly after joining a room. In the case that `/rooms/messages` was called for more events, the method would always paginate the timeline up until the limit or the end of the room history, which is only the expected behavior when matching the sender with a "glob".
Gnuxie
added a commit
that referenced
this pull request
Oct 19, 2021
Related to #132. The old code would call `/sync` with this filter. If a token was provided in the response of `/sync` for earlier messages, it would then use this same filter to call `/rooms/messages`. However, this filter does not do anything on that endpoint when we know the id of the sender, as it requires a RoomEventFilter and there is no warning or error from synapse about the structure of the filter being wrong. This was not noticed until after the related PR because `/sync` with the filter would usually be able to provide a user's entire history in one room. This is because in most cases a user is banned/redacted shortly after joining a room. In the case that `/rooms/messages` was called for more events, the method would always paginate the timeline up until the limit or the end of the room history, which is only the expected behavior when matching the sender with a "glob".
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a deprecated API but the call to sync on an account in a large
number of rooms is taking too much time (and can fail), even with strict filtering
limiting the response to a single room.
This code is used to fetch the events to redact from a timeline,
so it is especially problematic.