[meta] Room List API

[Hywan]

• Epic, https://github.com/vector-im/internal-planning/issues/405

Image generated with DeepAI

We want to provide a RoomList API inside matrix-sdk. The goal is to provide an abstraction on top of SlidingSync for classical usages of it, for example in common clients (ElementX, Fractal etc.). This can also be used by bots and so on.

This meta issue provides a list of issues to achieve this goal.

Step 1, the foundation

The idea is the following. It happens in 2 phases:

1. RoomList will create one all_rooms list, with Selective mode and timeline_limit = 0, and a range of 20/30 rooms, so that we can sync a small room list, with no timeline but at least their names,
2. Once 1 is sent, i.e. once this all_rooms has received a response, we change its mode to Growing with a batch of 100, still with timeline_limit = 0, and it will sync all rooms, one after the other:
   1. At the same time, we create a new visible_rooms list, with Selective mode and timeline_limit = 1
   2. At the same time, we will also add two more lists: one for spaces, one for invites.

Right now, it's not possible to change the mode of a list. But nothing is difficult: it's a matter of resetting it, and exposing a setter for the mode.

This workflow above avoids to merge multiple lists together, as described in #1644 and as done right now by ElementX iOS, which can be very tricky and error-prone.

Main PRs are:

• feat(ui): RoomList, step 1: the Foundation #1953

• Add ability to change the mode of a SlidingSyncList on-the-fly.
  • The SlidingSyncMode should hold the batch_size and maximum_number_of_rooms_to_fetch for Growing and Paging, instead of passing it to the list builder:
    • feat(sdk): SlidingSyncMode has richer variants #1941
  • Then, we can change the sync mode on-the-fly more easily as a simple
    SlidingSyncList::set_mode(SlidingSyncMode) method will be enough to re-configure the list with all the data it needs.
    • feat(sdk): Implement SlidingSyncList::set_sync_mode #1942
• Create the crates/matrix-sdk-ui/src/room_list.rs module. It's outside the crates/matrix-sdk/src/sliding_sync/ module on purpose.
• Forward the observable room_list from a particular list to RoomList (i.e. it will be from all_rooms).
• Expose a proper observable room_list to the public which is a “duplicate” of the previous list above, because it can be filtered. Maybe, let's take time to think about a FilteredObservable type, or a FilteredSubscriber, or something like this. This will allow to support fuzzy search. The general idea is to have a “source of truth room_list” and a “public room_list” that can be filtered
• Accept inputs from the client:
  • Update the timeline_limit maybe (not necessary)
  • Update the range
  • Update the batch size maybe (not necessary)
  • Update the required state
  • Update the filters
  • Be able to register observers for: state, maximum_number_of_rooms, room_list diffs etc. (not necessary)
  • Get a SlidingSyncRoom
  • Be able to notify when a room is opened or closed, i.e. we need to subscribe/unsubscribe from that room
    • feat(ui): Implement room subscriptions in RoomList + notification counts #2069
• Add unit tests.
• Add integration tests.

Step 2, moaaar features

• Add the invite list (need to define what it is)
  • feat(ffi): Implement RoomList::invites #2094
• Add fuzzy search
  • Use https://github.com/lotabout/fuzzy-matcher
  • feat(ui): Implement the “fuzzy match room name” filter #2335
• Ensure the room has at least one visible event in its timeline when the room appears in the search result
• Add a way to attach other info to the room, like red and green indicators, see https://github.com/vector-im/internal-planning/issues/327
  • Support unread notifications count
    • feat(ui): Implement room subscriptions in RoomList + notification counts #2069
• Add more Room API, as defined in https://github.com/vector-im/element-x-ios/blob/develop/ElementX/Sources/Services/Room/RoomProxy.swift#L66
  • feat(ui+ffi): Add various features to RoomList #2054
• Ensure extensions are correctly defined, [meta] Room List API #1911 (comment)
  • feat(ui+ffi): Add various features to RoomList #2054
• Provide a Service on top of RoomList:
  • feat(ui): Welcome to RoomListService #2128

Step 3, the clean up and appendix tasks

• Write RoomList FFI bindings
  • feat(ffi): Implement RoomList API #2035
• [SlidingSync] unique room identifiers #1977
• [meta] Room List API #1911 (comment)
• [meta] Room List API #1911 (comment)
• feat(ui): Add bump_event_types for all_rooms in RoomList #2097
• fix(sdk): Remove the Sliding Sync retry mechanism on M_UNKNOWN_POS #2103
• [meta] Room List API #1911 (comment)
• [meta] Room List API #1911 (comment)
  • chore(sdk): Log the SyncOps #2152
• Remove the bindings/matrix-sdk-ffi/src/sliding_sync.rs module, as we no longer need to provide and to maintain bindings for the moment
  • chore(ffi): Bye bye Sliding Sync bindings! #2222

Step 4, make RoomList fasteeeer

RoomList is fast, but sometimes we see some “slowness”. They might not be related to RoomList itself, but I think this issue is a good place to reference them and to explain them, as the root is this new RoomList API.

• Room::latest_event() needs to build a Timeline to get the latest event. It's the second Timeline that Room builds: the first one is Room::timeline(). It seems really suboptimal to do so. Re-enable the room list cache #2085 explains how storing the room's display name and the room's latest event directly inside RoomInfo would have a twofold advantage:
  • Make Room::latest_event() faster, thus removing the need to build an intermediate sneaky Timeline just for that (bonus: it also removes a crash in ElementX when they build this SDK in debug mode, hello rustc bug :-]),
  • We can finally turn on the Sliding Sync cache inside RoomList.
• Room::timeline() builds a new Timeline as we said at the previous item. Sometimes, TimelineBuilder::build can take times to finish because it retries to decrypt event.
• We also want to reduce the number of VectorDiff we send to the user. When a room goes ouf of range, it becomes Invalidated, then once its synced again, it becomes Filled again. The problem is that the client app have to create a new Room everytime, which is expensive because of the points above.
  • Short-term solutions are:
    • fix(sdk): Change SlidingSyncList::room_list's capacity #2147
    • feat: Cache the RoomListService's rooms #2163
  • Real, clean solutions are:
    • Make Room::latest_event fast (depends on Re-enable the room list cache #2085)
    • Remove the cache in RoomListService (revert feat: Cache the RoomListService's rooms #2163). Why? Because the cache will keep Rooms even if the user has left them. That's not annoying for the moment as it's an “edge case”, it rarely happens, but we want something clean, remember :-)? We need a cache to avoid re-creating Timelines everytime, which can be slow, hence using the RingBuffer I've recently introduced (feat(common): Propose a simple RingBuffer implementation #2154), so that we keep track/a cache of the latest most used rooms.
    • Reset the SlidingSyncList::room_list's capacity to a smaller value (partially revert fix(sdk): Change SlidingSyncList::room_list's capacity #2147)
      • chore(sdk): Remove one TODO in Sliding Sync #2332
    • Remove RoomListEntry::Invalidate ⚠️. Either we represent the fact that the room list entry is no more synced, or we simply don't care. The client app doesn't need this information, the SDK doesn't need this information neither, let's see if bots might need it?
    • Implement a new eyeball::Subscriber that does “batch updates”
      • https://codeberg.org/jplatte/async-rx/pulls/1
      • feat(ui): Batch the streams returned by RoomList::entries and RoomList::entries_filtered. #2322
    • Make Room very cheap to build

Here are the list of extra issues we track:

• Re-enable the room list cache #2085
• Cache DisplayName in RoomInfo #2089
• feat(ui): Spawn a new task to retry event decryption in Timeline #2107
• feat(ui): TimelineBuilder::build logs more data #2109
• Improve performance of RoomListService's rooms and associated events #2161

[manuroe]

This looks really great. Thanks for compiling the tasks. They look good to me.

I have just few remarks or questions:

I am bit scared with the tiny timeline_limit = 1 and that we do not prefetch more events to pre-fill the timeline or better resolve the last message. But, as discussed in a call, we can experiment and update the value.

Should we plan works around caching? Will data from all list be stored?

Update the range maybe

It is mandatory for the visible_rooms list in phase 2. We should be able to prioritise the display of the last message for the room being displayed over fetching data for the all_rooms list in background.

Ensure the room has at least one visible event in its timeline when the room appears in the search result

There will be a new API, an updated version of /messages, to be able to fetch messages for several rooms in a single request. Before it lands, we could call the existing /messages on every displayed room.

An extension to this behavior would be to use /messages on every room that does not have a displayable last message.

[Hywan]

The timeline_limit = 1 is from the actual ElementX-iOS code, but I'm happy to change to anything that makes your happy :-). As said, it'll super easy to modify, it won't change anything to the logic of the RoomList in itself.

[Hywan]

Should we plan works around caching? Will data from all list be stored?

We must rely on the caching API of SlidingSync itself. Adding another cache layer on top will just increase the surface of potential bugs I reckon. Having a better caching API for SlidingSync was already on our roadmap, even if PR like #1876 already improves it greatly.

[Hywan]

There will be a new API, an updated version of /messages, to be able to fetch messages for several rooms in a single request. Before it lands, we could call the existing /messages on every displayed room.

I wonder how it will fit… SlidingSync feeds a Timeline by using… well… sync'ed data. New data are also sync'ed y SS and then dispatched in the SDK to reach the Timeline handler. When calling another HTTP endpoint, I hope it won't create duplicated events. I understand that SS can't pick rooms to sync with, but maybe having all_rooms with a timeline_limit = 1 would be a solution?

[stefanceriu]

One note on something we just realised: once your session expires we need to restart the initial app SS flow (in which we start with a small range and small limits etc.) to avoid long initial syncs e.g. don't continue from a full all rooms range.
We also need to make sure we don't lose existing room subscriptions when we do that.

[stefanceriu]

@Hywan Can we please make sure we run this with all the "common extensions"?

[stefanceriu]

@Hywan Note: we need a replacement for the old SlidingSyncObserver.did_receive_sync_update so we have something to hook up the session verification banner and initial loading indicator to.

[stefanceriu]

@Hywan Like we talked about today we should introduce custom RoomListService list states: not-loaded, partially loaded (from cache or growing underneath) and loaded

[stefanceriu]

@Hywan We also talked about removing duplicates (checking move operations i.e delete + insert) and doing transactional diff publishing on both the room list and the timeline