api: add `ChatListChanged` and `ChatListItemChanged` events #4476

Simon-Laux · 2023-06-15T13:07:38Z

Todo:

Make sure all cases have events
Make sure all new cases will have events (this todo can be converted to an issue)
- add missing cases after solving Contact recently seen changes have missing events #5377
- maybe a case for feat: show reactions in summaries #5387 if that requires an update event
make a desktop pr using this (chat list update event deltachat-desktop#3268)
~~add debouncing of those events in core (draft in chatlistitem UI events debouncing #5171)~~ let's do this in a dedicated pr, if we need it at all
Test if all cases work (manually with desktop once this pr is ready to be merged)
~~Make a list for testers with all test-cases (I already have a list, but it's not readable by testers)~~ - I think it's not needed anymore - because of the automated test
Make tests for the majority of cases
rebase and remove openrpc.json from git history
~~rename DC_EVENT_UI_CHATLIST_CHANGED to something like DC_EVENT_UI_CHATLIST_ORDER_CHANGED~~
- clarified docs instead

missing optional tests:

Chatlist correctly updated after AEAP (on what devices?) (maybe not needed because contact name change is already tested and AEAP is still experimental)
change protection (1:1 chat gets guranteed encryption) (maybe triggered by new message anyway?)
securejoin dm chat with contact (group join is already tested)

Why now:

This makes desktop code simpler (only 3 cases and 2 events to listen to)
This should fix issues that currently are there or are sometimes hidden by a strange workaround in desktop
going actively over all cases will help us to fix chatlist bugs where it is not updated when it should be

deltachat-jsonrpc/openrpc/openrpc.json

r10s · 2023-06-21T08:03:23Z

ftr, this pr picks up suggestions and discussions from previously closed #2850

iequidoo · 2024-03-18T05:42:05Z

src/events/payload.rs

@@ -301,8 +301,8 @@ pub enum EventType {

    /// Inform UI that a single chat list item changed and needs to be rerendered
    /// If `chat_id` is set to None, then all currently visible chats need to be rerendered, and all not-visible items need to be cleared from cache if the UI has a cache.
-    UIChatListItemChanged {
-        /// The id of the changed chat or None for all chats.


Why the documentation for None was removed?

iequidoo · 2024-03-18T05:58:12Z

src/contact.rs

+
+        if updated_name {
+            // update the chats the contact that changed their name is part of
+            // (treefit): could make sense to only update chats where the last message is from the contact, but the db query for that is more expensive


But this loop can be expensive too, we don't know how many chats we have with the contact. Though even if we take only chats with the last message from this contact, we don't know. I'd avoid any loops and just emit an event with chat_id: None. Better to let the app decide which chats to update (those which are on the screen currently) and rate-limit these updates. Another solution is to add LIMIT to the SELECT statement and if we hit the limit, emit an event with chat_id: None, otherwise emit separate events for chats

src/events/ui_events.rs

r10s · 2024-03-18T12:35:50Z

i know, this is still a draft, just some thought:

up to now, events emitted by our event system are "cheap", meaning, no extra database access or calculation is needed for emitting events. the events are just emitting what is known anyways.

this PR partly introduces "expensive" events that need additional database calls, currently, only when a contact name is changed, however DC_EVENT_UI_CHATLIST_ITEM_CHANGED will probably be needed in more cases where eg. contacts-table or other related tables are changed. this is comparable expensive to calculate, leading to potentially many events. not sure if that is helping if on the other hand debouncing in core is needed for desktop.

as there is already a DC_EVENT_UI_CHATLIST_CHANGED maybe it is better to just fire that on contact-db changes, imu UI is responsible for reloading the list then and needs to deal with that anyways, that should be just fine in case joined data are updated (DC_EVENT_UI_CHATLIST_ITEM_CHANGED can still be fired in cases where the information is already there and "cheap")

if we stay with "expensive" events, we should make the code optional by a compiler switch or so; there is no need to slow down iOS, Android (and probably UbuntuTouch) that do not need these events

Simon-Laux · 2024-03-18T19:39:58Z

true emit_chatlist_items_changed_for_contact (update name or aeap) is quite expensive and could just be refresh the whole list (not order, just items)
emit_chatlist_item_changed_for_contacts_dm_chat (profile image changed, recently seen changed) on the other hand should stay IMO, as it is less expensive, If we need it we could also cache dm chat if it exists in the contacts table to make accessing it even faster.

also there is currently the question of how tho integrate it:
A. make a separate loop for ui events that deduplicates until the next event is requested from it
B. A but integrated in existing event loop (I would need link2xt's help for that, as I'm not deep enough into the future/async stuff)
C. do not dedup the events as it is fast enough, maybe add a dedicated event for 1:1/dm chat update to also save the db query and move the burden of that to the frontend
D. some combination/mix of other options or your idea?

Currently this pr has the following events/cases:

Order changed or what chatlist items are displayed (list of chat ids, not chat list item content): DC_EVENT_UI_CHATLIST_ITEM_CHANGED
content of one chatlistitem should be reloaded: DC_EVENT_UI_CHATLIST_ITEM_CHANGED (chat_id: Some(id))
content of all (cached) chatlistitems should be reloaded: DC_EVENT_UI_CHATLIST_ITEM_CHANGED (chat_id: None)

maybe we should more events:

contact changed so update the dm chat for the following contact id, but the ui would just also call core to convert contact id to dm chat id
reload all chats where contact name is in last message (ui could look in cache for this, when we add the contact id of the last message to the jsonrpc chatlistitem), but we can also just invalidate the cache and say all items should be reloaded

link2xt

I don't understand what UIEventsState does and why is it needed.

If it is for debouncing, I suggest to split this PR into PR that simply adds events and another PR on top of this that adds debouncing for these two events.

Last time we discussed debouncing for UI events my proposal was to remember highest Event.id of each event we want to debounce inside the EventEmitter (e.g. remember highest id of UIChatListItemChanged { chat_id: ChatId(5) }, UIChatListItemChanged { chat_id: ChatId(10) }, UIChatListChanged separately. Then when we try to retrieve event from EventEmitter we check if the item we just retrieved from the queue is not the newest one pushed (i.e. there is already the same item inside the queue that will be retrieved later) and get the next item from the queue if this is the case.

deltachat-ffi/deltachat.h

link2xt · 2024-03-19T22:10:44Z

deltachat-ffi/deltachat.h

+#define DC_EVENT_UI_CHATLIST_CHANGED              2300
+
+/**
+ * Inform UI that all or a single chat list item changed and needs to be rerendered


chat_id = 0 means that multiple chatlist item have changed, not "all"?

should invalidate cache and reload all items that are visible or where in cache in the ui - not sure how to word that

deltachat-jsonrpc/src/api/types/events.rs

src/events/payload.rs

src/events/ui_events.rs

Simon-Laux · 2024-03-20T20:13:55Z

src/contact.rs

+    }
+
+    /// Get all chats the contact is part of
+    pub async fn get_chats_with_contact(


do we want to keep this method, even though we probably won't use it anymore in this pr:

it't to expensive to run in emitting events compared to telling the ui just to reload the whole list

on the other hand this method could be useful on it's own maybe?

But currently at least in DC Desktop you can click on the contact and see the "Shared Chats" list. How does it work w/o such a method?

I have not checked, but current UIs probably use dc_get_chatlist with query_id set to contact ID.

iequidoo · 2024-03-21T00:46:14Z

src/contact.rs

+    }
+
+    /// Get all chats the contact is part of
+    pub async fn get_chats_with_contact(


But currently at least in DC Desktop you can click on the contact and see the "Shared Chats" list. How does it work w/o such a method?

iequidoo · 2024-03-21T01:10:54Z

src/contact.rs

@@ -1603,7 +1603,7 @@ pub(crate) async fn set_profile_image(
    if changed {
        contact.update_param(context).await?;
        context.emit_event(EventType::ContactsChanged(Some(contact_id)));
-        ui_events::emit_chatlist_item_changed_for_contacts_dm_chat(context, contact_id);
+        ui_events::emit_chatlist_item_changed_for_contact_chat(context, contact_id).await;


Why is it not sufficient for the app to know the contact ID? We better add contact_id + chat_type (1:1, group, etc.) to struct Summary so that the app can iterate over summaries and quickly find the chatlist item changed w/o doing extra db queries.

this is about the dm/1:1 chat with that contact (profile image and recently online), the db query for that should be cheap and saves us a round trip to core. But sure we can also discuss creating new dedicated events for it, my idea here is to have some small amount of chatlist events that the ui listens for. (with the motto: if you listen for all events with chatlist in the name you are set)

The name update is the expensive query, I already changed that to reload all cached items.

Yes, this db query is cheap, but if we just add all necessary info to struct Summary, we don't need any additional events at all. So, having EventType::ContactsChanged is sufficient. Also we don't need an extra round trip to the core because, i guess, the app already has the list of all summaries (it should be built to display the chat list) and can iterate over them (actually only over currently visible ones) to find out which items has changed.

Having all these different events to listen to is a mess, my goal here is to move the concern to core to give out events when the (granted virtual) resource of chatlist changes (be it order or item content).

I also plan to add tests for all the cases to core.

So in my mind we could also do this (I don't want to discuss exact naming at this step):

for avatar and recently online changes

add contact id to chat list items of DM chats

add ChatlistContactDirectMessagesChatChanged(contactId) event

-> currently in this case core gets the dm chat id and just updates that without the ui needing to do anything extra for this case

for contact name (and AEAP as it can change email and thus display name for contacts without a name)

add contact ids for all summaries

add ChatlistContactNameChanged(contactId) event

-> currently core tells ui to reload all cached/visible chatlistitems, using this event it could be slightly more efficient, but unclear if it's worth the extra ui effort as I'm sure someone will want to use the invalidate/reload all items escape hatch in the future for other things anyways.

I am against reusing EventType::ContactsChanged as those events are not precise enough and you have many of them to handle, so a new feature comes around and it needs a new event to be listened to in the UI, the point of this pr/effort is to make the ui part much easier that you exactly know which few events you need to handle to have handled all cases for when the chatlist updates, ideally as precise as (reasonable in db time constrains) possible, that the lazy virtual list in desktop can be partially updated as needed and not reload all chats when one of them changes.

deltachat-ffi/src/lib.rs

link2xt · 2024-04-05T13:16:25Z

Backup transfer in tests fail because of #5431

link2xt · 2024-04-07T14:22:47Z

@Simon-Laux Could you rebase this on top of main? Backup transfer should be fixed now that #5439 is merged.

Co-authored-by: link2xt <link2xt@testrun.org>

(te test was for groups, the chat is created later in that case)

deltachat-rpc-client/tests/test_chatlist_events.py

src/mimeparser.rs

Co-authored-by: link2xt <link2xt@testrun.org>

This was referenced Jun 15, 2023

introduce UIChatListItemChanged event #4475

Closed

chat list update event deltachat/deltachat-desktop#3268

Merged

link2xt reviewed Jun 21, 2023

View reviewed changes

deltachat-jsonrpc/openrpc/openrpc.json Outdated Show resolved Hide resolved

Simon-Laux self-assigned this Sep 28, 2023

link2xt deleted the branch main October 25, 2023 21:22

link2xt closed this Oct 25, 2023

link2xt reopened this Oct 25, 2023

Simon-Laux changed the base branch from master to main October 26, 2023 11:07

link2xt force-pushed the main branch from ca190eb to d7aecab Compare November 10, 2023 17:56

Simon-Laux force-pushed the simon/UIChatListItemChangedEvent2 branch from 1f5cdc4 to 7483aa5 Compare January 11, 2024 01:49

Simon-Laux force-pushed the simon/UIChatListItemChangedEvent2 branch from 2c37341 to bdfd970 Compare January 23, 2024 02:05

link2xt force-pushed the main branch 2 times, most recently from 1abb12e to 2af9ff1 Compare March 4, 2024 21:10

Simon-Laux force-pushed the simon/UIChatListItemChangedEvent2 branch from bdfd970 to 54ab9c3 Compare March 17, 2024 20:11

iequidoo reviewed Mar 18, 2024

View reviewed changes

link2xt reviewed Mar 19, 2024

View reviewed changes

Simon-Laux commented Mar 20, 2024

View reviewed changes

iequidoo reviewed Mar 21, 2024

View reviewed changes

link2xt reviewed Mar 22, 2024

View reviewed changes

deltachat-ffi/src/lib.rs Outdated Show resolved Hide resolved

Simon-Laux changed the title ~~feat: add UIChatListChanged and UIChatListItemChanged events~~ feat: add ChatListChanged and ChatListItemChanged events Mar 25, 2024

Simon-Laux mentioned this pull request Apr 2, 2024

feat: show reactions in summaries #5387

Merged

Simon-Laux force-pushed the simon/UIChatListItemChangedEvent2 branch from 688233d to 487db7d Compare April 2, 2024 19:40

Simon-Laux marked this pull request as ready for review April 2, 2024 20:14

Simon-Laux force-pushed the simon/UIChatListItemChangedEvent2 branch from 30ac692 to 061a63e Compare April 4, 2024 17:45

Simon-Laux mentioned this pull request Apr 5, 2024

Add stockstrings for reactions in chatlist & update chatlistitem on reaction event deltachat/deltachat-desktop#3749

Merged

Simon-Laux and others added 10 commits April 11, 2024 23:58

add test_block_contact_request

10a0a2c

add hint to get_backup that you need to start io afterwards

eb30768

fix multidevice tests

3fa8619

rename module to chatlist_events

728c80d

cargo fmt

57c0b12

fix python lint

f23a817

Update deltachat-jsonrpc/src/api.rs

d808db1

Co-authored-by: link2xt <link2xt@testrun.org>

fix vscode autocomplete in a better way

c06809f

use Account.wait_for_incoming_msg_event

d40aa73

don't use raw _rpc for methods for methods that don't need it

851a622

Simon-Laux force-pushed the simon/UIChatListItemChangedEvent2 branch from 5e94feb to 851a622 Compare April 11, 2024 21:59

fix securejoin chatlistevents test

07941e4

(te test was for groups, the chat is created later in that case)

link2xt reviewed Apr 12, 2024

View reviewed changes

deltachat-rpc-client/tests/test_chatlist_events.py Outdated Show resolved Hide resolved

Simon-Laux added 4 commits April 15, 2024 22:57

apply suggestion

221dda6

remove "order" from function naming

c7cac04

wait for both events

c6f7b0c

rm unused function I added in this pr get_chats_with_contact

4226f26

link2xt reviewed Apr 15, 2024

View reviewed changes

src/mimeparser.rs Outdated Show resolved Hide resolved

Update src/mimeparser.rs

5b0012e

Co-authored-by: link2xt <link2xt@testrun.org>

link2xt approved these changes Apr 15, 2024

View reviewed changes

link2xt enabled auto-merge (squash) April 15, 2024 22:20

link2xt disabled auto-merge April 15, 2024 22:20

link2xt enabled auto-merge (squash) April 15, 2024 22:21

link2xt changed the title ~~feat: add ChatListChanged and ChatListItemChanged events~~ api: add ChatListChanged and ChatListItemChanged events Apr 15, 2024

link2xt merged commit f9465f7 into main Apr 15, 2024
32 of 38 checks passed

link2xt deleted the simon/UIChatListItemChangedEvent2 branch April 15, 2024 22:35

This was referenced Apr 16, 2024

feat: experimental Webxdc Integration API, Maps Integration #5461

Merged

Split off functional contact tools into its own crate #5444

Merged

Simon-Laux mentioned this pull request Apr 16, 2024

fix test events::chatlist_events::test_chatlist_events::test_secure_join_group #5468

Merged

WofWca mentioned this pull request Jul 10, 2024

WIP: refactor: add failing test: arhive updated event #5764

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

api: add `ChatListChanged` and `ChatListItemChanged` events #4476

api: add `ChatListChanged` and `ChatListItemChanged` events #4476

Simon-Laux commented Jun 15, 2023 •

edited

Loading

r10s commented Jun 21, 2023

iequidoo Mar 18, 2024

iequidoo Mar 18, 2024 •

edited

Loading

r10s commented Mar 18, 2024 •

edited

Loading

Simon-Laux commented Mar 18, 2024 •

edited

Loading

link2xt left a comment

link2xt Mar 19, 2024

Simon-Laux Mar 20, 2024 •

edited

Loading

Simon-Laux Mar 20, 2024

iequidoo Mar 21, 2024

link2xt Mar 22, 2024

iequidoo Mar 21, 2024

iequidoo Mar 21, 2024

Simon-Laux Mar 22, 2024 •

edited

Loading

iequidoo Mar 22, 2024

Simon-Laux Mar 23, 2024 •

edited

Loading

link2xt commented Apr 5, 2024

link2xt commented Apr 7, 2024

api: add ChatListChanged and ChatListItemChanged events #4476

api: add ChatListChanged and ChatListItemChanged events #4476

Conversation

Simon-Laux commented Jun 15, 2023 • edited Loading

r10s commented Jun 21, 2023

Choose a reason for hiding this comment

iequidoo Mar 18, 2024 • edited Loading

Choose a reason for hiding this comment

r10s commented Mar 18, 2024 • edited Loading

Simon-Laux commented Mar 18, 2024 • edited Loading

link2xt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Simon-Laux Mar 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Simon-Laux Mar 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Simon-Laux Mar 23, 2024 • edited Loading

Choose a reason for hiding this comment

link2xt commented Apr 5, 2024

link2xt commented Apr 7, 2024

api: add `ChatListChanged` and `ChatListItemChanged` events #4476

api: add `ChatListChanged` and `ChatListItemChanged` events #4476

Simon-Laux commented Jun 15, 2023 •

edited

Loading

iequidoo Mar 18, 2024 •

edited

Loading

r10s commented Mar 18, 2024 •

edited

Loading

Simon-Laux commented Mar 18, 2024 •

edited

Loading

Simon-Laux Mar 20, 2024 •

edited

Loading

Simon-Laux Mar 22, 2024 •

edited

Loading

Simon-Laux Mar 23, 2024 •

edited

Loading