A 1:1 chat that received an older message is placed out of chronological order in the chat list when collected after an account restore #5088

gerryfrancis · 2023-12-06T15:22:59Z

Operating System (Linux/Mac/Windows/iOS/Android):
Android 11.
Delta Chat Version:
1.42.6 (nightly built 2023-12-05).
Expected behavior:
In chat overview, messages are listed in chronological order (by sent date/time) after a backup has been restored.
Actual behavior:
In chat overview, messages are not listed in chronological order (by sent date/time) after a backup has been restored. A 1:1 chat with its last incoming message sent way earlier is put on top of the chat list.
Steps to reproduce the problem:
I did not verify these steps due to time shortage and complexity, but I will try my best to summarize. Let's imagine we are Alice:
-- Alice disables "Move to Delta Chat folder", so emails of incoming messages are left in the IMAP Inbox folder.
-- Alice receives some messages from various chats in her Delta Chat client.
-- Alice performs a backup of her Delta Chat account.
-- Alice sends her QR invitation code to her already-known contact Deep Sea by copying/pasting it from the clipboard into the input field of their 1:1 chat, and by tapping the "Send" button afterwards.
-- Deep Sea reads and verifies Alice's QR code, so an automated verification email is sent by Deep Sea's Delta Chat client back to Alice's.
-- Deep Sea sends a "Test" message in a 1:1 chat to Alice.
-- After Alice has read Deep Sea's "Test" message, she receives quite a lot of various emails from various (already-known) contacts, except from Deep Sea. She reads all the messages, except those from contacts who are unknown to her.
-- Alice un- and reinstalls Delta Chat, and restores her Delta Chat account from the existing backup. Emails of a younger date/time are fetched from the IMAP Inbox folder to complete the process. (Result: The 1:1 chat with Deep Sea is listed on top of the chat list, right below the "Archived" group, although the last message in the chat is older than the ones in other chats.)
Screenshots:

(Note: The chat that is positioned between "Archived" and "Deep Sea" contains a message that Alice just sent off.)
Logs:
N/A.
Attachments:
-- Verification email sent from Deep Sea to Alice: email_1.txt
-- Email with "Test" message sent from Deep Sea to Alice: email_2.txt

iequidoo · 2023-12-06T23:29:43Z

Reproduced this

iequidoo · 2023-12-07T00:12:09Z

So, this is the reason:

Msg#156: info (Contact#Contact#Info): Messages are guaranteed to be end-to-end encrypted from now on. [NOTICED][INFO 🛡] [2023.12.06 20:42:23]
Msg#158🔒: iequidoo (Contact#Contact#15): Test [SEEN] [2023.12.06 20:23:30]

The "Messages are guaranteed to be end-to-end encrypted from now on." message gets the sort timestamp corresponding to the time when Inbox is fetched after the backup import. So, the next "Test" message has more recent sort timestamp (to be sorted to the end of the chat) while having the original "sent" timestamp.

iequidoo · 2023-12-07T00:25:14Z

Btw, things are already done in the right way for groups -- the "sent" timestamp is always passed to ChatId::create_multiuser_record() so the protection message has the correct sort timestamp calculated from the "sent" one. Probably the fix is to do the same for 1:1 chats.

gerryfrancis · 2023-12-07T05:12:26Z

@iequidoo Thank you so much for reproducing the issue and finding its cause! :)

…() (#5088) Before in some places it was correctly calculated by passing the "sent" timestamp to `calc_sort_timestamp()`, but in other places just the system time was used. In some complex scenarios like #5088 (restoration of a backup made before a contact verification) it led to wrong sort timestamps of protection messages and also messages following by them. But to reduce number of args passed to functions needing to calculate the sort timestamp, add message timestamps to `struct MimeMessage` which is anyway passed everywhere.

…f a new message (#5088) Drafts mustn't affect sorting of any other messages, they aren't even displayed in the chat window. Also hidden messages mustn't affect sorting of usual messages. But let hidden messages sort together with protection messages because hidden messages also can be or not be verified.

…() (#5088) Before in some places it was correctly calculated by passing the "sent" timestamp to `calc_sort_timestamp()`, but in other places just the system time was used. In some complex scenarios like #5088 (restoration of a backup made before a contact verification) it led to wrong sort timestamps of protection messages and also messages following by them. But to reduce number of args passed to functions needing to calculate the sort timestamp, add message timestamps to `struct MimeMessage` which is anyway passed everywhere.

…f a new message (#5088) Drafts mustn't affect sorting of any other messages, they aren't even displayed in the chat window. Also hidden messages mustn't affect sorting of usual messages. But let hidden messages sort together with protection messages because hidden messages also can be or not be verified, so let's preserve this information -- even it's not useful currently, it can be useful in the future versions.