Test cases for stuck unreads and notifs

[andybalaam]

Tests we are planning to write:

Current count = 146 test cases. List of test cases:

$ git diff develop..andybalaam/read-status-tests | sed -n 's/\(test\|describe\)("\(.*\)".*/\2/p'
@@ -354,4 +354,341 @@ Read receipts
+    new messages
+        in the main timeline
+            Sending a message makes a room unread
+            Reading latest message makes the room read
+            Reading an older message leaves the room unread
+            Marking a room as read makes it read
+            Sending a new message after marking as read makes it unread
+            A room with a new message is still unread after restart
+            A room where all messages are read is still read after restart
+        in threads
+            Sending a message makes a room unread
+            Reading the last threaded message makes the room read
+            Reading a thread message makes the thread read
+            Reading an older thread message (via permalink) leaves the thread unread
+            Reading only one thread's message does not make the room read
+            Reading only one thread's message make that thread read but not others
+            Reading the main timeline does not mark a thread message as read
+            Marking a room with unread threads as read makes it read
+            Sending a new thread message after marking as read makes it unread
+            A room with a new threaded message is still unread after restart
+            A room where all threaded messages are read is still read after restart
+        thread roots
+            Reading a thread root does not mark the thread as read
+            Reading a thread root within the thread view marks it as read in the main timeline
+            Creating a new thread based on a reply makes the room unread
+            Reading a thread whose root is a reply makes the room read
+    editing messages
+        in the main timeline
+            Editing a message makes a room unread
+            Reading an edit makes the room read
+            Marking a room as read after an edit makes it read
+            Editing a message after marking as read makes the room unread
+            Editing a reply after reading it makes the room unread
+            Editing a reply after marking as read makes the room unread
+            A room with an edit is still unread after restart
+            A room where all edits are read is still read after restart
+        in threads
+            An edit of a threaded message makes the room unread
+            Reading an edit of a threaded message makes the room read
+            Marking a room as read after an edit in a thread makes it read
+            Editing a thread message after marking as read makes the room unread
+            A room with an edited threaded message is still unread after restart
+            A room where all threaded edits are read is still read after restart
+        thread roots
+            An edit of a thread root makes the room unread
+            Reading an edit of a thread root makes the room read
+            Marking a room as read after an edit of a thread root makes it read
+            Editing a thread root after marking as read makes the room unread
+            Marking a room as read after an edit of a thread root that is a reply makes it read
+            Editing a thread root that is a reply after marking as read makes the room unread but not the thread
+    reactions
+        in the main timeline
+            Reacting to a message makes a room unread
+            Reading a reaction makes the room read
+            Marking a room as read after a reaction makes it read
+            Reacting to a message after marking as read makes the room unread
+            A room with a reaction is still unread after restart
+            A room where all reactions are read is still read after restart
+        in threads
+            A reaction to a threaded message makes the room unread
+            Reading a reaction to a threaded message makes the room read
+            Marking a room as read after a reaction in a thread makes it read
+            Reacting to a thread message after marking as read makes the room unread
+            A room with a reaction to a threaded message is still unread after restart
+            A room where all reactions in threads are read is still read after restart
+        thread roots
+            A reaction to a thread root makes the room unread
+            Reading a reaction to a thread root makes the room read
+            Marking a room as read after a reaction to a thread root makes it read
+            Reacting to a thread root after marking as read makes the room unread but not the thread
+    redactions
+        in the main timeline
+            Redacting the message pointed to by my receipt leaves the room read
+            Redacting a message after it was read makes the room unread
+            Reading an unread room after a redaction of the latest message makes it read
+            Reading an unread room after a redaction of an older message makes it read
+            Marking an unread room as read after a redaction makes it read
+            Sending and redacting a message after marking the room as read makes it unread
+            ?? Redacting a message after marking the room as read makes it unread
+            Reacting to a redacted message leaves the room read
+            Editing a redacted message leaves the room read
+            ?? Reading a reaction to a redacted message marks the room as read
+            ?? Reading an edit of a redacted message marks the room as read
+            Reading a reply to a redacted message marks the room as read
+            A room with an unread redaction is still unread after restart
+            A room with a read redaction is still read after restart
+        in threads
+            Redacting the threaded message pointed to by my receipt leaves the room read
+            Redacting a threaded message after it was read makes the room unread
+            Reading an unread thread after a redaction of the latest message makes it read
+            Reading an unread thread after a redaction of an older message makes it read
+            Marking an unread thread as read after a redaction makes it read
+            Sending and redacting a message after marking the thread as read makes it unread
+            ?? Redacting a message after marking the thread as read makes it unread
+            Reacting to a redacted message leaves the thread read
+            Editing a redacted message leaves the thread read
+            ?? Reading a reaction to a redacted message marks the thread as read
+            ?? Reading an edit of a redacted message marks the thread as read
+            Reading a reply to a redacted message marks the thread as read
+            A thread with an unread redaction is still unread after restart
+            A thread with a read redaction is still read after restart
+            A thread with an unread reply to a redacted message is still unread after restart
+            A thread with a read replt to a redacted message is still read after restart
+        thread roots
+            Redacting a thread root after it was read leaves the room read
+            Redacting a thread root after it was read makes the room unread
+            Redacting the root of an unread thread makes the room read
+            Sending a threaded message onto a redacted thread root leaves the room read
+            Reacting to a redacted thread root leaves the room read
+            Editing a redacted thread root leaves the room read
+            Replying to a redacted thread root makes the room unread
+            Reading a reply to a redacted thread root makes the room read
+    messages with missing referents
+        A message in an unknown thread is not visible and the room is read
+        When a message's thread root appears later the thread appears and the room is unread
+        An edit of an unknown message is not visible and the room is read
+        When an edit's message appears later the edited version appears and the room is unread
+        A reaction to an unknown message is not visible and the room is read
+        When an reactions's message appears later it appears and the room is unread
+    receipts with missing events
+        A receipt for an unknown message does not change the state of an unread room
+        A receipt for an unknown message does not change the state of a read room
+        A threaded receipt for an unknown message does not change the state of an unread thread
+        A threaded receipt for an unknown message does not change the state of a read thread
+        A threaded receipt for an unknown thread does not change the state of an unread thread
+        A threaded receipt for an unknown thread does not change the state of a read thread
+        A threaded receipt for a message on main does not change the state of an unread room
+        A threaded receipt for a message on main does not change the state of a read room
+        A main receipt for a message on a thread does not change the state of an unread room
+        A main receipt for a message on a thread does not change the state of a read room
+        A threaded receipt for a thread root does not mark it as read
+    Message ordering
+        in the main timeline
+            A receipt for the last event in sync order (even with wrong ts) marks a room as read
+            A receipt for a non-last event in sync order (even when ts makes it last) leaves room unread
+        in threads
+            A receipt for last threaded event in ts order (even when it was received non-last) marks a thread as read
+            A receipt for non-last threaded event in ts order (even when it was received last) leaves thread unread
+            A receipt for last threaded edit in ts order (even when it was received non-last) marks a thread as read
+            A receipt for non-last threaded edit in ts order (even when it was received last) leaves thread unread
+            A receipt for last threaded reaction in ts order (even when it was received non-last) marks a thread as read
+            A receipt for non-last threaded reaction in ts order (even when it was received last) leaves thread unread
+        thread roots
+            A receipt for last reaction to thread root in sync order (even when ts makes it last) marks room as read
+            A receipt for non-last reaction to thread root in sync order (even when ts makes it last) leaves room unread
+            A receipt for last edit to thread root in sync order (even when ts makes it last) marks room as read
+            A receipt for non-last edit to thread root in sync order (even when ts makes it last) leaves room unread
+    Ignored events
+        If all events after receipt are unimportant, the room is read
+        Sending an important event after unimportant ones makes the room unread
+        A receipt for the last unimportant event makes the room read, even if all are unimportant
+    Paging up
+        Paging up through old messages after a room is read leaves the room read
+        Paging up through old messages of an unread room leaves the room unread
+        Paging up to find old threads that were previously read leaves the room read
+        ?? Paging up to find old threads that were never read marks the room unread
+        After marking room as read, paging up to find old threads that were never read leaves the room read
+    Room list order
+        Rooms with unread threads appear at the top of room list if 'unread first' is selected
+    Notifications
+        in the main timeline
+            A new message that mentions me shows a notification
+            Reading a notifying message reduces the notification count in the room list, space and tab
+            Reading the last notifying message removes the notification marker from room list, space and tab
+            Editing a message to mentions me shows a notification
+            Reading the last notifying edited message removes the notification marker
+            Redacting a notifying message removes the notification marker
+        in threads
+            A new threaded message that mentions me shows a notification
+            Reading a notifying threaded message removes the notification count
+            Notification count remains steady when reading threads that contain seen notifications
+            Notification count remains steady when paging up thread view even when threads contain seen notifications
+            Notification count remains steady when paging up thread view after mark as unread even if older threads contain notifications
+            Redacting a notifying threaded message removes the notification marker

(Previous, less-organised notes follow.)

This is a list of test cases we should think about for stuck unreads and notifs. Some of them may already be covered, but this is a single place to try and collect ideas of every case we should cover.

For now, it's disorganised, but eventually once we have enough ideas we can make it more systematic.

• When we receive a reaction/edit to a thread root:

  • Is the main thread marked as unread?
  • Is the thread marked as unread?
  • Do we agree with Synapse on these cases (when it's a notif)?
  • Do we send receipts in both places when you read that reaction?

• When we receive a reaction/edit to a message that we don't have:

  • How can we decide which thread it is on? We could fetch its parent, but before that arrives we either have to hold on to it somewhere, or dump it on the main thread (which is what we do now)
  • If we receive a receipt for it on a thread, we should "know" it's on the thread and consider it read, even if we ended up puttin it on the wrong thread initially (e.g. we dumped it on the main thread because we didn't know where it should go)

• When a threaded receipt points to a thread root

  • Then if there are other events in the thread (presumably there are...) then the thread is unread

• If we fill in a thread later by calling the relations API (and recursive relations is OFF)

  • We should build the messages in the thread in the order we receive them, but then we fetch the reactions/edits to those messages later, and they arrive out of order
  • We should insert those reactions/edits in "sync order". For now, we guess at this because we don't have the sync order.
  • We should reflect this simulated order in which receipts we send i.e. we should not send receipts for events we are inserting early in the order
  • We should reflect this simulated order in how we consider receipts we receive i.e. we should:
    • consider receipts for old events (that we inserted earlier) not to mark newer events as read
    • consider receipts for newer events (before those we inserted) to make those older events as read

• Unthreaded receipts:

  • Mark threads and the main thread as unread for messages before them in sync order.
  • Marks earlier thread as read even if the receipt points at a message that is not on the thread
  • Marks thread root as read in both the main timeline and the thread root
  • Marks reaction/edit to thread root as read in both the main timeline and the thread root

• If our receipt points to an event we don't have

  • This could be because the event is old, so recent events mean we should consider the room/thread unread
  • BUT what if all the new events are "unimportant" e.g. name changes? Then we don't know. We can backfill to try and find out, but in the meantime we have to guess
  • Should we automatically send read receipts for "unimportant" events, even if you don't open the room?

[andybalaam]

Re-opening as we committed part of the work in matrix-org/matrix-react-sdk#11301 but we left lots of tests as skip so we have more to do.

[jtrees]

Will these tests also get implemented for other clients/sdks?

[andybalaam]

Will these tests also get implemented for other clients/sdks?

Hi @jtrees that would be for those clients and sdks to decide. If anyone wanted to collaborate on some kind of shareable test case list or clever (maybe trafficlight based?) cross-client test setup I'd be very excited to talk to them!

[dbkr]

We (mostly florian) went through and fixed all the ones that were skipped, so we believe this is actually done now.