Unread channels appear as if read after a few minutes

[Xaphiosis]

Summary

On an MM server with data migrated via the REST API, all unread channels stop looking unread after a few minutes (including any notification indicator on tab in web client).

Steps to reproduce

This is difficult; we do not know why this happened, and have created a second server with identical setup but without the REST API migration where works fine. We do not want to undertake migration again and have the same thing happen.

Both servers are self-hosted mattermost on Ubuntu 22.04:

Mattermost Version: 9.6.1
Database Schema Version: 119
Build Number: 8390314674
Database: postgres

For our server with the REST API migrated messages, the steps to reproduce are:

1. have someone else post a message in a channel you don't have open
2. channel appears unread, notification and unread status show up in tab for web client
3. do nothing for a while (some minutes)
4. channel now appears read, notification status in tab for web client is gone, it looks as if there's nothing new
5. refresh the web client, channel appears unread again; steps 3-5 can be repeated indefinitely
   Additionally:

• trying to "Mark as Unread" a channel after the timeout above has passed results in no visible effect until a refresh

For our new, identical setup Mattermost server minus the bots and imported messages, with identical config as far as we can tell, everything works fine.

Expected behavior

An unread channel keeps appearing unread when the user has not opened it or read it.

Observed behavior (that appears unintentional)

We don't see anything interesting showing up in the logs, the channels just switch to appearing unread.

Possible fixes

Unknown. We are looking for someone with Mattermost knowledge to assist with possible causes that would make an unread channel appear as read, and how we could go about debugging this.
This could have something with having created bot users to produce messages via the REST API, but we don't see how that would result in this behaviour.

I have come across someone having vaguely similar problem here: https://forum.mattermost.com/t/channels-with-unread-messages-do-not-appear-bold-and-bright/17933
but sadly there's no indication of what could cause this problem, and in their case marking channels as unread works fine.
All other issues I've come across are about things staying unread despite being read, and thus not related.

We would be very grateful for any assistance with this, as we have not been able to figure out any mechanism ourselves by which such a behaviour is possible.

[Xaphiosis]

We have just tried another migration, this time using mmctl export, and now our new server has picked up strange behaviour with not notifying us of unread messages in channels. This is surprising behaviour, what possible data could have been imported that results in such behaviour?!

[lsf37]

After wiping the new instance that showed the same problem, starting from scratch and sanitising the data gained from mmctl export, and then importing again, it looks like we may have solved the issue. What we have done to sanitise the data is:

• set all msg_count and msg_count_root values to 0 for all channels in the user records
• the last_viewed_at value for the town hall channel was 5 years in the future for all users. Reset that to a point in the past.
• there was one channel with an invalid deleted_at value that the import complained about. Reset that to 0.
• the import to the previous instance via the REST API produced posts that had properties set "from_webhook":"true" and "override_username":".." with the post being from another user (the one who did the import). This was the case also for direct channels that user was not a part of. We substituted user with override_username and removed that property.

After those changes, the import worked cleanly and the new instance seems to be showing unread notifications correctly so far.

It's still unclear to us which of the changes above was the reason for fixing the failure and what to do when it happens again.

[amyblais]

Per Mattermost guidelines, GitHub issues are mainly used for bug reports: https://handbook.mattermost.com/contributors/contributors/ways-to-contribute/.

For troubleshooting questions, would you be open to using https://forum.mattermost.com/? You can also post your question at https://community.mattermost.com/core/channels/peer-to-peer-help.

[lsf37]

Well, this is clearly a bug -- it shouldn't be possible to get into such a state just via posting messages and REST API.

[amyblais]

Opened https://mattermost.atlassian.net/browse/MM-58023.

[agnivade]

Hi @Xaphiosis - could you clarify what the issue is at the moment. From what I understand, at first you did "data migrated via the REST API, ", and then you migrated via mmctl export. The first time, it looked like unread channels appear read. And then the second time, you were not getting notified of unread channels at all. Is that accurate?

I don't know what is "REST API migration". So it would be good if you could clarify that.