Message history or retention period

[phillip-white-sociomantic]

Is it currently possible to set the history period the messages are kept? I imagine this needs to be somewhere otherwise eventually the server will get slow with thousands of messages. Most chat services have a purging feature which can be configured.

[timabbott]

There is currently no way to automatically purge old messages -- many users value the ability to see their message history from years ago, and so that's the default.

There are no material performance issues with storing 50M messages in a single database server's history (though eventually you start to need more disk).

That said, if this feature were desired to save disk (or just to be able to have one's data eventually go away!), it shouldn't be too hard to write (the main challenge being handling caching properly), though we'd want to be very careful about testing since buggy deletion code is a great way to delete all the messages :).

[phillip-white-sociomantic]

ok, understood

[timabbott]

@blablacio is working on this feature!

[timabbott]

Here's some notes on how we should implement this feature:

• We'll add a retention policy feature for the realm specifying how long messages are kept before being automatically deleted. Implementation-wise, I'd like to use a model where after the N days the messages are deleted from Message, but added to a new table e.g. ArchivedMessage, and kept there for a week before being deleted finally, so that we have time to catch bugs for data is lost permanently.
• I think it will work to create models like 'class ArchivedMessage(Message)' that inherits from Message and a similar table inheriting from UserMessage, and just add to the new table whatever field(s) we need to keep track of when they'll be deleted. We might need to some set Django meta fields on the classes to ensure that ArchivedMessage rows are only present in the new tables. That approach should minimize the maintenance involved when we modify the Message tables.
• Since bugs in this system would result in data loss, we should make sure to have a really good test suite for this feature.
• The command-line tool should be a Django management command that calls library functions inside a new file zerver/lib/retention.py, similar to how zerver/management/commands/fill_memcached_caches.py works.
• Eventually we'll add a cron job in puppet/zulip/files/cron to run the management command.
• Once we do the Message stuff correctly, we'll need to do corresponding changes in Attachment and MessageReaction to handle this properly.

I'd like to merge incremental pieces of this implementation as we go rather than building the whole thing before we merge anything. e.g., first add tooling to just determine which messages should be archived with good tests, then add a function to move a set of messages from Message to ArchivedMessage rows with tests, then a function to do the same with ArchivedUserMessage, etc. Ideally we'd be merging code for this feature every day or two, with good tests, rather than building up a big monolithic implementation.

[kkanahin]

@timabbott I've done first stage(added tool for getting expired messages). Please review PR #2119. I need your feedback to continue work on it.