Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MSC1763: Proposal for specifying configurable message retention periods #1763

Open
wants to merge 16 commits into
base: master
from

Conversation

Projects
None yet
9 participants
@ara4n
Copy link
Member

commented Dec 30, 2018

Rendered

@ara4n

This comment has been minimized.

Copy link
Member Author

commented Dec 30, 2018

This is an attempt to unblock #440 and #447 and solve the problem of how to specify configurable per-room and per-user message retention. This is particularly driven by Disroot & Hackint disabling their Matrix bridges due to concerns over unlimited message retention, and meanwhile there are also corporate Matrix users on the horizon who require configurable history retention. Hence trying to slay the beast for once and for all...

Show resolved Hide resolved proposals/1763-configurable-retention-periods.md Outdated
Show resolved Hide resolved proposals/1763-configurable-retention-periods.md Outdated
Show resolved Hide resolved proposals/1763-configurable-retention-periods.md Outdated
Show resolved Hide resolved proposals/1763-configurable-retention-periods.md
Show resolved Hide resolved proposals/1763-configurable-retention-periods.md
Show resolved Hide resolved proposals/1763-configurable-retention-periods.md
Show resolved Hide resolved proposals/1763-configurable-retention-periods.md
Show resolved Hide resolved proposals/1763-configurable-retention-periods.md
short max_lifetime and/or self_destruct set true which promptly self-destruct.

One solution for this could be for server implementations to implement a quarantine
mode which initially marks purged events as quarantined for N days before deleting

This comment has been minimized.

Copy link
@maxidorius

maxidorius Dec 30, 2018

Contributor

This is in direct conflict with the proposal mandatory behaviour:

they MUST prune the event from their DBs.

This comment has been minimized.

Copy link
@ara4n

ara4n Dec 30, 2018

Author Member

correct. it's a suggestion for an alternate approach.

@@ -0,0 +1,243 @@
# Proposal for specifying configurable retention periods for messages.

This comment has been minimized.

Copy link
@maxidorius

maxidorius Dec 30, 2018

Contributor

This proposal doesn't address the following points:

  • Precise and clear algorithm to decide if an event should be pruned taking all possible configurations/variables into account
  • Impact on all endpoints dealing with events in all specs (Client, Server, App, Identity, Push)
  • PDU signature with new fields
  • Redaction on the new fields
  • Pruning of state events
  • Resolution of DAG holes
  • Moderation of potentially abusive messages as "MUST be purged from DBs", leaving no way to moderators to inspect said messages

This comment has been minimized.

Copy link
@ara4n

ara4n Dec 30, 2018

Author Member

Impact on all endpoints dealing with events in all specs (Client, Server, App, Identity, Push)

What do you have in mind? We already purge events via the purge APIs without problems, and the end result here is no different.

This comment has been minimized.

Copy link
@maxidorius

maxidorius Dec 30, 2018

Contributor

Simple example: what if a event is supposed to disappear while it's being queued/processed by an application service, or a push server? Are they supposed to also reflect it into their local storage or in the remote network or to delete/rollback a push notification?

There are other situations which are pretty obvious if a proper review of all endpoints impacted by this is made.

This comment has been minimized.

Copy link
@ara4n

ara4n Dec 30, 2018

Author Member

I've spelt out that events when purged should be removed from both queues & DBs, and that downstream notifications should be rolled back where possible, in 4646fcd

Show resolved Hide resolved proposals/1763-configurable-retention-periods.md
Show resolved Hide resolved proposals/1763-configurable-retention-periods.md Outdated
Show resolved Hide resolved proposals/1763-configurable-retention-periods.md Outdated
Issues below)

TODO: do we want to pass these in as querystring params when sending, instead of
putting them inside event.content?

This comment has been minimized.

Copy link
@turt2live

turt2live Dec 31, 2018

Member

@mscbot concern todo_comments

TODO comments should be resolved before a FCP begins.

Show resolved Hide resolved proposals/1763-configurable-retention-periods.md
Show resolved Hide resolved proposals/1763-configurable-retention-periods.md
Show resolved Hide resolved proposals/1763-configurable-retention-periods.md Outdated
Show resolved Hide resolved proposals/1763-configurable-retention-periods.md Outdated
If `expire_on_clients` is true, then clients should also calculate expiration for
said events and delete them from their local stores as required.

## Pruning algorithm

This comment has been minimized.

Copy link
@turt2live

turt2live Dec 31, 2018

Member

It's unclear how this affects the sync stream of clients.

For example, given a device which happened to be not syncing prior to an event's arrival and subsequent purge, can the server optimize the event out of the sync stream for that device?

Another case would be a client which is back paginating a room to a now-expired event, and has back paginated that far in the past. The client may expect to see the event due to the tokens used.

Similar to the last case, a sync should be repeatable with a given token. How is one meant to handle a now-expired event appearing in an old sync for a device when a client uses that old token? This can happen in the case of clients crashing and losing some number of tokens, or bots losing track of where they are in the sync stream.

This comment has been minimized.

Copy link
@ara4n

ara4n Jan 4, 2019

Author Member

is this really a problem? i don't want to have to go through every API spelling out "purged events do not appear in the response". Yes, this means that calling the same /sync or /messages API twice may give different results if some of the events have disappeared in the interim, and yes, the server will be able to remove them from the response. And yes, the server will need to ensure that tokens still work despite events disappearing whilst syncing.

This comment has been minimized.

Copy link
@turt2live

turt2live Jan 4, 2019

Member

A mention that servers and clients should be aware that events do disappear from event streams is probably fine. When it comes to the actual spec, the module would probably have a one-liner saying that, and explain that tokens are not repeatable in these cases.

It may only be a problem for cases where large swathes of the room are lost to retention, making clients do hundreds of backfill requests before they see an event again.

This comment has been minimized.

Copy link
@ara4n

ara4n Jan 5, 2019

Author Member

this feels like something which would be added in the spec PR, tbh

Show resolved Hide resolved proposals/1763-configurable-retention-periods.md

@richvdh richvdh self-requested a review Jan 2, 2019

@anoadragon453 anoadragon453 removed the T-Core label Jan 4, 2019

ara4n added some commits Jan 4, 2019


`self_destruct`:
the duration in seconds after which servers (and optionally clients) must
remove this event after seeing an explicit read receipt delivered for it.

This comment has been minimized.

Copy link
@Kegsay

Kegsay Jan 31, 2019

Contributor

after seeing an explicit read receipt delivered for it

Does the clock start the moment the first read receipt is seen, or does it refresh so long as there is a new read receipt (the latter leading to a LRU-style cache)?

@ara4n

This comment has been minimized.

Copy link
Member Author

commented Feb 18, 2019

One thought: i wonder if we should do something to expire megolm key backup for sessions whose events have all been deleted from the server...

@richvdh richvdh removed their request for review Feb 27, 2019

whether they are knowingly participating in a future one-sided conversation or
not).

We would also like to discourage users from setting low message retention as a

This comment has been minimized.

Copy link
@Mikaela

Mikaela May 8, 2019

What is considered as a low message retention?

This comment has been minimized.

Copy link
@cyphar

cyphar May 9, 2019

I'm also not sure I see why short-lived messages are a bad thing. Maybe this is the case for public discussion-style rooms (where admins could set a large lower bound on message lifetimes) but for some private rooms it definitely can make sense to have short retention policies (a feature I quite like in Signal is that you can set your message scrollback to be capped to $x messages, so even if you get stopped at an airport you won't be giving up years of private conversations).

(and possibly letting users know how likely history is to be retained, or conversely
letting servers know if they need to step up to retain history). The disadvantage
is that it could make for very complex UX for end-users: "Warning, some servers in
this room have overridden history retention to conflict with your preferences" etc.

This comment has been minimized.

Copy link
@Mikaela

Mikaela May 8, 2019

In my opinion the room admin(s) should be respected here and if there is a warning that some server is conflicting with room admin desires, won't the room admin just start ACL banning?


This proposal does not try to solve the problems of:
* GDPR erasure (as this involves retrospectively changing the lifetime of
messages)

This comment has been minimized.

Copy link
@cyphar

cyphar May 8, 2019

Doesn't GDPR erasure also require somewhat stronger security properties than best-effort erasure (like the mxid encryption proposals)?

@gerg5c42g542g2c54g52c

This comment has been minimized.

Copy link

commented May 14, 2019

One thought: i wonder if we should do something to expire megolm key backup for sessions whose events have all been deleted from the server...

Yes, we should and we should delete the corresponding keys.

It's debatable as to whether we're better off applying the redaction algorithm to expired events (and thus keep the integrity of the DAG intact, at the expense of leaking metadata), or whether to purge instead (as per the current proposal), which will punch holes in the DAG and potentially break the ability to backpaginate the room.

We should either have both options available in clients or only purge. I think keeping the integrity of the DAG in this case is only useful for non-serious use cases like sending a message I love you with 1 second expiration mark and all the messages before and after it with a keep forever mark.
If one uses this to actually cleanse stuff for privacy reasons, space freeing or to keep things clean, it won't matter, because all history will be accessible anyway - if retention is 1 month, everything older than that will be deleted anyway.

Should room retention be announced in a room per-server? The advantage is full flexibility in terms of servers announcing their different policies for a room (and possibly letting users know how likely history is to be retained, or conversely letting servers know if they need to step up to retain history). The disadvantage is that it could make for very complex UX for end-users: "Warning, some servers in this room have overridden history retention to conflict with your preferences" etc.

I don't think it should be - it should be an agreement between the participants of the room. But there could a server-wide policy of maximum retention to save space, in this case a participant cannot set it higher than the lowest maximum retention limit of the participating servers.

There's scope for abuse where users can send abusive messages into a room with a short max_lifetime and/or self_destruct set true which promptly self-destruct.
One solution for this could be for server implementations to implement a quarantine mode which initially marks purged events as quarantined for N days before deleting them entirely, allowing server admins to address abuse concerns.

A quarantine isn't a good solution to this. Just give room admins an option to prevent messages with short max_lifetime.

@Half-Shot Half-Shot referenced this pull request May 14, 2019

Closed

expire messages #5188

@aaronraimist aaronraimist referenced this pull request May 16, 2019

Closed

room destruction #5189

@moh10ly

This comment has been minimized.

Copy link

commented May 20, 2019

Is it also possible to know how are we supposed to delete the data. assuming we have huge amount of files, data etc in a room how do we delete it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.