Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Message limit/message expiration - message retention policy #795

Closed
fjorgemota opened this issue Sep 13, 2015 · 30 comments
Closed

Message limit/message expiration - message retention policy #795

fjorgemota opened this issue Sep 13, 2015 · 30 comments

Comments

@fjorgemota
Copy link

Hello!

I'm researching chat tools that are opensource and Rocket.Chat seems like a great possibility.

However, one thing that I am searching is someway to limit the number of messages posted in the service and/or expiring old messages. In the RocketChat's documentation, I found in Schema Definition the field 'expireAt': https://github.com/RocketChat/Rocket.Chat/wiki/Schema-Definition, but, when installing the version 0.5.0 here, it seems to not create that, and there's no more references to message expiration in RocketChat's platform.

Is there a hidden setting to expire messages? Or it's just a plan for now? And, yet, it's there a plan for message limit too? I didn't found anything for that in RocketChat documentation.

Thanks. And sorry for my bad english. ✌️

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

@geekgonecrazy
Copy link
Contributor

Curious what is the application for this? This is one of the reasons why a lot of people move away from a lot of other services..

@fjorgemota
Copy link
Author

Just to avoid excessive use of storage space. =)

I have a "team" of colleagues on the university where I study and we discuss a lot about the various disciplines of our course and about a lot of other nerd topics and these type of things.

Up to now, we're using Hipchat's service, but, as there a problems that cause downtime in the last days/months, we are searching to use other services. We thought about migrating to Slack, but we really don't want to depend too much about third party services that can have uptime problems as Hipchat is having recently, just because we generally talk in periods that do not include normal work times (like weekends, for example). Whatsapp isn't a possibilty because not everyone in our group have a smartphone (as we're just students), and Telegram, well, can suffer from downtime as well as we can't install a server for it.

So, that's my use case. Seems like a joke, but it's not, and for our study the use of a chat service improves resolutions of problems, allow us to study together to exams and many other things. (we have a group at facebook too, but it's virtually abandoned after we started using a chat service like Hipchat)

As MongoDB have support to expiring documents automatically, and RocketChat seems to have a field just for that - at least in the documentation, I thought that RocketChat have that support, and I don't found any other issues about that. (which seems to confirm your comment about this being a reason why a lot of people move away from a lot of other services)

Thanks.

What you think about?

PS: I thought about capped collections. But, I don't know about the possible behavior of Meteor/RocketChat with it.

PS 2: Note that, because we're students, we really don't need all the messages. At maximum the messages of a semester or two is sufficient for us. (and it probably will be cheaper to host, too)

@geekgonecrazy
Copy link
Contributor

Space wise your biggest limitation is going to be file uploads. I've hosted forums and stuff in the past that had many thousands of lines of text and was only a few megs. Infact I think I saw a slack export of 10k messages that was under 1Mb.

That aside. I don't think its necessarily a bad idea. What about a tool that could prune messages older then a certain date? Lets you pick the date and prunes everything older then that?

Expiring is obviously an option. Just seems a bit scary to give that much power to the database. Also we would have to switch that functionality on and off for the db when the user changed. Idk how much of an issue that would or wouldn't be.

@fjorgemota
Copy link
Author

Yep, maybe the messages will not ocuppy so much storage, my main concern is about some texts and longer codes that we post on the chat, that have a bit of frequency..I don't have a idea, however, of how much storage space that occupies

A tool to prune messages is a good option too. I think initially about 'expiring' or 'limiting', but I think that a tool with that utility is good too. (specially if it delete the attachments of the messages too)

Nice idea. =)

@fjorgemota
Copy link
Author

Well.

I exported the messages of the group and we that generated a folder with 21MB of chat (in JSON format). Considering that Hipchat's probably cropped that export at the limit of 25k messages, and our biggest room have 81k messages, well...It's a bit. :P (and that without considering attachments..)

@geekgonecrazy
Copy link
Contributor

@fjorgemota so it could be big :) I can see where small teams after a while would need some way of pruning down their disk footprint.

@Sing-Li
Copy link
Member

Sing-Li commented Sep 13, 2015

👍 Great idea! It will make Rocket.Chat adaptable to many more installation scenarios (fixed sized SD card on a Raspberry Pi 2 - for example).

In the longer term, with two classes of users in mind:

  • those who want to host Rocket.Chat for the masses
  • enterprise and corporate users faced with compliance and audit

Message retention and per-user-quota will be completely driven by policies. Policies will require a 'roles and permissions' system being in place (there is a ticket for that somewhere). Basically, a policy may say - users in a student role will have only 5 MB storage quota, and their message retention is 3 weeks. The system will then enforce the policy.

Meanwhile, any implementation of pruning or per-user capping of storage compatible with the above will add value immediately.

@rodrigok
Copy link
Member

O liked the idea, we can do this in 2 steps

  1. General quantity/size limit or date expiration (we can do this direct in MongoDB)
  2. Limits for quantity or time based on user's role (depends on role implementation and needs to be implemented in the application layer) and we can't limit by size.

The first step is relatively easy to do.

@AdrianoCahete
Copy link
Contributor

enterprise and corporate users faced with compliance and audit

For audit, we need that ALL the operations are logged. From "giving permissions" to "leave channel".

@Sing-Li
Copy link
Member

Sing-Li commented Sep 14, 2015

😀 @AdrianoCahete I'd say you're about a year ahead of your time.

@AdrianoCahete
Copy link
Contributor

@Sing-Li Just habit.... I work in company who developing a system for Governance, Risks and Compliance. :P

@engelgabriel engelgabriel added this to the 0.20.0 milestone Feb 22, 2016
@engelgabriel engelgabriel modified the milestones: Next, 0.20.0 Feb 22, 2016
@Sing-Li Sing-Li changed the title Support to message limit/message expiration Message limit/message expiration - message retention policy Mar 22, 2016
@ma-karai
Copy link

Maybe Indicate the History the following way.

capture

In this case every message before Saturday, November 8, 2014 will be deleted

@dandv
Copy link

dandv commented Mar 30, 2016

Perhaps a more stringent use case than space limitations, is message retention policy from legal departments. In some large companies, chat messages must not be stored beyond X days, to limit the risk of exposure in case of subpoenas/legal discovery processes.

@freddielam
Copy link

O liked the idea, we can do this in 2 steps

  1. General quantity/size limit or date expiration (we can do this direct in MongoDB)

was looking for how to do this,

so while waiting for their feature to be done here's how it can be done.

from mongo shell

use rocketchat
db.rocketchat_message.createIndex( { "ts": 1 }, { expireAfterSeconds: 3600 } )

this will expire any message after 1 hour, even existing messages.

@geekgonecrazy
Copy link
Contributor

@freddielam this is for sure an easy way of accomplishing this. 😄

@ghost
Copy link

ghost commented Jun 7, 2016

@freddielam is it possible to only expire the oembed messages?

@timmc
Copy link

timmc commented Mar 12, 2017

Additional use-case, along the same lines as the legal/corporate one: Activist groups or people talking about sensitive issues often want auto-expiring messages/retention policies.

@freddielam
Copy link

freddielam commented Mar 12, 2017 via email

@flantascience
Copy link

I'd be interested in having an expiration date on file uploads as well. Maybe messages too... but definitely file uploads. Is anything brewing with this possibility?

@localguru
Copy link
Contributor

Is there anything going on in this direction?

@geekgonecrazy is the above workaround on database level "secure" or could that cause inconsistency, e.g. files attached to a post left in database?

use rocketchat
db.rocketchat_message.createIndex( { "ts": 1 }, { expireAfterSeconds: 3600 } )

@geekgonecrazy
Copy link
Contributor

@localguru I think it'd leave orphaned files :( if using gridfs you could likely add the index there as well. If you are using a object store on s3 / file system or others.. It'd leave the file there.

@maxpen
Copy link

maxpen commented Oct 10, 2017

Yes again! lol

@hartois
Copy link

hartois commented Nov 13, 2017

I'm planning on using a Rocket.Chat in our company. And I'm also very interested in the space limitation for the user. I'm afraid that users in a short time using attachments will quickly deplete the available space.
Is there any news on this issue?

@marzie82
Copy link

Are there any news regarding space limitation? We would be very interested in such function! Of course it's great if you don't need to delete old Messages but after some years it's a different situation. Thnanks!!!

@docsncode
Copy link

Closing as this feature is not on our short term roadmap, we will revisit once other priorities have been addressed.

@engelgabriel
Copy link
Member

Done via #11236

@theorenck theorenck removed this from the Mid-term milestone Dec 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests