RE: retention policy for cached content and media

[hugogameiro]

The recent PR #19232 adding a retention policy for cached content and media can significantly reduce the resources used by Mastodon. In particular, the storage needs. This is something that admins will benefit from and reduce the network's carbon footprint. So, I am happy that it has been introduced.

Still, the more I think about it, the more I believe that deleting all remote content that is older than X days from the database (set in 'Content cache retentions period') will cause unintentional consequences for the end user, as I mentioned on #19232 (comment).

If I bookmarked a post one year ago. I expect it to be in my bookmarks one year later, even if the server that originated the post is no longer online and the content cache retention period set by the admin is 6 months. The same goes for boosts, favourites, DMs or posts I replied to. If we use the date as the single factor for deletion, we break functionality that I believe is important.

Now, I understand that only deleting posts that are older than a date if they were not interacted with like it's the case of tootctl status remove can be "computationally heavy procedure that creates extra database indices". But we could explore ways to make this content filter of interacted posts less demanding.

One idea that comes to mind is to add a counter for each post. Every time a local user bookmarks, favourites, boosts, is mentioned on or replies to a post, the counter on that post increases by one. If a local user removes from bookmarks, favourites, boosts, or delete their reply, it decreases by one. This way, the cache deletion would filter posts older than X date and where the number on that counter would be zero.

That same counter could also be used for the 'Media cache retention period' for post attachment media. As some cannot be redownloaded if the server goes offline and in some cases the post without the media makes little sense or is just blank.

There could be something I am missing, or this could be way more complex than I think. But I believe some more thinking could be done before merging this PR to the stable release.

[lazyatom]

This feels like a very pragmatic suggestion -- essentially "reference counting" the remote content.

[pisscotheque]

I would be very keen to see something like this incorporated into the change as well. We do desperately need a way to purge old statuses more easily but I would be very hesitant to use it if it didn't discriminate between content that local users had interacted with and that which they hadn't. This would make it a much more useful tool.

[nemobis]

Also worth noting that on many instances the storage utilisation is dominated by profiles' media rather than posts. #9567

[chromaniac2]

I got hit by this on a recently launched instance by Vivaldi Browser team. My bookmarks pointing to older posts were disappearing. They managed to locate the issue to this setting and switched it to unlimited. Hopefully, some resolution can be found to keep interactions while still saving resources for instance owners!

[ruario]

Thanks for the report by the way @Sushubh

[cham2019]

One idea that comes to mind is to add a counter for each post. Every time a local user bookmarks, favourites, boosts, is mentioned on or replies to a post, the counter on that post increases by one. If a local user removes from bookmarks, favourites, boosts, or delete their reply, it decreases by one. This way, the cache deletion would filter posts older than X date and where the number on that counter would be zero.

This is an great suggestion to make this retention policy feature more useful. Hope it gets implemented.

[saschafoerster]

I agree fully. After understanding the consequences of the retention policy, I set it to unlimited, as I would expect my bookmarks, etc. to be stable and existing, even after years. I would also like to ask to question, what about accounts that are in "monument mode" for instance, when someone died? I personally think till now, that I would like to keep the content of that account as well.

[mackuba]

One idea that comes to mind is to add a counter for each post. Every time a local user bookmarks, favourites, boosts, is mentioned on or replies to a post, the counter on that post increases by one. If a local user removes from bookmarks, favourites, boosts, or delete their reply, it decreases by one. This way, the cache deletion would filter posts older than X date and where the number on that counter would be zero.

I think this could even be much simpler - probably in >99% of cases, once a user bookmarks, favorites, boosts etc. a status, they will not take that back later. So there could be just one flag "this status was favorited/bookmarked etc." that is just set once and never reset. If a post was favorited or bookmarked once, it almost certainly still is, so we can just as well keep that post without re-checking that. The additional storage savings we could get from trying to find all those statuses that were later removed from all bookmarks etc. would be minimal, for a lot more work.

[nondeterminischtick]

Will this action lose replies from external servers as well? So it will look like 2 years ago I was just talking to myself 😄... Or will they be retained?

I can live with disappearing boosts, favourites, and bookmarks, but would hope the context tree of my posts would remain.

[nehemiagurl]

if I understand correctly, replies are lost as well. anything that isn't in-house is gone, unless it is for some reason refetched.

[nemobis]

Yes. Replies cannot be re-fetched either, because references to them are removed as well.

[rollecode]

Just realized this from here.

This value is blank by default, so it never deletes the remote content cache.

The current Mastodon implementation makes this a dangerous setting and will indiscriminately delete remote content older than the number of days set, whether the content was interacted with or not. Meaning, that no matter if a local user bookmarked, favourited, or boosted a remote post or even if a post is a remote reply to a local post, all will be deleted once the number of days has been reached, which is irreversible.

You can read a discussion I started in the Mastodon GitHub about the consequences of enabling this setting to see if the Mastodon dev team finds an alternative implementation.
Until Mastodon releases an improvement for Content cache retention period, I suggest leaving this value blank or, if you want/need to set it, make it something like 730 days (2 years) or higher but any value set will still cause data loss.

[VladlenO]

I just lost everything by enabling this feature. They are talking about cache, but in fact this is deleting data without the ability to update the cache. Fucking drug addicts who wrote about cache.

[cooljeanius]

I've been told that an issue I noticed with posts in threads getting disassociated from one another might actually be due to this policy: https://social.treehouse.systems/@egallager/111898518131801953

[nehemiagurl]

This really needs to be solved at some point. As an instance admin I get to a point that I either need to ask people for more money just to host old external toots, ~99% of which nobody cares for by this point, or I have to enable the really darconian feature, fully knowing that I delete people's old bookmarks in the process.
Just adding a simple tick once the post has been interacted with and never changing it again would be so helpful in ways I cannot explain. I will turn this feature on in a heartbeat had this filter existed.

[arinoch]

I had this feature on until I came across this discussion. Fortunately the policy was long enough that it wasn't affecting anything and my instance is new enough that there's not much to affect, but yes absolutely please build in a check for at least 1 bookmark on the local instance before it gets nuked. I'm not so picky about boosts, but if you want people to bookmark things, those things should still be there. I turned this feature off in a hot minute after reading this, and will turn it back on just as quickly if it gets fixed.

[colinstu12]

I agree as well. This is starting to bite us more and more as previous stuff gets wiped out. I personally would prefer it be any interaction and not just bookmarks (favs, boosts, been replied to).

[Mai-Lapyst]

It should also be mentioned that it needs to be desiced if only interacted-with posts should remain, or all posts in that thread, since some user (myself included) may only interact / bookmark the top-most toot(s) but is actually interested into the complete discussion / thread to read it later on or similar.

[nehemiagurl]

this will be much harder to implement than a simple "has this been interacted with" and as such seems unlikely to be implemented.

[Mai-Lapyst]

I dont think it is so much harder. It should be considered, since otherwise people would write bots or automated systems to auto-interact with large threads just to get these toots to stay onto the server. Ofc this can still be done without this addition, but it would certainly reduce the need for such bots / tools and thus the users of these.

And sure, for a big instance with a very big thread this certainly can eat a lot of resources up qickly, but theres always the single-user and tiny instances that just want to save their favorite messages & threads of their favourite humans. It could be made an setting so a instance admin can control how many posts should be saved implicitly.

[colinstu12]

I'd say any single interaction within any thread should also flag the rest of the thread as well to be retained.
Or at the very least let this be configurable.
I hate how old threads are breaking due to this, everything in the thread would be ideally retained - even if only like the first or last or middle post was interacted with.

[Shanesan]

I have created a feature request specifically for this issue: #30449

[arinoch]

Is there any progress on this issue? I'd prefer not to have to inevitably upgrade the server my instance runs on simply because I ran out of space.

[mackuba]

If I understand it correctly, it was implemented here: #27733, but it wasn't released in any stable version yet, scheduled for the 4.3 release I guess, whenever that comes.

[mackuba]

Ah, no, sorry, I think that's something a bit different

[edmael]

Any progress on this issue?
Can we implement something "easier" while the discussion keeps going on?
Something like: "if a toot has been re-tootet/interacted/bookmarked do not remove it during the cache cleaning".

[edmael]

I really don't get - and I'm being literal here, not criticizing or anything - why a simple implementation based on a little check is not being implemented.

The Fediverse is growing and if we don't address issues with this right now that we're expanding there's the serious risk of shooting into our own feet and block this growth.

[nemobis]

I agree with you all but just checking, have you already tried https://ricard.dev/improving-mastodons-disk-usage/ ? For example on my instance I have a daily cronjob

#!/bin/bash
# Copied from https://ricard.dev/improving-mastodons-disk-usage/

# Prune remote accounts that never interacted with a local user
RAILS_ENV=production /home/mastodon/.rbenv/shims/ruby /home/mastodon/live/bin/tootctl accounts prune;

# Remove remote statuses that local users never interacted with older than x days
RAILS_ENV=production /home/mastodon/.rbenv/shims/ruby /home/mastodon/live/bin/tootctl statuses remove;

# Remove media attachments older than x days
RAILS_ENV=production /home/mastodon/.rbenv/shims/ruby /home/mastodon/live/bin/tootctl media remove;

# Remove all headers (including people I follow)
RAILS_ENV=production /home/mastodon/.rbenv/shims/ruby /home/mastodon/live/bin/tootctl media remove --remove-headers --include-follows;

# Remove link previews older than x days
RAILS_ENV=production /home/mastodon/.rbenv/shims/ruby /home/mastodon/live/bin/tootctl preview_cards remove;

# Remove files not linked to any post
RAILS_ENV=production /home/mastodon/.rbenv/shims/ruby /home/mastodon/live/bin/tootctl media remove-orphans;

[pisscotheque]

This kind of solution is good for people who fully run their own instances, but admins like myself who use a hosting service can't do that. What I'm hoping for is a set of options that's exposed in the admin interface.

[Shanesan]

Ricard's solution is a little heavy handed as deleting media like that will delete it forever, and won't be recoverable (from my historical use of it). It's why I created a suggestion to add a bit more customization to the /admin/settings/content_retention page.

[jeroenhabets]

Agree, I also looked into the scripts and API but then found out the only real solution is implementing the issue you suggested: #30449

[ruario]

I had to flip the setting, knowing the major downsides because for me too, it would become too expensive otherwise. Now, due to the downsides of having done so, I am seriously considering giving up on my vanity (single person) instance and just switching to another instance with more people where I donate to those that maintain it. It's a real shame but currently as things stand, this is unworkable for me. If I retain everything it is too expensive, if I don't I get a sub standard experience.

[edmael]

Hey, have you tried this?
https://ricard.dev/improving-mastodons-disk-usage/

[jeroenhabets]

Yes, but then found out the only real solution is implementing this issue: #30449

[nehemiagurl]

i recently noticed that the "automatic post deletion" tab on Mastodon includes this little section:

this seems like the exact thing requested in this discussion. i'm not a developer, but would it be this hard to import this to the instance level as well?

[ludovicdeluna]

As a user (not an administrator), I've set this up for all my accounts. From my point of view, this has nothing to do with disk usage, as I don't know how Mastodon works internally.

This concerns the content I publish online. I prefer to choose what's really important, and the rest can be forgotten after a while.

This Github thread is more about disk usage. I don't know exactly what is requested by EU laws, but from my little understanding: delete the copies of my toots on other instances after a while. It could be for regulation purpose, or disk space.

For my very limited understanding, I don't care.

Go head, remove them. But if users from those instance bookmark them, please they should be kept in this case.

If the only interaction is just boosts or stars, yeah, burn them out after a while. This is not "bookmarks" for me, just a way to congrats the author.

What I'm not comfortable with, it is: where live the "true" Mastodon thread? In the emitter's instance? This is the only one which could survive to any disk saving. Keep the complete thread interaction safe from the original emitter (the user who created the thread). Rest is just copies.

The cache, for me, is just a simple "copy". Only the original matters. The cache is simply a way to access information more easily / instantly. If the cache can be rebuilt from the original, I'm fine. If the original is deleted, that's it. Mastodon is not a replacement for archive.org.

Sorry for the noises. I love Mastodon, but it is a magic box.

[henshaw]

Adding to the request to please implement something like this #30449 because it's getting more and more expensive to run a single-person instance.

[jeroenhabets]

[Updated: fix PR link]
For any Mastodon admin following or finding this discussion (or issue #30449) desperate to reclaim some disk space: @northerner 's PR #36200 worked wonders for me 🚀 🙏 .

Reclaimed 300GB disk space without mangling my favourites, boosts, and posts after running in a tmux session on a few years old Mastodon server:

export RAILS_ENV=production && ~/live/bin/tootctl media remove --keep-interacted

The initial run took 20 hours, while Mastodon remained fully usable. Fortunately subsequent runs finish within seconds. And @ClearlyClaire 's query suggestion might dramatically improve the initial run.

[ludovicchabant]

+1 to what @henshaw said. Mastodon is just not sustainable right now for small servers. I really hope someone can merge that PR or do something about it 🤞

[henshaw]

Yeah, it keeps getting more and more expensive as time goes on. For me, it's less about the media storage and more about database size that keeps getting exponentially larger. I'm committed and willing to pay for it with who I'm hosting it with, but as @ludovicchabant said, it's not sustainable for most people wanting a single-user instance. I assume that solving this would also benefit community servers too.

[jeroenhabets]

@northerner's PR #36200 has already been merged by @ClearlyClaire! After seeing your comments @ludovicchabant , @henshaw I noticed my comment accidentally had this issue link #30449 also for the PR, instead of PR #36200.

P.S. If you cannot wait until the next release, you could use the, now even improved, code of PR #36200 like I did.

[henshaw]

@jeroenhabets that's exciting! I'm using managed hosting via @hugogameiro, so I'll have to wait until the stable release, but I'm looking forward to the update.

[hugogameiro]

@henshaw The PR #36200 is for media storage. It's a great improvement, but it won't affect database usage.

[northerner]

@ClearlyClaire Could the existing counts in the StatusStat model be used to implement this data pruning feature?

I noticed the count differentiates between favourites_count and untrusted_favourites_count so we know if the favourites were from local instance users or remote. However, quotes and replies have a single count, does that include quotes and replies from remote instances? If that's the case we can't differentiate and use this to filter which statuses can be safely deleted.

https://github.com/mastodon/mastodon/blob/main/app/models/status_stat.rb