-
-
Notifications
You must be signed in to change notification settings - Fork 7.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add command to remove avatar and header images of inactive remote accounts from the local database #22149
Conversation
Thank you @ykzts ! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please note that both the linting errors
- Line 59 in lib/mastodon/media_cli.rb
- Line 324 in lib/mastodon/media_cli.rb
are unrelated to this commit.
Thank you @ineffyble and @ykzts for triggering the tests! I would really appreciate it if you could trigger them again since I corrected the linting errors. Please note that both the linting errors were unrelated to this commit, but I corrected them hoping it would make the merge faster. |
@connorshea I would really appreciate it if you could take a look at this pull request! I believe it would be a quick review for you since it is very similar to the other pull request I had made. I hope I am not being pushy. 🕯️🕯️🕯️🕯️ |
Should this perhaps be a part of |
@ClearlyClaire Thank you for your suggestions! I have incorporated them. @Gargron I kept the command separate initially just to keep the changes modular, but now I have made it part of |
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
@ClearlyClaire Thank you so much for taking the time to review the code! I would really appreciate it if you could trigger the CircleCI Checks. I am logged in to CircleCI and it has access to my fork, but last time it had to be triggered by @ineffyble. |
Hi, I can't see any way to trigger it on my end, but given that we don't have tests for |
Hi Claire, thank you very much for your follow-up! Watching is much quicker than reading. Pls see my toot below :) What I've done is as follows. in "mastodon/app/controllers/api/v1/accounts_controller.rb"
That's it! This came based on what you kindly told me, again, thank you very much, Claire! |
I cannot see the time to access my laptop and incorporate this into my instance. I'm the only user and because of avatars and headers my cached media partition is growing too fast. In about one month it reached its 20 Gb limit, even deleting the other cached media daily. Thank you @evanphilip! |
Is that then also executed through the retention policy for cached content and media? |
…ounts from the local database (mastodon#22149) * Add tootctl subcommand media remove-profile-media * Trigger workflows * Correcting external linting * External linting error * External linting fix * Merging with remove command * Linting * Correct long option names Co-authored-by: Claire <claire.github-309c@sitedethib.com> * Correct long option names Co-authored-by: Claire <claire.github-309c@sitedethib.com> * Correct long option names Co-authored-by: Claire <claire.github-309c@sitedethib.com> * Remove saving a list of purged accounts Co-authored-by: Claire <claire.github-309c@sitedethib.com>
#1259) This cherry-picks [this pull request commit](mastodon#22149) into Hometown. It will be coming in a future Mastodon release but we will get it early. Basically it adds options to `tootctl media remove`: > Removes locally cached copies of media attachments (and optionally profile headers and avatars) from other servers. By default, only media attachements are removed. The --days option specifies how old media attachments have to be before they are removed. In case of avatars and headers, it specifies how old the last webfinger request and update to the user has to be before they are pruned. It defaults to 7 days. If --prune-profiles is specified, only avatars and headers are removed. If --remove-headers is specified, only headers are removed. If --include-follows is specified along with --prune-profiles or --remove-headers, all non-local profiles will be pruned irrespective of follow status. By default, only accounts that are not followed by or following anyone locally are pruned. Relates to but does not fully address #1209 because there needs to be a web UI component, too. Co-authored-by: Evan <35814742+evanphilip@users.noreply.github.com> Co-authored-by: Claire <claire.github-309c@sitedethib.com>
No. |
…ounts from the local database (mastodon#22149) * Add tootctl subcommand media remove-profile-media * Trigger workflows * Correcting external linting * External linting error * External linting fix * Merging with remove command * Linting * Correct long option names Co-authored-by: Claire <claire.github-309c@sitedethib.com> * Correct long option names Co-authored-by: Claire <claire.github-309c@sitedethib.com> * Correct long option names Co-authored-by: Claire <claire.github-309c@sitedethib.com> * Remove saving a list of purged accounts Co-authored-by: Claire <claire.github-309c@sitedethib.com>
Any brave soul willing to update the documentation to include these new params? https://github.com/mastodon/documentation/blob/master/content/en/admin/tootctl.md |
@quicoto I assumed the CLI documentation was automatically generated from the docstring. Should have checked. I won't have access to a full-size computer this week, so it would be great is someone could add this. |
I can't seem to get Could not find command "remove_headers". |
@OccultWarlock I noticed the same and then realized it's only been merged into main, but not into any of the latest releases. Here's hoping that will be soon. edit: Ah yes, it's in the coming 4.1.0 release |
If it's any help I installed v4.1.0rc2 and ran the following commands successfully: |
@evanphilip PR with the Docs updated has been created, someone needs to review it Thank you |
i hate to raise this concern two months after it has been merged (and after it has been included in a tagged release), but looking at the way this command is called and the functionality it provides, it seems incredibly confusing. there is no clean way to specify what gets removed. it essentially has three completely different behaviors depending on which flag is provided, and the flags are mutually exclusive. additionally, using certain flags will change the meaning of other flags. i'm not sure how best to fix this, but i would propose at minimum reworking the flags so that they are inclusive rather than exclusive, and so that you can use multiple flags together:
additionally, i would consider allowing |
@trwnh I am not very happy with my implementation and I agree that it is confusing. I would be glad to rework the flags myself if we can reach some consensus. Maybe we should open an issue? I like your suggestion a lot. There would not be a way to leave media attachments untouched, but I don’t think that would be a problem. |
I can confirm, that it was confusing. I was looking forward in winning back a lot of space, but I had to look several times into documentation to fully understand the scope of the different flags. After using all of the commands, I only saved about 9GBs of about 200GBs used. I hoped that I can reduce much more. Was my expectation wrong? I hoped that the 30GB of remote avatars and 60GB of remote headers would be reduced significantly. |
@saschafoerster Could you run the following as a test?
This will ONLY clear up avatars and headers, but it should clear up practically everything (since it is a dry run, it won’t actually do anything but will show what it will do). If this doesn’t clear up almost everything, something is amiss. |
@evanphilip i opened #23628 |
@evanphilip I tried this (I am using a docker installation):
I tried it then by removing the --verbose part:
But then I am a bit afraid, if I am deleting things that are not reloaded on request from external servers. :) |
@saschafoerster Thank you for your feedback! I would certainly not recommend setting
|
I opened #24070 for the |
I did not intend to remove |
This implements a new sub-commands for
tootctl media
calledremove-profile-media
to remove avatar and header images of remote accounts that appear inactive from the local database.Fixes #9567 : absence of method to remove avatar and header images, leading to excessive disk usage.
This PR is just a slight modification of #21066 by @dunkelstern, which was withdrawn by the author since
last_webfingered_at
was not a satisfactory way to determine old accounts. While imperfect, the existingtootctl accounts cull
usesupdated_at
andlast_webfingered_at
. Maybe it is the best one can do?I have also added an option which lets one keep avatars.
All credit to @dunkelstern , I am just desperate to clear my storage and it looks like there are others like me who could use this too!