Change tootctl to use inline parallelization instead of Sidekiq #11776
Conversation
Gargron
force-pushed
the
fix-parallelize-tootctl-media-remove
branch
5 times, most recently
from
d4aea03
to
1d8940c
Compare
Gargron
changed the title
tootctl media remove to use inline parallelization
Gargron
force-pushed
the
fix-parallelize-tootctl-media-remove
branch
from
1d8940c
to
4178725
Compare
|
Won't removing the workers cause issues if someone updates their instance while such tasks are queued? |
Since the workers are only used by the tootctl commands such timing is unlikely. I would prefer not to keep dead code around! |
lib/mastodon/cli_helper.rb
Outdated
| scope.reorder(nil).find_each do |item| | ||
| progress.total = futures.size + 1 if progress.total < futures.size + 1 | ||
|
|
||
| futures << Concurrent::Future.execute(executor: pool) do |
There was a problem hiding this comment.
From what I understand, even though find_each work in batches, this code will go through all the items in the query and put them in the queue, which might not be what we want, wrt. RAM usage.
Would it be possible to reap finished jobs as soon as possible and block whenever the pool is busy?
There was a problem hiding this comment.
The futures start processing as soon as they are created, the map over values at the end is just to block execution until they all finish. What is stored are the futures and the return values, which are numbers in our case, not the ActiveRecord records, so it should help with memory usage.
There was a problem hiding this comment.
I'm still worried about it being too much if the tasks are slow. I guess it could be interesting to see how it runs for pruning old remote media on m.s
Gargron
force-pushed
the
fix-parallelize-tootctl-media-remove
branch
2 times, most recently
from
0d488ed
to
1fe542e
Compare
Gargron
force-pushed
the
fix-parallelize-tootctl-media-remove
branch
from
1fe542e
to
d78dc49
Compare
- Remove --background option - Add --concurrency(=5) option - Add progress bars
Gargron
force-pushed
the
fix-parallelize-tootctl-media-remove
branch
from
d78dc49
to
c87c6d5
Compare
…odon#11776) - Remove --background option - Add --concurrency(=5) option - Add progress bars
…allelization instead of Sidekiq - Remove --background option - Add --concurrency(=5) option - Add progress bars
.-c/--concurrencyflag--backgroundflagCommands affected by change (unless specified otherwise, progress bar and parallelization):
tootctl accounts rotate(not parallelized)tootctl accounts culltootctl accounts refreshtootctl accounts followtootctl accounts unfollowtootctl accounts reset-relationships(not parallelized)tootctl cache recounttootctl domains purgetootctl domains crawltootctl feeds buildtootctl media removetootctl preview_cards removeExamples:
tootctl media remove --days=60 -c 20will remove media older than 60 days, 20 at a time