Sidekiq stops processing tasks, queue grows to huge size

[joenepraat]

As told to @Gargron yesterday, Sidekiq stopped processing tasks and the queue kept growing to ~40.000. Restarting Sidekiq restarted processing (and temporary increasing the # of Sidekiq threads cleared the queue).

This morning I encountered the same issue. The # of processed tasks didn't grow anymore and the queue was at ~20.000 tasks and growing. I found these errors in the systemd journal 8 times between 01:16 and 03:08. This is the last one:

Nov 29 03:08:31 todon.nl bundle[6377]: /home/mastodon/live/vendor/bundle/ruby/2.5.0/gems/http-3.3.0/lib/http/connection.rb:129: warning: HTTP::Timeout::Null#closed? at /home/mastodon/.rbenv/versions/2.5.3/lib/ruby/2.5.0/forwardable.rb:157 forwarding to private method NilClass#closed?

Restarting Sidekiq restarted processing tasks.

Specifications

Mastodon 2.6.2
Sidekiq 5.2.2
Debian 9.6

[joenepraat]

P.S. This started after updating to Mastodon 2.6.2 (from 2.6.1)

[joenepraat]

I think the error message (what is more a warning when I look better) I added here is not related. I had these messages today already a few times without impact.

Please tell me how where to look for?

[Gargron]

I have asked you to check if some Sidekiq jobs are taking unnaturally long, that would shed light on the cause of the issue. On the Sidekiq "busy" tab, there is a listing of all jobs that are currently being processed, with a column that displays when the job started processing. Usually they should all say "just now" or "20 seconds ago". You will notice a problem if some of them say "20 minutes ago" or "21 hours ago".

Lacking this information, I can guess that this might be related to c39d7e7, which I intend to release in 2.6.3 asap, but it would be nice to get confirmation.

[joenepraat]

I checked that column, but they were all recent or a certain amount of seconds ago.

Let me see how things work after upgrading to 2.6.3.

[ClearlyClaire]

I've seen it happen on a fellow admin's instance, with ThreadResolveWorker, ProcessingWorker and DeliveryWorker.
I suspect it's because of #9329 and would thus be fixed by #9381. I instructed them to cherry-pick the latter, we will see if that works out.

[kit-ty-kate]

cherry-picking those two PRs worked for me so far. It unlocked the froze processes

[Gargron]

I released 2.6.3

[joenepraat]

Thanks everyone!