Extreme high push queue, because of remote account status deletion

[joenepraat]

Since last night I have on Todon.nl periodically very high push queues. Last night even more than 400.000 jobs where in the queue. I successfully increased the number of jobs from 400 to 1000, and this brought the number eventually back to 0, but later today I had 2 à 3 peaks of ~50.000. The result is that outbound federation is periodically delayed.

After investigating the queue I found that almost all jobs are deletion of toots from one particular remote user. I find this strange, cause I expect remote requests for status deletions in the pull queue (inbound federation).

Expected behaviour

Remote status deletions should use the pull queue, but more importantly; it should not intervene in the operation of another servers.

Actual behaviour

Remote status deletions use the push queue and let that queue grow very high. This is because every job sends a deletion to another remote user.

Here is an anonymized example output taken from the push queue in Sidekiq:

"{\"@context\":[\"https://www.w3.org/ns/activitystreams\",{\"ostatus\":\"http://ostatus.org#\",\"atomUri\":\"ostatus:atomUri\"}],\"id\":\"https://remote-instance.tld/users/Example-user/statuses/101674154054826596#delete\",\"type\":\"Delete\",\"actor\":\"https://remote-instance.tld/users/Example-user\",\"to\":[\"https://www.w3.org/ns/activitystreams#Public\"],\"object\":{\"id\":\"https://remote-instance.tld/users/Example-user/statuses/101674154054826596\",\"type\":\"Tombstone\",\"atomUri\":\"https://remote-instance.tld/users/Example-user/statuses/101674154054826596\"},\"signature\":{\"type\":\"RsaSignature2017\",\"creator\":\"https://remote-instance.tld/users/Example-user#main-key\",\"created\":\"2019-06-08T19:01:49Z\",\"signatureValue\":\"CjxFBxeIocfPGVU/OvSl1hJirAEbEFSX0nX/SeHWTNQrauX+cB9b02tL+ZKpCSP+OD3N24gzqTeCDuNKC7GKC2r/S8JRAuo1W2EsmyQ31hEGiORTZdsJtRrEEM2W+NS0SW2IarmQ/iupwcQJeAFjB9N4Bt/EIex5MPDUBZ/rZoPXSmsskgDYnZCuR21jzQ82GGBSUSo6/bWzx7ioFj5Vjv+SsOIquyC3flsr5MS/6WFXn24KsEJjehXiJSch6/U2DuxphzqwM7lUL5S4KJ3QFv9AUnZMyVnKAY95a3HYW+PUkVH0WAJbwJ+Gud9San+MYwBZTQ4e9IPo/m8FVVhfVg==\"}}", 129269, "https://another-remote-instance.tld/inbox"

Steps to reproduce the problem

Nothing, this is completely out of my control.

Specifications

v2.8.4

[shleeable]

Glad this is getting resolved.. it's likely the cause of slamming my instance last week.

[joenepraat]

Thank you @Gargron . Any change this can be part of a hot fix release?

My workaround for this particular account was to suspend it locally on Todon.nl, so it removed all statuses, and unsuspend the account afterwards.

[joenepraat]

@Gargron The fix seems to work in RC1. The only (minor) problem now is that Sidekiq doesn't display a difference between low priority and other priorities. So it is more difficult to follow what's going on. Maybe moving the low priority jobs to a separate queue can help with that?