Replies: 3 comments
-
Seeing this as well. Any information would be great to have |
Beta Was this translation helpful? Give feedback.
-
I am constantly getting this as well, and from what I can tell it's because the retry timeout for RedisLock is set to a default of 10 seconds while the default expiration time for RedisLock is 15 minutes, per #16291 When called from ActivityPub::Activity::Announce this causes Sidekiq to retry until its timeout and then throw Mastodon::RaceConditionError which causes another Sidekiq retry. Looking at 7071 executions of ActivityPub::ProcessingWorker it appears that most of them complete in less than 120 seconds so I'm going to look into changing the RedisLock timeout and its retry timeout: |
Beta Was this translation helpful? Give feedback.
-
I was seeing this too, when the queue backlog was really high. I created multiple Sidekiq systemd processes (mentioned in the docs) for the |
Beta Was this translation helpful? Give feedback.
-
Error log being outputted by mastodon-sidekiq in journalctl
Jan 08 21:47:34 nyc bundle[1710760]: 2021-01-09T05:47:34.894Z pid=1710760 tid=ost090nuw class=ActivityPub::ProcessingWorker jid=5822d522630807dac01a3579 elapsed=10.024 INFO: fail Jan 08 21:47:34 nyc bundle[1710760]: 2021-01-09T05:47:34.895Z pid=1710760 tid=ost090nuw WARN: {"context":"Job raised exception","job":{"retry":8,"queue":"default","backtrace":true,"class":"ActivityPub::ProcessingWorker","args":[245,"{\"@context\": \"https://www.w3.org/ns/activitystreams\", \"type\": \"Announce\", \"to\": [\"https://relay.libranet.de/followers\"], \"actor\": \"https://relay.libranet.de/actor\", \"object\": \"https://mstdn.foxfam.club/users/EthanVanSciver/statuses/105522963683767033\", \"id\": \"https://relay.libranet.de/activities/7907ea1e-d999-41a7-a38d-80bd73c4cc3b\"}",null],"jid":"5822d522630807dac01a3579","created_at":1610159500.08584,"enqueued_at":1610159500.0859277},"jobstr":"{\"retry\":8,\"queue\":\"default\",\"backtrace\":true,\"class\":\"ActivityPub::ProcessingWorker\",\"args\":[245,\"{\\\"@context\\\": \\\"https://www.w3.org/ns/activitystreams\\\", \\\"type\\\": \\\"Announce\\\", \\\"to\\\": [\\\"https://relay.libranet.de/followers\\\"], \\\"actor\\\": \\\"https://relay.libranet.de/actor\\\", \\\"object\\\": \\\"https://mstdn.foxfam.club/users/EthanVanSciver/statuses/105522963683767033\\\", \\\"id\\\": \\\"https://relay.libranet.de/activities/7907ea1e-d999-41a7-a38d-80bd73c4cc3b\\\"}\",null],\"jid\":\"5822d522630807dac01a3579\",\"created_at\":1610159500.08584,\"enqueued_at\":1610159500.0859277}"} Jan 08 21:47:34 nyc bundle[1710760]: 2021-01-09T05:47:34.895Z pid=1710760 tid=ost090nuw WARN: Mastodon::RaceConditionError: Mastodon::RaceConditionError Jan 08 21:47:34 nyc bundle[1710760]: 2021-01-09T05:47:34.895Z pid=1710760 tid=ost090nuw WARN: /home/mastodon/live/app/lib/activitypub/activity/announce.rb:28:in `block in perform' Jan 08 21:47:34 nyc bundle[1710760]: /home/mastodon/live/vendor/bundle/ruby/2.6.0/gems/mario-redis-lock-1.2.1/lib/redis_lock.rb:49:in `acquire' Jan 08 21:47:34 nyc bundle[1710760]: /home/mastodon/live/app/lib/activitypub/activity/announce.rb:7:in `perform'
I don't believe I was having this issue before my disk filled up the other day and I had to purge some extremely large logs(kern.log and syslog combined 128GB) and reboot.
Beta Was this translation helpful? Give feedback.
All reactions