You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Apr 1, 2025. It is now read-only.
Copy file name to clipboardExpand all lines: README.md
+9Lines changed: 9 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -113,6 +113,15 @@ Usually a command to describe what is going on on a pod is
113
113
Starting celery with '--without-mingle' prevented it from crashing everytime a new worker was started in parallel.
114
114
More info: https://stackoverflow.com/questions/55249197/what-are-the-consequences-of-disabling-gossip-mingle-and-heartbeat-for-celery-w
115
115
116
+
## Never-ending tasks in ci_fetch queue
117
+
118
+
Sometimes celery suffers from `celery_chord_unlock` tasks that never reach a timeout and causes the fetch-workers to
119
+
do useless work. I still have not found the reason for this yet, but until then, it's convenient to purge the ci_fetch queue.
120
+
121
+
1. First kill all fetch-workers
122
+
2. ssh -i tmp/qareports_private_ssh_key `cat terraform/generated/production_rabbitmq_host_public` (you'll need to run `./qareports production queues` first)
123
+
3. sudo rabbitmqctl purge_queue ci_fetch
124
+
116
125
## Emails
117
126
118
127
We're currently using AWS Simple Email Service aka SES to send emails. On an account that SES was never used, AWS puts it under sandbox
0 commit comments