Large celery translate queue that is not clearing

[RobinWeitzel]

Describe the issue

My Weblate instance is very slow and after investigating I found that the DB was under constant load. I found that there are over 1000 translate tasks in the celery queue that seem to be stuck. The number is not reducing (even after waiting for days).

There are no errors in the logs and I update to the newest version to see if this helps (it did not). I also tried to clear the queue using
docker exec weblate-docker weblate_ 1 celery -A weblate.utils purge -f
However, celery stated that there are no messages to be purged:

Looking at the active and scheduled celery queues I also can not see over 1000 tasks:

How can I get rid of the translate tasks?

I already tried

• I've read and searched the documentation.
• I've searched for similar issues in this repository.

Steps to reproduce the behavior

Not clear what the root cause is, as such it is unclear how to reproduce this issue.

Expected behavior

Translate queue shows 0 once translations are done.

Screenshots

Exception traceback

No errors in the logs, no crashes.

How do you run Weblate?

Docker container

Weblate versions

No response

Weblate deploy checks

No response

Additional context

No response

[nijel]

There might be something wrong with the PostgreSQL database that made the queries hang.

The task queue cannot be cleared because the worker might have prefetched the queue. If you would stop the translate worker, the queue would be restored in redis and the management command would work.

For the next release, the performance should get slightly better due to d5a83bc

[github-actions]

This issue looks more like a support question than an issue. We strive to answer these reasonably fast, but purchasing the support subscription is not only more responsible and faster for your business but also makes Weblate stronger.

In case your question is already answered, making a donation is the right way to say thank you!

[RobinWeitzel]

Hi @nijel, thanks for the reply :)

How would I stop the translate worker?
The queue keeps getting larger so I definitely want to find a solution.
I searched for it in the docs but could not find the right commands.
Will this have any other impacts if we do this?

[nijel]

supervisorctl stop celery-translate inside the container should stop it.

[RobinWeitzel]

Hi @nijel ,

I tried the command but sadly this also did not do the trick.
Even after stopping the purge command still claimed that there were no messages in the queue.

However, when I restarted the celery task (with supervisorctl start celery-translate) I noticed that the queue had decreased by 10. To confirm I stoped and started the task a second time and again the queue went down.

After further testing I found that repeatedly stoping and starting the task decreases the queue by 0-10 tasks each time.
Thus, I wrote a script that stoped and started the task 1000 times (waiting 10 seconds in between each command) and now the queue is empty :)

I will keep an eye on the queue over the next week and if I see it building up again I will set up a weekly script that stops and starts the task 10 times each day to purge anything stuck in the translate queue.

Thanks for your help!

[github-actions]

The issue you have reported is now resolved. If you don’t feel it’s right, please follow its labels to get a clue for further steps.

• In case you see a similar problem, please open a separate issue.
• If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.

[nijel]

That's weird, it should process tasks continuously, not this way.

Anyway, I've added some supervisor hints to our docs in 9743509.