Could not acquire a mutex lock for the queue (queue).

[BAM-Dennis]

Description

I just updated Craft from 3.4 to 3.7.5. I used the ./craft update all command to update the system using the command line.
Everything went fine and alongside raft i also updated Neo, Supertable and some more irrelevant plugins.

After that some more updates appeared, which had another minimum System Requirement for Neo and Supertable. Again i used ./craft update all command and the migrations went fine.

But now i have 1,8 million jobs in the queue with following error message: Could not acquire a mutex lock for the queue (queue).

I retried all jobs using ./craft queue-manager/queue/retry-all but if instantly fails with:

Jobs are mostly the following: Applying new propagation method to Neo blocks
We are on a 34 Site instance with a few 1000 Entries.

Exception 'yii\base\Exception' with message 'Could not acquire a mutex lock for the queue (queue).'

in /home/***/staging/vendor/craftcms/cms/src/queue/Queue.php:780

Stack trace:
#0 /home/***/staging/vendor/craftcms/cms/src/queue/Queue.php(618): craft\queue\Queue->_lock(Object(Closure))
#1 /home/***/staging/vendor/craftcms/cms/src/queue/Queue.php(130): craft\queue\Queue->reserve()
#2 [internal function]: craft\queue\Queue->craft\queue\{closure}(Object(Closure))
#3 /home/***/staging/vendor/yiisoft/yii2-queue/src/cli/Queue.php(117): call_user_func(Object(Closure), Object(Closure))
#4 /home/***/staging/vendor/craftcms/cms/src/queue/Queue.php(140): yii\queue\cli\Queue->runWorker(Object(Closure))
#5 /home/***/staging/vendor/craftcms/cms/src/queue/Command.php(77): craft\queue\Queue->run()
#6 [internal function]: craft\queue\Command->actionRun()
#7 /home/***/staging/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#8 /home/***/staging/vendor/yiisoft/yii2/base/Controller.php(181): yii\base\InlineAction->runWithParams(Array)
#9 /home/***/staging/vendor/yiisoft/yii2/console/Controller.php(184): yii\base\Controller->runAction('run', Array)
#10 /home/***/staging/vendor/yiisoft/yii2/base/Module.php(534): yii\console\Controller->runAction('run', Array)
#11 /home/***/staging/vendor/yiisoft/yii2/console/Application.php(181): yii\base\Module->runAction('queue/run', Array)
#12 /home/***/staging/vendor/craftcms/cms/src/console/Application.php(89): yii\console\Application->runAction('queue/run', Array)
#13 /home/***/staging/vendor/yiisoft/yii2/console/Application.php(148): craft\console\Application->runAction('queue/run', Array)
#14 /home/***/staging/vendor/yiisoft/yii2/base/Application.php(392): yii\console\Application->handleRequest(Object(craft\console\Request))
#15 /home/***/staging/craft(22): yii\base\Application->run()
#16 {main}

Steps to reproduce

1. Update from 3.4 to 3.75
2. Could not acquire a mutex lock for the queue (queue). appears afterwards

Other information

• Neo version: 2.11.2
• Craft version: 3.7.5
• What is the affected Neo field's propagation method: Mostly "Only save blocks to the site they were created in" (none)

[brandonkelly]

Most likely this is due to your DB server getting into a weird state. Can you try rebooting it?

[BAM-Dennis]

@brandonkelly We have supervisor running in the background to handle the queue with numprocs=2

If i stop 1 of those 2 processes, i don't get "Could not acquire a mutex lock for the queue (queue)." error.
The moment i start the second process, i instantly receive the error on nearly any job item one after another.

About the DB Server getting into a "weird state". Might this happen because of a DB misconfiguration?

[angrybrad]

My guess is part of the reason it happens when there are two runners is because there is a resource contention issue on the database side as a result of the 1.8 million jobs in the queue table.

If you knock the runners down to 1, let it process all of the 1.8 million jobs, then bump it back to 2, do you still get the same error for any newly added jobs?

[BAM-Dennis]

hey @angrybrad the main problem is, that executing one single job takes 5 seconds so 1.8 million would take around 100days to complete.

i have checked what and why neo is reapplying the propagation methods, and i am not sure if i just might truncate the table.
everything looked fine in the first case, but i have no idea if that breaks stuff in the future.

i am of course testing this on a staging system, so i can always rollback to craft 3.4, Maybe updating one minor version by one could help? i dunno if doing 3 minor version steps in 1 single update is good.

what do you think?

[angrybrad]

Yeah... 1.8 million jobs seems a bit excessive for Neo to be adding in this context.

that executing one single job takes 5 seconds so 1.8 million would take around 100days to complete.

Even with 2 runners, that's 50 days, which doesn't seem practical.

Probably wouldn't hurt to do smaller incremental updates and let whatever jobs need to run in between each one to finish before attempting the next one and see if that makes things more managable.

[BAM-Dennis]

What is the recommended way to handle this?
Update version numbers in composer and after install call ./craft migrate/all ?

[angrybrad]

@BAM-Dennis Yup... that'll do it.

[angrybrad]

Going to go ahead and close this, but just comment back if you can't get it sorted.