Horizon: Error while reading line from the server. (Redis/KVRocks)

[thbley]

Laravel Version

v12.1.1

PHP Version

8.4.4

Database Driver & Version

MariaDB 10.11.11

Description

Sometimes we get the exception "Error while reading line from the server."

In Horizon we have defined 'maxTime' => 3600, for each worker.

In KVRocks we have defined "timeout 7200".

So basically a worker should stop after 3600s and if it's not stopped, KVrocks terminates the connection after 7200s of inactivity.

Looking at the code from vendor/laravel/framework/src/Illuminate/Queue/Worker.php in with method daemon(), the method stopIfNecessary() is called at the end of the loop.

Looking at the stack trace, we get the exception when calling getNextJob() inside the method daemon().

So I'm suggesting to call stopIfNecessary() additionally before or inside getNextJob() or implement an automatic re-connect for "Error while reading line from the server":

            $status = $this->stopIfNecessary(
                $options, $lastRestart, $startTime, $jobsProcessed, $job
            );

            if (! is_null($status)) {
                return $this->stop($status, $options);
            }

            // First, we will attempt to get the next job off of the queue. We will also
            // register the timeout handler and reset the alarm for this job so it is
            // not stuck in a frozen state forever. Then, we can fire off this job.
            $job = $this->getNextJob(
                $this->manager->connection($connectionName), $queue
            );

Array
(
    [exception] => Array
        (
            [class] => Predis\Connection\ConnectionException
            [message] => Error while reading line from the server. [tcp://127.0.0.1:6379]
            [code] => 0
            [file] => vendor/predis/predis/src/Connection/AbstractConnection.php:144
        )

    [trace] => #0 vendor/predis/predis/src/Connection/StreamConnection.php(291): Predis\Connection\AbstractConnection->onConnectionError()
#1 vendor/predis/predis/src/Connection/AbstractConnection.php(132): Predis\Connection\StreamConnection->read()
#2 vendor/predis/predis/src/Connection/AbstractConnection.php(124): Predis\Connection\AbstractConnection->readResponse()
#3 vendor/predis/predis/src/Client.php(381): Predis\Connection\AbstractConnection->executeCommand()
#4 vendor/predis/predis/src/Client.php(335): Predis\Client->executeCommand()
#5 vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(116): Predis\Client->__call()
#6 vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(229): Illuminate\Redis\Connections\Connection->command()
#7 vendor/laravel/framework/src/Illuminate/Queue/RedisQueue.php(318): Illuminate\Redis\Connections\Connection->__call()
#8 vendor/laravel/framework/src/Illuminate/Queue/RedisQueue.php(250): Illuminate\Queue\RedisQueue->retrieveNextJob()
#9 vendor/laravel/horizon/src/RedisQueue.php(139): Illuminate\Queue\RedisQueue->pop()
#10 /vendor/laravel/framework/src/Illuminate/Queue/Worker.php(351): Laravel\Horizon\RedisQueue->pop()
#11 /vendor/laravel/framework/src/Illuminate/Queue/Worker.php(366): Illuminate\Queue\Worker->{closure:Illuminate\Queue\Worker::getNextJob():350}()
#12 /vendor/laravel/framework/src/Illuminate/Queue/Worker.php(164): Illuminate\Queue\Worker->getNextJob()
#13 /vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(149): Illuminate\Queue\Worker->daemon()
#14 /vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(132): Illuminate\Queue\Console\WorkCommand->runWorker()
#15 /vendor/laravel/horizon/src/Console/WorkCommand.php(51): Illuminate\Queue\Console\WorkCommand->handle()
#16 /vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Laravel\Horizon\Console\WorkCommand->handle()
#17 /vendor/laravel/framework/src/Illuminate/Container/Util.php(43): Illuminate\Container\BoundMethod::{closure:Illuminate\Container\BoundMethod::call():35}()
#18 /vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(95): Illuminate\Container\Util::unwrapIfClosure()
#19 /vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod()
#20 /vendor/laravel/framework/src/Illuminate/Container/Container.php(754): Illuminate\Container\BoundMethod::call()
#21 /vendor/laravel/framework/src/Illuminate/Console/Command.php(213): Illuminate\Container\Container->call()
#22 /vendor/symfony/console/Command/Command.php(279): Illuminate\Console\Command->execute()
#23 /vendor/laravel/framework/src/Illuminate/Console/Command.php(182): Symfony\Component\Console\Command\Command->run()
#24 /vendor/symfony/console/Application.php(1094): Illuminate\Console\Command->run()
#25 /vendor/symfony/console/Application.php(342): Symfony\Component\Console\Application->doRunCommand()
#26 /vendor/symfony/console/Application.php(193): Symfony\Component\Console\Application->doRun()
#27 /vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(198): Symfony\Component\Console\Application->run()
#28 /artisan(35): Illuminate\Foundation\Console\Kernel->handle()
#29 {main}

Steps To Reproduce

Let a queue process a job and don't send a new job for 2 hours (or other values depending on your timeout setting in Redis/KVRocks).

[thbley]

Related: laravel/horizon#366 https://laracasts.com/discuss/channels/forge/queue-with-predis-error-while-reading-line-from-the-server

We use predis/predis v2.3.0

[github-actions]

Thank you for reporting this issue!

As Laravel is an open source project, we rely on the community to help us diagnose and fix issues as it is not possible to research and fix every issue reported to us via GitHub.

If possible, please make a pull request fixing the issue you have described, along with corresponding tests. All pull requests are promptly reviewed by the Laravel team.

Thank you!

[rohDevellopper]

Hi, can I work on this?

[thbley]

Yes sure.

[rohDevellopper]

I think that to solve it you have to Adjust the block_for Setting
Lower the block_for value in your queue configuration to ensure workers check for termination conditions more frequently. This prevents prolonged blocking on Redis connections:

1 - change (config/queue.php):
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => 5, // Adjust to a lower value (e.g., 5 seconds)
],

2 - change (config/database.php) :
'redis' => [
'client' => env('REDIS_CLIENT', 'predis'),
'default' => [
'host' => env('REDIS_HOST', '127.0.0.1'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => env('REDIS_DB', 0),
'read_write_timeout' => 3600, // Less than KVRocks' 7200s
],
],

3 - Create a custom worker to catch connection exceptions and reconnect in (app/Queue/Worker.php)
4 - Update Horizon's configuration to use your custom worker in (App\Providers\HorizonServiceProvider.php):
use App\Queue\Worker;
use Laravel\Horizon\Horizon;

    public function boot()
    {
        parent::boot();
    
        Horizon::useWorker(Worker::class);
    }

[thbley]

I already had 'block_for' => 5. Changing 'read_write_timeout' to 3600 removed all the errors.

Thank You!