Redis WRONGTYPE Operation against a key holding the wrong kind of value

[jtallinger]

Hi,

I'm scheduling a few jobs to be queued every minute. It has been working fine during past few days when testing. Then yesterday I added manual tags and it was working fine for couple of hours, but when server hit midnight UTC time something weird started to happen. Redis is saying wrong type fault?

Of 3 jobs only the first one would be added to the queue. Error log:

WRONGTYPE Operation against a key holding the wrong kind of value {"exception":"[object] (Predis\Response\ServerException(code: 0): WRONGTYPE Operation against a key holding the wrong kind of value at /home/.../public_html/vendor/predis/predis/src/Pipeline/Pipeline.php:102)
[stacktrace]
#0 /home/.../public_html/vendor/predis/predis/src/Pipeline/Pipeline.php(146): Predis\Pipeline\Pipeline->exception(Object(Predis\Connection\StreamConnection), Object(Predis\Response\Error))
#1 /home/.../public_html/vendor/predis/predis/src/Pipeline/Pipeline.php(165): Predis\Pipeline\Pipeline->executePipeline(Object(Predis\Connection\StreamConnection), Object(SplQueue))
#2 /home/.../public_html/vendor/predis/predis/src/Pipeline/Pipeline.php(214): Predis\Pipeline\Pipeline->flushPipeline()
#3 /home/.../public_html/vendor/predis/predis/src/Client.php(445): Predis\Pipeline\Pipeline->execute(Object(Closure))
#4 /home/.../public_html/vendor/predis/predis/src/Client.php(396): Predis\Client->createPipeline(NULL, Object(Closure))
#5 /home/.../public_html/vendor/predis/predis/src/Client.php(418): Predis\Client->sharedContextFactory('createPipeline', Array)
#6 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(96): Predis\Client->pipeline(Object(Closure))
#7 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(108): Illuminate\Redis\Connections\Connection->command('pipeline', Array)
#8 /home/.../public_html/vendor/laravel/horizon/src/Repositories/RedisTagRepository.php(84): Illuminate\Redis\Connections\Connection->__call('pipeline', Array)
#9 /home/.../public_html/vendor/laravel/horizon/src/Listeners/StoreMonitoredTags.php(39): Laravel\Horizon\Repositories\RedisTagRepository->add('2560', Array)
#10 [internal function]: Laravel\Horizon\Listeners\StoreMonitoredTags->handle(Object(Laravel\Horizon\Events\JobPushed))
#11 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(379): call_user_func_array(Array, Array)
#12 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(209): Illuminate\Events\Dispatcher->Illuminate\Events\{closure}('Laravel\\Horizon...', Array)
#13 /home/.../public_html/vendor/laravel/horizon/src/RedisQueue.php(155): Illuminate\Events\Dispatcher->dispatch('Laravel\\Horizon...')
#14 /home/.../public_html/vendor/laravel/horizon/src/RedisQueue.php(62): Laravel\Horizon\RedisQueue->event('high', Object(Laravel\Horizon\Events\JobPushed))
#15 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Support/helpers.php(1041): Laravel\Horizon\RedisQueue->Laravel\Horizon\{closure}('2560')
#16 /home/.../public_html/vendor/laravel/horizon/src/RedisQueue.php(63): tap('2560', Object(Closure))
#17 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Queue/RedisQueue.php(91): Laravel\Horizon\RedisQueue->pushRaw('{"displayName":...', 'high')
#18 /home/.../public_html/vendor/laravel/horizon/src/RedisQueue.php(46): Illuminate\Queue\RedisQueue->push(Object(App\Jobs\processMonitoringEvents), '', 'high')
#19 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Queue/Queue.php(37): Laravel\Horizon\RedisQueue->push(Object(App\Jobs\processMonitoringEvents), '', 'high')
#20 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(177): Illuminate\Queue\Queue->pushOn('high', Object(App\Jobs\processMonitoringEvents))
#21 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(160): Illuminate\Bus\Dispatcher->pushCommandToQueue(Object(Laravel\Horizon\RedisQueue), Object(App\Jobs\processMonitoringEvents))
#22 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(73): Illuminate\Bus\Dispatcher->dispatchToQueue(Object(App\Jobs\processMonitoringEvents))
#23 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Foundation/Bus/PendingDispatch.php(112): Illuminate\Bus\Dispatcher->dispatch(Object(App\Jobs\processMonitoringEvents))
#24 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Console/Scheduling/Schedule.php(99): Illuminate\Foundation\Bus\PendingDispatch->__destruct()
#25 [internal function]: Illuminate\Console\Scheduling\Schedule->Illuminate\Console\Scheduling\{closure}()
#26 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(29): call_user_func_array(Object(Closure), Array)
#27 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(75): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#28 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Object(Closure), Object(Closure))
#29 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Container/Container.php(564): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Object(Closure), Array, NULL)
#30 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Console/Scheduling/CallbackEvent.php(74): Illuminate\Container\Container->call(Object(Closure), Array)
#31 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Console/Scheduling/ScheduleRunCommand.php(111): Illuminate\Console\Scheduling\CallbackEvent->run(Object(Illuminate\Foundation\Application))
#32 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Console/Scheduling/ScheduleRunCommand.php(75): Illuminate\Console\Scheduling\ScheduleRunCommand->runEvent(Object(Illuminate\Console\Scheduling\CallbackEvent))
#33 [internal function]: Illuminate\Console\Scheduling\ScheduleRunCommand->handle()
#34 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(29): call_user_func_array(Array, Array)
#35 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(87): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#36 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(31): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#37 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Container/Container.php(564): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#38 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Console/Command.php(184): Illuminate\Container\Container->call(Array)
#39 /home/.../public_html/vendor/symfony/console/Command/Command.php(251): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#40 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Console/Command.php(171): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#41 /home/.../public_html/vendor/symfony/console/Application.php(886): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#42 /home/.../public_html/vendor/symfony/console/Application.php(262): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Console\Scheduling\ScheduleRunCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#43 /home/.../public_html/vendor/symfony/console/Application.php(145): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#44 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Console/Application.php(89): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#45 /home/.../public_html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(122): Illuminate\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#46 /home/.../public_html/artisan(35): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#47 {main}
"}

This kept going on for next several hours until I removed the manual tags from job class. The setup is fairly simple:

$schedule->job(new processMonitoringAnalyzing(), 'main')->everyMinute()->withoutOverlapping();
$schedule->job(new processMonitoringCleaningFrequent(), 'main')->everyMinute()->withoutOverlapping();
$schedule->job(new processMonitoringEvents(), 'high')->everyMinute()->withoutOverlapping();

And jobs look like:

public function handle()
{
$service = new MonitorAppController();
$service->cleanUpEventData();
}

public function tags()
{
return ['monitoring'];
}

Adding tags to jobs are not mentioned in https://laravel.com/docs/5.6/queues. Unsure where this really belongs. Adding job to queue fails (which is not part of Horizon) but tags() are added by Horizon.

I'm using Laravel 5.6.12 with Apache/2.4.10 (Debian), PHP 7.2.7 and Redis server v=4.0.10.

Any help appreciated. :)

[driesvints]

Heya. Docs for tagging can be found here: https://laravel.com/docs/5.7/horizon#tags. It's something Horizon specific.

Let us know if this problem still happens for you on the latest versions of Horizon and Laravel.