Shlink returning 500 errors randomly

[kenske]

We have Shlink 3.3.2 running in a kubernetes cluster. It works well for the most part, but I just noticed we're getting random 500 errors. I can reproduce the error by sending a request for the same short link multiple times. About 20% of the responses will be 500 errors. Looking in the shlink logs, I see errors like these:

Shlink.ERROR - TypeError: Cannot assign DoctrineProxies\__CG__\Shlinkio\Shlink\Core\Domain\Entity\Domain to property Shlinkio\Shlink\Core\ShortUrl\Entity\ShortUrl::$authorApiKey of type ?Shlinkio\Shlink\Rest\Entity\ApiKey in /etc/shlink/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:2912

Or these:

Shlink.ERROR - BadMethodCallException: Undefined method "findOneWithDomainFallback". The method name must start with either findBy, findOneBy or countBy! in /etc/shlink/vendor/doctrine/orm/lib/Doctrine/ORM/EntityRepository.php:283

How can I fix this?

[acelaya]

Are the errors always coming from the same replica/pod?

Can you share the whole error trace?

Can you share the exact request you make with all the arguments?

[kenske]

We are running 2 replicas (using the shlinkio/shlink:3.3.2 docker image). The errors are coming from either pod.

We can reproduce the error with a simple request to the short link:

curl -I https://shlink.example.com/the-short-link

If we run that command, we get a 302 response most of the time, but then get a 500 error sometimes.

Here are a couple of stack traces:

Shlink.ERROR - Error processing task 3: [TypeError - 0] Shlinkio\Shlink\Config\Factory\ValinorConfigFactory::getMapper(): Return value must be of type CuyZ\Valinor\Mapper\TreeMapper, CuyZ\Valinor\MapperBuilder returned
#0 /etc/shlink/vendor/shlinkio/shlink-config/src/Factory/ValinorConfigFactory.php(26): Shlinkio\Shlink\Config\Factory\ValinorConfigFactory::getMapper(Object(Laminas\ServiceManager\ServiceManager))
#1 /etc/shlink/vendor/laminas/laminas-servicemanager/src/ServiceManager.php(608): Shlinkio\Shlink\Config\Factory\ValinorConfigFactory::__callStatic('config.app_opti...', Array)
#2 /etc/shlink/vendor/laminas/laminas-servicemanager/src/ServiceManager.php(230): Laminas\ServiceManager\ServiceManager->doCreate('Shlinkio\\Shlink...')
#3 [internal function]: Laminas\ServiceManager\ServiceManager->get('Shlinkio\\Shlink...')
#4 /etc/shlink/vendor/laminas/laminas-servicemanager/src/AbstractFactory/ConfigAbstractFactory.php(74): array_map(Array, Array)
#5 /etc/shlink/vendor/laminas/laminas-servicemanager/src/ServiceManager.php(566): Laminas\ServiceManager\AbstractFactory\ConfigAbstractFactory->__invoke(Object(Laminas\ServiceManager\ServiceManager), 'Shlinkio\\Shlink...', NULL)
#6 /etc/shlink/module/Core/src/EventDispatcher/CloseDbConnectionEventListenerDelegator.php(18): Laminas\ServiceManager\ServiceManager->Laminas\ServiceManager\{closure}()
#7 /etc/shlink/vendor/laminas/laminas-servicemanager/src/ServiceManager.php(586): Shlinkio\Shlink\Core\EventDispatcher\CloseDbConnectionEventListenerDelegator->__invoke(Object(Laminas\ServiceManager\ServiceManager), 'Shlinkio\\Shlink...', Object(Closure), NULL)
#8 /etc/shlink/vendor/mezzio/mezzio-swoole/src/Task/DeferredServiceListenerDelegator.php(43): Laminas\ServiceManager\ServiceManager::Laminas\ServiceManager\{closure}()
#9 /etc/shlink/vendor/laminas/laminas-servicemanager/src/ServiceManager.php(586): Mezzio\Swoole\Task\DeferredServiceListenerDelegator->__invoke(Object(Laminas\ServiceManager\ServiceManager), 'Shlinkio\\Shlink...', Object(Closure), NULL)
#10 /etc/shlink/vendor/laminas/laminas-servicemanager/src/ServiceManager.php(589): Laminas\ServiceManager\ServiceManager::Laminas\ServiceManager\{closure}()
#11 /etc/shlink/vendor/laminas/laminas-servicemanager/src/ServiceManager.php(610): Laminas\ServiceManager\ServiceManager->createDelegatorFromName('Shlinkio\\Shlink...', NULL)
#12 /etc/shlink/vendor/laminas/laminas-servicemanager/src/ServiceManager.php(230): Laminas\ServiceManager\ServiceManager->doCreate('Shlinkio\\Shlink...')
#13 /etc/shlink/vendor/mezzio/mezzio-swoole/src/Task/ServiceBasedTask.php(48): Laminas\ServiceManager\ServiceManager->get('Shlinkio\\Shlink...')
#14 /etc/shlink/vendor/mezzio/mezzio-swoole/src/Task/TaskInvokerListener.php(49): Mezzio\Swoole\Task\ServiceBasedTask->__invoke(Object(Laminas\ServiceManager\ServiceManager))
#15 /etc/shlink/vendor/mezzio/mezzio-swoole/src/Event/EventDispatcher.php(39): Mezzio\Swoole\Task\TaskInvokerListener->__invoke(Object(Mezzio\Swoole\Event\TaskEvent))
#16 /etc/shlink/vendor/mezzio/mezzio-swoole/src/SwooleRequestHandlerRunner.php(185): Mezzio\Swoole\Event\EventDispatcher->dispatch(Object(Mezzio\Swoole\Event\TaskEvent))
#17 [internal function]: Mezzio\Swoole\SwooleRequestHandlerRunner->onTask(Object(Swoole\Http\Server), 3, 9, Object(Mezzio\Swoole\Task\ServiceBasedTask))
#18 /etc/shlink/vendor/mezzio/mezzio-swoole/src/SwooleRequestHandlerRunner.php(79): Swoole\Server->start()
#19 /etc/shlink/vendor/mezzio/mezzio/src/Application.php(52): Mezzio\Swoole\SwooleRequestHandlerRunner->run()
#20 /etc/shlink/vendor/mezzio/mezzio-swoole/src/Command/StartCommand.php(125): Mezzio\Application->run()
#21 /etc/shlink/vendor/symfony/console/Command/Command.php(308): Mezzio\Swoole\Command\StartCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#22 /etc/shlink/vendor/symfony/console/Application.php(1020): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#23 /etc/shlink/vendor/symfony/console/Application.php(299): Symfony\Component\Console\Application->doRunCommand(Object(Mezzio\Swoole\Command\StartCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#24 /etc/shlink/vendor/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#25 /etc/shlink/vendor/laminas/laminas-cli/bin/laminas(54): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#26 /etc/shlink/vendor/bin/laminas(120): include('/etc/shlink/ven...')
#27 {main}

Another one:

Shlink.ERROR - BadMethodCallException: Undefined method "findOneWithDomainFallback". The method name must start with either findBy, findOneBy or countBy! in /etc/shlink/vendor/doctrine/orm/lib/Doctrine/ORM/EntityRepository.php:283
Stack trace:
#0 /etc/shlink/module/Core/src/ShortUrl/ShortUrlResolver.php(42): Doctrine\ORM\EntityRepository->__call('findOneWithDoma...', Array)
#1 /etc/shlink/module/Core/src/Action/AbstractTrackingAction.php(32): Shlinkio\Shlink\Core\ShortUrl\ShortUrlResolver->resolveEnabledShortUrl(Object(Shlinkio\Shlink\Core\ShortUrl\Model\ShortUrlIdentifier))
#2 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(27): Shlinkio\Shlink\Core\Action\AbstractTrackingAction->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#3 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#4 /etc/shlink/module/Core/src/ShortUrl/Middleware/TrimTrailingSlashMiddleware.php(25): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#5 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(27): Shlinkio\Shlink\Core\ShortUrl\Middleware\TrimTrailingSlashMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#6 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#7 /etc/shlink/vendor/akrabat/ip-address-middleware/src/IpAddress.php(132): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#8 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(27): RKA\Middleware\IpAddress->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#9 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#10 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(75): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#11 /etc/shlink/vendor/mezzio/mezzio-router/src/Route.php(85): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#12 /etc/shlink/vendor/mezzio/mezzio-router/src/RouteResult.php(97): Mezzio\Router\Route->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#13 /etc/shlink/vendor/mezzio/mezzio-router/src/Middleware/DispatchMiddleware.php(30): Mezzio\Router\RouteResult->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#14 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(27): Mezzio\Router\Middleware\DispatchMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#15 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#16 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(75): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#17 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#18 /etc/shlink/vendor/laminas/laminas-stratigility/src/Middleware/PathMiddlewareDecorator.php(42): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#19 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Laminas\Stratigility\Middleware\PathMiddlewareDecorator->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#20 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(42): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#21 /etc/shlink/vendor/mezzio/mezzio-router/src/Middleware/ImplicitHeadMiddleware.php(103): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#22 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(27): Mezzio\Router\Middleware\ImplicitHeadMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#23 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#24 /etc/shlink/vendor/mezzio/mezzio-router/src/Middleware/RouteMiddleware.php(50): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#25 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(27): Mezzio\Router\Middleware\RouteMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#26 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#27 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(75): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#28 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#29 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(42): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#30 /etc/shlink/vendor/shlinkio/shlink-common/src/Middleware/CloseDbConnectionMiddleware.php(24): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#31 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(27): Shlinkio\Shlink\Common\Middleware\CloseDbConnectionMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#32 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#33 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(75): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#34 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#35 /etc/shlink/vendor/laminas/laminas-stratigility/src/Middleware/PathMiddlewareDecorator.php(42): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#36 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Laminas\Stratigility\Middleware\PathMiddlewareDecorator->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#37 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(42): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#38 /etc/shlink/module/Rest/src/Middleware/CrossDomainMiddleware.php(24): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#39 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(27): Shlinkio\Shlink\Rest\Middleware\CrossDomainMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#40 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#41 /etc/shlink/vendor/laminas/laminas-stratigility/src/Middleware/ErrorHandler.php(129): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#42 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(27): Laminas\Stratigility\Middleware\ErrorHandler->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#43 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#44 /etc/shlink/vendor/php-middleware/request-id/src/RequestIdMiddleware.php(32): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#45 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(27): PhpMiddleware\RequestId\RequestIdMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#46 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#47 /etc/shlink/vendor/shlinkio/shlink-common/src/Middleware/ContentLengthMiddleware.php(24): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#48 /etc/shlink/vendor/mezzio/mezzio/src/Middleware/LazyLoadingMiddleware.php(27): Shlinkio\Shlink\Common\Middleware\ContentLengthMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#49 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Mezzio\Middleware\LazyLoadingMiddleware->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#50 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(75): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#51 /etc/shlink/vendor/laminas/laminas-stratigility/src/Next.php(49): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\Next))
#52 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(75): Laminas\Stratigility\Next->handle(Object(Laminas\Diactoros\ServerRequest))
#53 /etc/shlink/vendor/laminas/laminas-stratigility/src/MiddlewarePipe.php(64): Laminas\Stratigility\MiddlewarePipe->process(Object(Laminas\Diactoros\ServerRequest), Object(Laminas\Stratigility\EmptyPipelineHandler))
#54 /etc/shlink/vendor/mezzio/mezzio-swoole/src/Event/RequestHandlerRequestListener.php(116): Laminas\Stratigility\MiddlewarePipe->handle(Object(Laminas\Diactoros\ServerRequest))
#55 /etc/shlink/vendor/mezzio/mezzio-swoole/src/Event/EventDispatcher.php(39): Mezzio\Swoole\Event\RequestHandlerRequestListener->__invoke(Object(Mezzio\Swoole\Event\RequestEvent))
#56 /etc/shlink/vendor/mezzio/mezzio-swoole/src/SwooleRequestHandlerRunner.php(135): Mezzio\Swoole\Event\EventDispatcher->dispatch(Object(Mezzio\Swoole\Event\RequestEvent))
#57 [internal function]: Mezzio\Swoole\SwooleRequestHandlerRunner->onRequest(Object(Swoole\Http\Request), Object(Swoole\Http\Response))
#58 {main}