Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better "on limit exit code" managment for messenger worker vs supervisor #42029

Closed
quazardous opened this issue Jul 8, 2021 · 8 comments
Closed

Comments

@quazardous
Copy link

quazardous commented Jul 8, 2021

Description

With autorestart=unexpected and exitcodes=0 Supervisor don't restart a program if exit code is 0 (wich is cool because it's what we need).

But when using workers with limits (memory or time), the worker shuts down with a 0 exit code ans supervisor won't restart it (that's not cool).

So maybe adding a way to specify the "on limit exit code" would be usefull. like --on-limit-exit-code 1

Example

bin/console messenger:consume -vv --time-limit=3600 --on-limit-exit-code=1
@quazardous quazardous changed the title Better exit code on limit managment for messenger worker vs supervisor Better "on limit exit code" managment for messenger worker vs supervisor Jul 8, 2021
@quazardous
Copy link
Author

here is a quick workaround with an overridden command

<?php

namespace App\Command;

use Symfony\Component\Messenger\Command\ConsumeMessagesCommand as ConsumeMessagesCommandBase;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use App\EventListener\OnWorkerMessageLimitListener;
use App\EventListener\OnWorkerFailureLimitListener;
use Symfony\Component\Console\Input\InputOption;
use App\EventListener\OnWorkerMemoryLimitListener;
use App\EventListener\OnWorkerTimeLimitListener;
use Symfony\Component\Messenger\RoutableMessageBus;
use Psr\Container\ContainerInterface;
use Symfony\Component\EventDispatcher\EventDispatcherInterface;
use Psr\Log\LoggerInterface;

class ConsumeMessagesCommand extends ConsumeMessagesCommandBase
{
    private $eventDispatcher;

    public function __construct(RoutableMessageBus $routableBus, ContainerInterface $receiverLocator, EventDispatcherInterface $eventDispatcher, LoggerInterface $logger = null, array $receiverNames = [])
    {
        parent::__construct($routableBus, $receiverLocator, $eventDispatcher, $logger, $receiverNames);
        $this->eventDispatcher = $eventDispatcher;
    }

    protected function configure(): void
    {
        parent::configure();
        $this
            ->setName('app:messenger:consume')
            ->addOption('on-limit-exit-code', 'X', InputOption::VALUE_REQUIRED, 'Exit code when limits are reached', 0)
        ;
    }
    
    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $exitCode = 0;

        if ($onLimitExitCode = intval($input->getOption('on-limit-exit-code'))) {
            $setExitCode = function () use (&$exitCode, $onLimitExitCode) {
                $exitCode = $onLimitExitCode;
            };
            if ($limit = $input->getOption('limit')) {
                $this->eventDispatcher->addSubscriber(new OnWorkerMessageLimitListener($setExitCode, $limit));
            }
    
            if ($failureLimit = $input->getOption('failure-limit')) {
                $this->eventDispatcher->addSubscriber(new OnWorkerFailureLimitListener($setExitCode, $failureLimit));
            }
    
            if ($memoryLimit = $input->getOption('memory-limit')) {
                $this->eventDispatcher->addSubscriber(new OnWorkerMemoryLimitListener($setExitCode, $this->convertToBytes($memoryLimit)));
            }
    
            if ($timeLimit = $input->getOption('time-limit')) {
                $this->eventDispatcher->addSubscriber(new OnWorkerTimeLimitListener($setExitCode, $timeLimit));
            }
        }

        parent::execute($input, $output);
        return $exitCode;
    }
    
}

@ruudk
Copy link
Contributor

ruudk commented Aug 29, 2021

I'm wondering, why do you use autorestart=unexpected in your setup?

In my opinion you want to keep all things running forever, regardless of the exit code, until you stop or shutdown supervisor.

@carsonbot
Copy link

Thank you for this issue.
There has not been a lot of activity here for a while. Has this been resolved?

@carsonbot
Copy link

Friendly reminder that this issue exists. If I don't hear anything I'll close this.

@quazardous
Copy link
Author

I'm wondering, why do you use autorestart=unexpected in your setup?

In my opinion you want to keep all things running forever, regardless of the exit code, until you stop or shutdown supervisor.

I'll explain bellow

Friendly reminder that this issue exists. If I don't hear anything I'll close this.

Here is my use case.

I want supervisor to always restart the worker on unexpected/limit.

But for now when workers stops on limits it exits with 0.

But I'm using 0 exit to tell supervisor not to restart.
The software has a pre shutdown state: workers finish there jobs and shutdown one by one.

I'm fine with my workaround. So if no one needs such things feel free to close.

@carsonbot carsonbot removed the Stalled label Mar 16, 2022
@carsonbot
Copy link

Thank you for this issue.
There has not been a lot of activity here for a while. Has this been resolved?

@carsonbot
Copy link

Just a quick reminder to make a comment on this. If I don't hear anything I'll close this.

@carsonbot
Copy link

Hey,

I didn't hear anything so I'm going to close it. Feel free to comment if this is still relevant, I can always reopen!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants