Graceful Restart Multiprocess

[smathersPersonal]

I've been playing around with different ways to achieve graceful restart. I have 1 method that actually works but its not very efficient.

$server = new Swoole\Server('[::]', 9090, SWOOLE_PROCESS);
$sTables = new SwooleTables();
$sTables->worker = ProcessHelper::setupWorkerTable();
$cpus = swoole_cpu_num();
$server->set([
  'log_level' => 4,
  'enable_coroutine' => true,
  'reactor_num' => $cpus * 2,
  'worker_num' => $cpus * 6,
  'hook_flags' => SWOOLE_HOOK_TCP | SWOOLE_HOOK_NATIVE_CURL | SWOOLE_HOOK_SLEEP | SWOOLE_HOOK_BLOCKING_FUNCTION,
   'dispatch_func' => function($server, $fd, $type) use ($sTables) {
      return ProcessHelper::dispatchFunction($sTables,$fd,$type);
    },
]);

To get the above to work,

1. Whenever a new worker starts i write to a swoole table some details. Cnt represents process count i care about to not get interrupted. When one of those processes starts i incr (self::$sTables?->worker->incr(self::getPid(self::$server), 'cnt');)

  public static function setupWorkerTable(): Swoole\Table {
    $workerTable = new Swoole\Table(2000);
    $workerTable->column('pid', Swoole\Table::TYPE_INT);
    $workerTable->column('reset', Swoole\Table::TYPE_INT);
    $workerTable->column('time', Swoole\Table::TYPE_INT);
    $workerTable->column('workerId', Swoole\Table::TYPE_INT);
    $workerTable->column('stop_connections', Swoole\Table::TYPE_INT);
    $workerTable->column('cnt', Swoole\Table::TYPE_INT);
    $workerTable->create();
    return $workerTable;
  }

2. Graceful restart sends a message to the swoole server from outside saying a graceful restart is requested.
3. Swoole server sets a reset flag on each worker in the $sTables, as well as, sets another flag with the first worker to try and restart (stop_connections)

4. The worker that is trying to restart is setup with a poller to check 'cnt'. This poller is a simple swoole_timer_tick()
5. Once the worker has cnt = 0, its safe to restart and worker cleans up, picks a new worker to try to restart and stops()

All the above will work with a server that does not take alot of requests, by chance eventually u should reach a cnt of 0

Now here is the inefficient part, duplicate "fixed" dispatch mode in a php function with 1 change.
That is check the $stables to make sure if its a new connection dont pick workers that have "stop_connections" set (in order words, don't pick a worker that is trying to restart)!

So in order to make this better

I've tried using swooles graceful shutdown in the newest version of swoole, v6.0.0

$server = new Swoole\Server('[::]', 9090, SWOOLE_BASE);
$sTables = new SwooleTables();
$sTables->worker = ProcessHelper::setupWorkerTable();
$cpus = swoole_cpu_num();
$server->set([
  'log_level' => 4,
  'enable_coroutine' => true,
  'reactor_num' => $cpus * 2,
  'worker_num' => $cpus * 6,
  'hook_flags' => SWOOLE_HOOK_TCP | SWOOLE_HOOK_NATIVE_CURL | SWOOLE_HOOK_SLEEP | SWOOLE_HOOK_BLOCKING_FUNCTION,
  'max_wait_time' => 10800 //3 hr
]);

So to explain this, SWOOLE_BASE is suppose to be more efficient right? max_wait_time of 3 hours because this system controls phone calls, some of those phone calls go quite long. And i cannot interrupt the flow of phone events to the application, otherwise the call drops or is mis handled.

Trial 1, issue graceful restart via cmdline: kill -USR1 main_process_PID
I can see it working, but instead of the workers blocking new connections. All new connections and and existing ones are blocked (existing connections stop receiving tcp events). This is no good the phone call i had tested this with fails to handle properly
Trial 2, Use some of the pieces above in that i store details in $sTable, and slowly restart workers one by one using the stop() graceful feature. self::$server->stop($row['workerId'],true);, the true means wait for event loop to be empty. This worked mostly, but as testing got on and chance kicked in, my phone call eventually hit a worker that was in this gracefully state and was blocked from making a new connection instead of picking a worker that was not restarting! So in the end this was no better then above.

Q: Was wondering what peoples thoughts on all this?
Feature: Maybe a new dispatch mode that pays attention to graceful restarts and stops new connections from dispatching there?

Thanks a lot for reading!