tkt-38155: fix(middlewared/jail): handle start/stop gracefully using iocage

[william-gr]

Ticket: #38155

[themylogin]

Are we supposed to wait for jails to terminate properly?

Won't SIGTERM abort our stop_on_shutdown call?

[william-gr]

@themylogin thats why its still WIP. Any ideas on how to fix that? I was thinking of cancel all tasks only after all terminate() calls?

[themylogin]

I am not familiar with FreeBSD shutdown system. Will a blocking call last in rc.shutdown.local? Then we might just call middleware method (with reasonable timeout) instead of sending event. It would be great to delay middleware shutdown for this time and not to involve terminate here at all, it would be a pain for development when middleware restarts occur frequently.

[william-gr]

@themylogin it would but I would rather not.

In your approach there is no reasonable timeout we can ever come up with, it needs to be up to the plugin.
Who knows how many VMs/jails there are to wait long enough?

I dont see the problem in involving terminate ? I am handling that pain using the event and the lock, whats the flaw in that?

It has not effect whatsoever in development since the shutdown event is not sent unless you actually shutdown.

[themylogin]

I dont see the problem in involving terminate? I am handling that pain using the event and the lock, whats the flaw in that?

The problem I've imagined is rc sending SIGKILL to middlewared that spent too much time not responding to SIGTERM.

[william-gr]

And why is that a problem? All services are the same way.

We would be just transferring responsibility, same happens on shutdown, rc.shutdown.local is just part of /etc/rc.d/local.

[themylogin]

I see. My concern is that timeout that is reasonable for shutting down one process may not be enough for shutting down a jail (which may consist of several processes) or even several jails, no matter if shutdown is sequential or parallel. If we do it in middleware, we risk unclear middleware shutdown, if we do it in shutdown rc script, we risk everything below it, if we do it in separate rc script we still may fail to shutdown all jails in time. These are my worries if users would run something fragile like PostgreSQL in jails.

[william-gr]

What timeout are you talking about? There is no timeout in waiting pid to die in rc subsystem.

[themylogin]

I've noted I am not so familiar with it :-D no concerns then.
Does it also mean that something can easily block shutdown entirely?

[william-gr]

Yes, in fact recently we had a problem where consul-alerts would block and not shutdown.

[themylogin]

Last concern: if IOC stop won't block, LGTM!