Alive client connections and shutdown sequence #dilemma

[E-gy]

Any connections kept alive in the background will linger and stop server from shutting down on shutdown request.

Current behaviour:

When everything works fine:

1. Connection opened
2. Exchange of keep-alive request
3. Exchange of plz close request
4. Connection closed
5. Shutdown sequence initiated
6. Server is shut down 😄

When everything stops working fine:

1. Connection opened
2. Exchange of keep-alive request
3. Shutdown sequence initiated
4. Connection receives plz close request
5. ∞. ... (we're in limbo and there's no way out) 😟

To understand what is happening here's overview of shutdown sequence:

1. Listening sockets are shut down
2. IO completion infrastructure is closed
3. IO Yengine is gone
4. Block-await until work queue is empty and notificers are empty and all worker threads are unoccupied
5. Close work queue
6. Yengine is gone
7. Daemons are slayed

The problem is that after step 2 any IO operation is not notified of further progress (this includes connections being accepted, data received or sent, ...). Worseover after step 3, any interaction with IOYengine is UB. And because notifications go through notificers, notificers are never emptied and the system lingers on step 4.

Now, yes, there are 2 problems here - one that requires introducing a block-await in YAsynC IO before step 2 to make sure no IO operations are pending, and the can of worms that previous solution opens...

See, for classical IO operations which are supposed to complete, they can simply be awaited (ex. file IO). For IO operations that are lingering with clearly nothing to do - well, they can clean up close themselves like good bois (ex. listening sockets). But what about connections that are being kept alive..?

The above is made worse by the abstraction distance provided by YAsynC. That yes made cross-platform universal SSL a breeze, not even taek can be made aware of the nature of IO resource that it takes.

[E-gy]

First order of business - after shut-down sequence is initiated, reply exclusively with Connection: close.

[E-gy]

The should connection be closed is a grey line anyway - the initial data may be in transit, or it may be being processed by OS and not passed to application yet, or the initial line may be received but not the headers...
What is clear is:

• Requests that have been fully received must be processed and replied to
• Connections [keep-alive] where no request is being received should be closed [immediately]
• Connections where requests have been partially sufficiently received can be either processed or closed

What is less clear is for partially incompletely received connections. Also how layered processing [SSL] plays into this. And how to define sufficient reception for processing.

[E-gy]

Status update:
Currently the system will block until all alive connections are closed. New connections will not be accepted however.

[E-gy]

Fixed in a424791.