Allow input and output streams of connected_socket to be closed independently

[tgrabiec]

For posix_connected_socket_impl (and tls::session [1]) when either input or output streams are closed, the whole socket is implicitly closed by closing pollable_fd. After the socket is closed, no further actions are allowed on it. It also cannot be closed concurrently with any operations on the socket (pollable_fd::close() destroys pollable_fd_state and closes the file descriptor). Because of that, the use of input and output streams must be synchronized with regard to closing of each of them. We cannot close the output stream when input stream is still in use, and vice versa. We cannot close input stream after closing the output stream.

One problem is that this makes using the streams correctly complicated. Another problem is that the general contract for input and output streams is that they should be closed before they're destroyed. Here we're violating that by requiring one of the streams to not be closed. This is a problem if the ownership of one of the streams is passed to generic code which will try to close it.

I propose to change this so that:

• input and output streams of a connected_socket can be closed independently. Closing input or output stream of a connected_socket would only close the read or the write side of the socket respectively. E.g. a close() on input_stream associated with posix_connected_socket_impl would behave like shutdown_read().
• connected_socket object can be destroyed before the streams. It can be decomposed into input and output streams. When both streams are closed, the underlying socket, which is shared between them, is also closed.

[1] Similar applies to tls::session, if input stream is closed after the output stream, we will crash:

   #1 0x64b724 in reactor_backend_epoll::writeable(pollable_fd_state&) core/reactor.cc:494
    #2 0x8af0c7 in reactor::writeable(pollable_fd_state&) core/reactor.hh:949
    #3 0xcaf286 in pollable_fd::write_some(net::packet&) core/reactor.hh:1209
    #4 0xcaf947 in pollable_fd::write_all(net::packet&) core/reactor.hh:1235
    #5 0xc899c6 in net::posix_data_sink_impl::put(net::packet) net/posix-stack.cc:318
    #6 0x4faeee in data_sink::put(net::packet) core/iostream.hh:112
    #7 0x1023013 in seastar::tls::session::vec_push(giovec_t const*, int) net/tls.cc:771
    #8 0x10223e9 in seastar::tls::session::vec_push_wrapper(void*, giovec_t const*, int) net/tls.cc:699
    #9 0x7f14f10a6478  (/lib64/libgnutls.so.30+0x30478)
    #10 0x7f14f10a0f70  (/lib64/libgnutls.so.30+0x2af70)
    #11 0x7f14f10c3397 in gnutls_alert_send (/lib64/libgnutls.so.30+0x4d397)
    #12 0x7f14f10a2d48 in gnutls_bye (/lib64/libgnutls.so.30+0x2cd48)
    #13 0x10247cc in seastar::tls::session::shutdown(gnutls_close_request_t) net/tls.cc:850
...

[tgrabiec]

Fixed by 2494d4c

[tgrabiec]

Turns out that this wasn't fixed for the TLS sockets. Closing the TLS output stream will close the underlying socket's output stream (session::_out), but that one is used by both TLS sink and sources. tls_connected_socket_impl::source_impl::close() invokes session::shutdown_input(), which invokes shutdown(GNUTLS_SHUT_RDWR), which writes to session::_out.

Perhaps shutdown(GNUTLS_SHUT_RDWR) should only be performed when whole session is done (both input and output streams are done), and not when just the input stream is out of scope. And TLS sink shouldn't close session::_out in shutdown_output(). @elcallio thoughts?

[elcallio]

Do you mean that streams::close should not relate to socket_shutdown? If so, should that not be the case in normal tcp socket as well? And what is the result of stream close then? Just setting flags?
Note that our (current) contract is that the shutdown operations wake up a pending reader, which is also propagated through the stream close right now.

[tgrabiec]

2017-08-21 11:18 GMT+02:00 Calle Wilund <notifications@github.com>:

Do you mean that streams::close should not relate to socket_shutdown? If so, should that not be the case in normal tcp socket as well? And what is the result of stream close then? Just setting flags?

That's not what I wanted, at least not intentionally. The problem is that we want streams in both directions to be independent, so you can for example close the read side before you close the write side, or the other way around. Currently that is not the case for the TLS impl: 1) closing the write side closes the session output socket, which will be needed to close the read side (closing the read side writes to that socket). If you close the write side before you close the read side, then read side closing will fail, you won't be able to perform SHUTDOWN_RDWR. 2) if you close the read side before you close the write side, that issues SHUTDOWN_RDWR, which implicitly disallows further writes. The question is, can we rearrange the implementation of TLS socket so that the requirements of independence of the streams can be met? Or is TLS incompatible with this abstraction.

…

Note that our (current) contract is that the shutdown operations wake up a pending reader, which is also propagated through the stream close right now.

[tgrabiec]

To give more context here, I've noticed this while trying to add auto-close to streams (https://github.com/scylladb/seastar-dev/commit/e516f105ae31c47665c3de6eb3d4595ce7cf1a46), where a stream is closed automatically when goes out of scope unless explicitly closed earlier. This depends on streams being independent. Currently causes TLS tests to fail due to exception being thrown from input stream shutdown if it is closed after the output side is closed.

[elcallio]

Yes, the gnutls goodbye semantics are indeed incompatible with independent close.
The problem comes from trying to make well-behaved TLS "transparent". If we stop doing the goodbye handshake automatically, maybe putting them fully in the close-timer, we could make in/out indepentent, sort of, but then we again run into the problem that shutdown is supposed to wake up readers. I.e. a TLS shutdown_input is required to do an underlying socket shutdown (to wake up any reader). But after that of course, we cannot do a handshake at close time. So then that automagic niceness falls apart too. Having a wormholed wake-up could fix this, but ugly...

We could 100% abandon well-behaved-ness when operating transparently, and require a caller to be TLS aware and do a different termination w. handshake if he wants to adhere to procedure, and if not, just use socket level close. Problem with that is that currently all TLS is hidden in socket_impl layer.

[gleb-cloudius]

On Mon, Aug 21, 2017 at 02:18:31AM -0700, Calle Wilund wrote: Do you mean that streams::close should not relate to socket_shutdown? If so, should that not be the case in normal tcp socket as well? And what is the result of stream close then? Just setting flags? Note that our (current) contract is that the shutdown operations wake up a pending reader, which is also propagated through the stream close right now.

As I noted in scylladb/scylladb#2244 (comment) the commit already broke the contract for regular tcp.

…

-- Gleb.

[tgrabiec]

@elcallio What is the well-behaved way of using the streams of the TLS socket regarding life cycle of the input/output streams? Can I shutdown each side concurrently, do I have to do it in certain order?

[elcallio]

It should be ok to do from both ends, though only one of the goodbyes will be the actual goodbye (I think). But as stated earlier, it is probably "better" (outside seastar) to make an application aware of the handshake and build it into whatever state machine you run, i.e. the "client" side does the goodbye. Again, the whole bye-bye handshaking does not fully sit well with transparent TLS.

[elcallio]

FYI, another prominent case of TLS transparent layering, Java, does not allow streams to operate separately. I.e. closing either input or output stream handshakes and closes the entire connection. (Unless I read the code wrong...)

[nyh]

Turns out that this wasn't fixed for the TLS sockets. Closing the TLS output stream will close the underlying socket's output stream (session::_out), but that one is used by both TLS sink and sources. tls_connected_socket_impl::source_impl::close() invokes session::shutdown_input(), which invokes shutdown(GNUTLS_SHUT_RDWR), which writes to session::_out.

Perhaps shutdown(GNUTLS_SHUT_RDWR) should only be performed when whole session is done (both input and output streams are done), and not when just the input stream is out of scope. And TLS sink shouldn't close session::_out in shutdown_output(). @elcallio thoughts?

Wow, this issue was known for all these years, and I just rediscovered it now in #906. As I explain in #906, the fact that closing just one stream (input or output) immediately closes the entire connection not only is an inconsistent API (because Seastar's TCP sockets don't do this), it also breaks Seastar's HTTP server which assumes it can continue to write to the output stream even after closing the input stream.

[nyh]

Yes, the gnutls goodbye semantics are indeed incompatible with independent close.
The problem comes from trying to make well-behaved TLS "transparent". If we stop doing the goodbye handshake automatically, maybe putting them fully in the close-timer, we could make in/out indepentent, sort of, but then we again run into the problem that shutdown is supposed to wake up readers. I.e. a TLS shutdown_input is required to do an underlying socket shutdown (to wake up any reader).

I don't understand the problem here, maybe you can explain this again assuming I know nothing about gnutls and knew in the past - but forgot - the details of the TLS protocol.

Let me explain why I'm asking:

The Seastar "tradition" is that streams (and files, and almost anything else) need to be close()ed before destroyed. So an application using a TLS socket will, at some point, close() both the input and output stream. What I and I think also @tgrabiec have been suggesting is that the TLS implementation will not do anything until both streams are closed.
You are saying here that it is important that the input stream's close(), if called first, do something - you said "wake up readers". What readers are these? Why is this important for Seastar or its users that these "readers" be woken when just the input stream is closed, and it's not enough that it happens after both input and output streams are closed? What prevents us from waking up these readers (whatever that means...) without closing the TLS session so that the output stream would continue to work?

[elcallio]

This again is known issues, the close-all approach was done in 5832c7f, before this we did indeed try to do partial shutdowns, but this caused a number of issues with clients, esp. when doing transparent io. Partially because our general socket abstractions have some issues when it comes to enforcing proper state management (no real raii-behaviour for example).

As stated above, the current approach instead models on JDK behaviour of sockets/streams. Which in turn means that one can argue that the non-tls socket usage in seastar apps is borked and should be fixed instead.
Note that the seastar design with potentially multiple input/output streams per socket is also a bit of an issue here. (If we have two input streams, and one calls close() - is the socket read closed?)

The problem again is that to ensure proper TLS handshaking when actually disconnecting (be it shutdown_or or shutdown_rdwr) is that both sides need to ack the other. (Initiator can skip acking - but if you have two sides both shutting down write for example, they need to ack each other.
Now, since we can't shutdown reading, the only approaches are to either a.) do full rdwr shutdown in this case (old code), or b.) handle this "soft", by simply refusing to return any read bytes to client layer. The latter sounds great - but there are nasty issues, such that shutdown(SHUT_RD) expecting already buffered data to remain readable (not sure if this is the case with seastar).
If the underlying medium is a posix pipe, we are also expected to cause errors on the writer as soon as SHUT_RD like is done by reader side, i.e. it should be visible to writing end - not just silently buffering more data. Not sure if this matters.
But the bottom line is that we cannot actually shut down things on a wire level.

And to relate to your question: It is quite vital IIRC that a shutdown(WR) cause reactable side effects on the reading side, i.e. wakes things up, otherwise things like rpc etc will deadlock. The only way to ensure this via TLS is via a shutdown message. And I can't 100% remember if shutdown_wr is enough.

[nyh]

before this we did indeed try to do partial shutdowns, but this caused a number of issues with clients, esp. when doing transparent io.

What is "transparent io"? I saw you used this phrase a few times, but didn't understand the context.

As stated above, the current approach instead models on JDK behaviour of sockets/streams. Which in turn means that one can argue that the non-tls socket usage in seastar apps is borked and should be fixed instead.

Right. The main issue is that normal (TCP) sockets and TLS sockets behave differently, so code written and heavily tested for TCP (including Seastar's own HTTP server, see #906, but potentially other Seastar-based applications as well) and then changed to use TLS will stop working correctly.
It just doesn't make sense to have TLS sockets behave one way and TCP sockets behave a different way, given that you know that most applications - including our HTTP server - will want exactly the same application code to work on both TLS and TCP sockets.

Note that the seastar design with potentially multiple input/output streams per socket is also a bit of an issue here. (If we have two input streams, and one calls close() - is the socket read closed?)

I don't know if we allow two input streams (what if two fibers read from these two input streams?), but if we do, the solution is simple - use a reference count. The connection will be closed when all the references - both writes and readers - are gone.

The problem again is that to ensure proper TLS handshaking when actually disconnecting (be it shutdown_or or shutdown_rdwr) is that both sides need to ack the other. (Initiator can skip acking - but if you have two sides both shutting down write for example, they need to ack each other.
Now, since we can't shutdown reading, the only approaches are to either a.) do full rdwr shutdown in this case (old code), or b.) handle this "soft", by simply refusing to return any read bytes to client layer.

After the application closed the (last?) input stream, and you don't allow calling the function to get the input stream again (you can throw an exception in this case), there is no way that the application can read more - it won't have any way to do it. The low level implementation may read and discard data if it does arrive, I guess.

The latter sounds great - but there are nasty issues, such that shutdown(SHUT_RD) expecting already buffered data to remain readable (not sure if this is the case with seastar).

This is not relevant to the Seastar API - if the user calls close() on the input-stream it got, it can't use it any more. Seastar's close() is a kind of futurized destructor that C++ is missing.

If the underlying medium is a posix pipe, we are also expected to cause errors on the writer as soon as SHUT_RD like is done by reader side, i.e. it should be visible to writing end - not just silently buffering more data. Not sure if this matters.

I agree, but I am not sure it's relevant to Seastar's API. I don't think (but maybe @tgrabiec or @avikivity can corrrect me) that we ever suggested that close()ing an input stream of a socket guarantees that the writer will know this when it tries to send.

And to relate to your question: It is quite vital IIRC that a shutdown(WR) cause reactable side effects on the reading side, i.e. wakes things up, otherwise things like rpc etc will deadlock. The only way to ensure this via TLS is via a shutdown message. And I can't 100% remember if shutdown_wr is enough.

This appears to be the heart of the matter. When you "shutdown(WR)" (I assume you mean close the output stream), it appears you want to wake up the remote side. But since with the current code, at that point it can't send you a reply (you already closed the TLS session!), you have no use for the input stream as well. So in the existing code, if RPC want to close the connection and alert the remote side, it can close both input and output streams. If it does this, then under my suggestion the TLS connection to be closed and the remote side to be woken like you wanted. What am I missing?

[elcallio]

before this we did indeed try to do partial shutdowns, but this caused a number of issues with clients, esp. when doing transparent io.

What is "transparent io"? I saw you used this phrase a few times, but didn't understand the context.

Using a TLS socket as if it was any other socket, i.e. abstracted away.

As stated above, the current approach instead models on JDK behaviour of sockets/streams. Which in turn means that one can argue that the non-tls socket usage in seastar apps is borked and should be fixed instead.

Right. The main issue is that normal (TCP) sockets and TLS sockets behave differently, so code written and heavily tested for TCP (including Seastar's own HTTP server, see #906, but potentially other Seastar-based applications as well) and then changed to use TLS will stop working correctly.
It just doesn't make sense to have TLS sockets behave one way and TCP sockets behave a different way, given that you know that most applications - including our HTTP server - will want exactly the same application code to work on both TLS and TCP sockets.

Which typically means one has to avoid things like closing input before being done with output. You can just as well argue that our non-tls sockets allow to many things.

Note that the seastar design with potentially multiple input/output streams per socket is also a bit of an issue here. (If we have two input streams, and one calls close() - is the socket read closed?)

I don't know if we allow two input streams (what if two fibers read from these two input streams?), but if we do, the solution is simple - use a reference count. The connection will be closed when all the references - both writes and readers - are gone.

But what if client does not call close? Which is very well known to happen. Of course, can be handled with extra statefulness in source/sink, but it also needs to be backed in session.

I would argue that enforcing that a client keeps input alive, and closes output first is not that hard a condition for transparent (I used it again!) socket usage.

The problem again is that to ensure proper TLS handshaking when actually disconnecting (be it shutdown_or or shutdown_rdwr) is that both sides need to ack the other. (Initiator can skip acking - but if you have two sides both shutting down write for example, they need to ack each other.
Now, since we can't shutdown reading, the only approaches are to either a.) do full rdwr shutdown in this case (old code), or b.) handle this "soft", by simply refusing to return any read bytes to client layer.

After the application closed the (last?) input stream, and you don't allow calling the function to get the input stream again (you can throw an exception in this case), there is no way that the application can read more - it won't have any way to do it. The low level implementation may read and discard data if it does arrive, I guess.

The latter sounds great - but there are nasty issues, such that shutdown(SHUT_RD) expecting already buffered data to remain readable (not sure if this is the case with seastar).

This is not relevant to the Seastar API - if the user calls close() on the input-stream it got, it can't use it any more. Seastar's close() is a kind of futurized destructor that C++ is missing.

There is nothing enforcing this in streams.

If the underlying medium is a posix pipe, we are also expected to cause errors on the writer as soon as SHUT_RD like is done by reader side, i.e. it should be visible to writing end - not just silently buffering more data. Not sure if this matters.

I agree, but I am not sure it's relevant to Seastar's API. I don't think (but maybe @tgrabiec or @avikivity can corrrect me) that we ever suggested that close()ing an input stream of a socket guarantees that the writer will know this when it tries to send.

Guarantee and "happen to rely on it" are two different things.

And to relate to your question: It is quite vital IIRC that a shutdown(WR) cause reactable side effects on the reading side, i.e. wakes things up, otherwise things like rpc etc will deadlock. The only way to ensure this via TLS is via a shutdown message. And I can't 100% remember if shutdown_wr is enough.

This appears to be the heart of the matter. When you "shutdown(WR)" (I assume you mean close the output stream), it appears you want to wake up the remote side. But since with the current code, at that point it can't send you a reply (you already closed the TLS session!), you have no use for the input stream as well. So in the existing code, if RPC want to close the connection and alert the remote side, it can close both input and output streams. If it does this, then under my suggestion the TLS connection to be closed and the remote side to be woken like you wanted. What am I missing?

IIRC, said close of everything did/does not happen. Just like you want TLS behaviour to change so it fits into one set of "legacy" code, the current mode exists to large part to ensure it does work with other existing "legacy" code.

[nyh]

Right. The main issue is that normal (TCP) sockets and TLS sockets behave differently, so code written and heavily tested for TCP (including Seastar's own HTTP server, see #906, but potentially other Seastar-based applications as well) and then changed to use TLS will stop working correctly.
It just doesn't make sense to have TLS sockets behave one way and TCP sockets behave a different way, given that you know that most applications - including our HTTP server - will want exactly the same application code to work on both TLS and TCP sockets.

Which typically means one has to avoid things like closing input before being done with output. You can just as well argue that our non-tls sockets allow to many things.

I agree that the problem that the non-TLS and TLS sockets differ doesn't immediately say that the TLS one needs to be fixed and not the other way around. However, what I am suggesting is that it is likely that Seastar applications are developed and tested with non-TLS first, and then break when modified to use TLS, and not the other way around. Furthermore, I resurrected this discussion because Seastar's HTTP server had exactly the same problem - it was developed and tested mostly for non-TLS, and it turns out it doesn't work correctly (in a specific scenario) for TLS because of this difference. Yes, it can be modified to work for both, but wouldn't it be nice if both just had the same interface so it wouldn't need to be modified in the first place?

But what if client does not call close? Which is very well known to happen. Of course, can be handled with extra statefulness in source/sink, but it also needs to be backed in session.

We could decrease the reference count in the destructor, not close(), and then close() would just check if it's the last reference, and if it is, wait for the session destruction. Is this good enough for your use case?

Anyway, it should be documented whether or not it is necessary to close() these streams. You are suggesting now it is fine to not close() both streams, and close just one. But it is also fine not to close any of them? If that's not fine, why is it fine to close just one? Isn't that weird?

I would argue that enforcing that a client keeps input alive, and closes output first is not that hard a condition for transparent (I used it again!) socket usage.

It's not hard, but it's completely arbitrary, and as a matter of fact, whoever wrote the HTTP server code (not me...) didn't know this condition existed. And why didn't he know this condition existed? Because it doesn't in the non-TLS case. Which makes this condition non-transparent, instead of transparent.

I added in #906 a comment on how I can indeed work around this condition - close both streams together and this will work on both TLS and non-TLS sockets. But this will just solve the HTTP bug and leave us with the inconsistent API that can bite other users in the future.

This is not relevant to the Seastar API - if the user calls close() on the input-stream it got, it can't use it any more. Seastar's close() is a kind of futurized destructor that C++ is missing.

There is nothing enforcing this in streams.

As usual, we have no useful documentation saying what close() is supposed to mean on a stream.
For a file stream, close() will call the file's close() - you can't use it again after you call it.
I don't know how our TCP sockets behave... Can you close an input stream and then read from it again, or open a new one?

This appears to be the heart of the matter. When you "shutdown(WR)" (I assume you mean close the output stream), it appears you want to wake up the remote side. But since with the current code, at that point it can't send you a reply (you already closed the TLS session!), you have no use for the input stream as well. So in the existing code, if RPC want to close the connection and alert the remote side, it can close both input and output streams. If it does this, then under my suggestion the TLS connection to be closed and the remote side to be woken like you wanted. What am I missing?

IIRC, said close of everything did/does not happen. Just like you want TLS behaviour to change so it fits into one set of "legacy" code, the current mode exists to large part to ensure it does work with other existing "legacy" code.

I think I see now - you're saying that the legacy Seastar-using code doesn't close() both streams - just one, and should continue to work unmodified.
Do you know if the other, non-closed stream, is at least destroyed before the close()ed one? If the answer is yes, then my solution (reduce the reference count in the destructor) will fit this legacy user as well.

[elcallio]

Right. The main issue is that normal (TCP) sockets and TLS sockets behave differently, so code written and heavily tested for TCP (including Seastar's own HTTP server, see #906, but potentially other Seastar-based applications as well) and then changed to use TLS will stop working correctly.
It just doesn't make sense to have TLS sockets behave one way and TCP sockets behave a different way, given that you know that most applications - including our HTTP server - will want exactly the same application code to work on both TLS and TCP sockets.

Which typically means one has to avoid things like closing input before being done with output. You can just as well argue that our non-tls sockets allow to many things.

I agree that the problem that the non-TLS and TLS sockets differ doesn't immediately say that the TLS one needs to be fixed and not the other way around. However, what I am suggesting is that it is likely that Seastar applications are developed and tested with non-TLS first, and then break when modified to use TLS, and not the other way around. Furthermore, I resurrected this discussion because Seastar's HTTP server had exactly the same problem - it was developed and tested mostly for non-TLS, and it turns out it doesn't work correctly (in a specific scenario) for TLS because of this difference. Yes, it can be modified to work for both, but wouldn't it be nice if both just had the same interface so it wouldn't need to be modified in the first place?

Sure, but I am not certain that it can ever 100% happen. I think there are pitfalls where trying to ensure that the TLS state machine is executed as it should while allowing all the permutations of semi-allowed socket usage you have with non-TLS sockets will fail eventually and in places they remain hidden until they are at a customers and things go escalate. Because that is what traditionally happened.

But what if client does not call close? Which is very well known to happen. Of course, can be handled with extra statefulness in source/sink, but it also needs to be backed in session.

We could decrease the reference count in the destructor, not close(), and then close() would just check if it's the last reference, and if it is, wait for the session destruction. Is this good enough for your use case?

Yes, I am aware of how you can do it. My point is the added statefulness, and that stateful things always break. Always.

Anyway, it should be documented whether or not it is necessary to close() these streams. You are suggesting now it is fine to not close() both streams, and close just one. But it is also fine not to close any of them? If that's not fine, why is it fine to close just one? Isn't that weird?

The problem is that it is ok for "normal" sockets. And has/does happened over and over. Our posix/native IP stack is quite forgiving w/r to violating the "explicit destructors" just as it is forgiving visavi order of closing things. So we have had code that does things haphazardly.

I would argue that enforcing that a client keeps input alive, and closes output first is not that hard a condition for transparent (I used it again!) socket usage.

It's not hard, but it's completely arbitrary, and as a matter of fact, whoever wrote the HTTP server code (not me...) didn't know this condition existed. And why didn't he know this condition existed? Because it doesn't in the non-TLS case. Which makes this condition non-transparent, instead of transparent.

But my point is again that the non-TLS case is a bit to forgiving. The http server was initially written way before the TLS layer existed, so it can be partially forgiven for taking liberties.

I added in #906 a comment on how I can indeed work around this condition - close both streams together and this will work on both TLS and non-TLS sockets. But this will just solve the HTTP bug and leave us with the inconsistent API that can bite other users in the future.

This is not relevant to the Seastar API - if the user calls close() on the input-stream it got, it can't use it any more. Seastar's close() is a kind of futurized destructor that C++ is missing.

There is nothing enforcing this in streams.

As usual, we have no useful documentation saying what close() is supposed to mean on a stream.
For a file stream, close() will call the file's close() - you can't use it again after you call it.
I don't know how our TCP sockets behave... Can you close an input stream and then read from it again, or open a new one?

Socket streams will defer to the source close, which will typically close the actual socket in the proper direction (i.e. shutdown RD). So in this case, closing one input stream will close any other as well.
Funny thing: The native stack will close_write() when you close the source (input stream), which I assume, but cannot be sure, is a bug.
But we have other (non-socket) underlying sink/source implementations that behave quite differently. Technically, TLS is required to work "the same" when wrapped around one of these.

This appears to be the heart of the matter. When you "shutdown(WR)" (I assume you mean close the output stream), it appears you want to wake up the remote side. But since with the current code, at that point it can't send you a reply (you already closed the TLS session!), you have no use for the input stream as well. So in the existing code, if RPC want to close the connection and alert the remote side, it can close both input and output streams. If it does this, then under my suggestion the TLS connection to be closed and the remote side to be woken like you wanted. What am I missing?

IIRC, said close of everything did/does not happen. Just like you want TLS behaviour to change so it fits into one set of "legacy" code, the current mode exists to large part to ensure it does work with other existing "legacy" code.

I think I see now - you're saying that the legacy Seastar-using code doesn't close() both streams - just one, and should continue to work unmodified.
Do you know if the other, non-closed stream, is at least destroyed before the close()ed one? If the answer is yes, then my solution (reduce the reference count in the destructor) will fit this legacy user as well.

I cant recall right now where the various issues where. But doing things in destructor is IIRC not good enough in some cases and will cause problems. I really hope I am wrong about this, because honestly I would love for the state machine to just be automatic and super forgiving like you suggest. I am just pointing out that things are like they are for very real and terrible reasons. And poking them might have side effects that does not show up until they are in data centers somewhere and causing angry bug reports.

[elcallio]

Btw, one more really terrible scenario: What if a client gets input stream, reads some, then closes it. There are no references to writers, so we would ideally want to do a full shutdown handshake now (avoiding having background tasks running). But then it would be impossible to after this get the output stream and write, something that a "normal" socket usually allows fine.
So then our only option is to do actual close on RAII-death, but resurrect ourselves and run in background task for handshake.
Granted, we more or less do that today already, except it is at least initiated by close() call, not by instance life time. Not sure if it matters, but again, the need for handshaking makes TLS inherently different from "normal" tcp/ip streams.