OpenSSL QUIC usage in curl

[icing]

With curl/curl#12734 we have an experimental PR that adds HTTP/3 support to curl using the OpenSSL's own QUIC implementation. It has shortcomings which may be the result of us missing features in the API or them being absent. I open this discussion to find out which is which and how we can improve our use of the API or where the API needs improvement.

1. Requests without BODY

Update: this is resolved with OpenSSL 3.3.x and curl 8.7.1.

When sending a HTTP request without BODY (commonly a GET), nghttp3 gives us the data to be written to QUIC, plus the indication of FIN. This we translate into (roughly):

SSL_write_ex(s->ssl, buf, blen,  &written);
...
if(fin)
  SSL_stream_conclude(s->ssl, 0);

This seems to result in QUIC STREAM packets with the data and a final empty STREAM with the FIN set. This is sub-optimal, as server implementations may start processing the request before seeing the last STREAM+FIN, thus needing to check for a request body. HTTP/3 gateways to a HTTP/1.1 may have to add Transfer-Encoding: chunked to such a request. This is sub-optimal for GET requests that make a majority of the traffic.

Question: is there a way to write data to the SSL* together with the FIN?

2. Send Blocking

When doing large uploads, the stream/connection window will get exhausted, and QUIC waits for server ACKs to make progress. This is detected by writes to a stream SSL* returning SSL_ERROR_WANT_WRITE. We then block the nghttp3 stream for futher processing.

What we lack in curl is the indication when to correctly unblock that stream processing again. For that we currently use SSL_want_write(stream->ssl). When that returns FALSE, we unblock. However, this returns FALSE much too early, unblocking the stream only to immediately giving SSL_ERROR_WANT_WRITE again. This leads to curl using 100% cpu on a single large upload that is merely under flow control.

Question: how can we manage this situation better?

Update: we use SSL_net_write_desired(connssl) to decide the POLLOUT status. A single, blocked stream seems to result in us keep on triggering writes.

3. Idle Timeout

Update: this is resolved with OpenSSL 3.3.x and curl 8.7.1.

In curl, we have a configurable idle time for a connection until it is considered stale and will be discarded. In other QUIC stacks we set this in the transport parameters so that the peer learns about it. We also inspect the remote transport parameter to learn what the peer's idle time is. The minimum of both then is the effective time we use in our connection pools.

Question: is there a way to set the local and query the remote idle time?

4. Left Bidi streams

Update: this is resolved with OpenSSL 3.3.x and curl 8.7.1.

QUIC negotiates and updates the max number of bidi streams a peer may open. This is important to know for curl in order to decide when to use an existing connection, open a new one, or delay starting the transfer. Since the number of streams currently open is an internal state of the QUIC stack and depends on various things, curl needs an up-to-date indication for this. For example, the quiche implementation has a quiche_conn_peer_streams_left_bidi(conn) that gives us this.

Question: is there a facility in OpenSSL's QUIC to get that information?

5. Efficient Stream Receives

When curl polls the QUIC socket for POLLIN and then invokes receiving and processing the data, we have found no indications or callbacks that inform us which QUIC streams need action afterwards. Our current implementation therefore needs to SSL_read_ex(stream->ssl) on all open streams of a connection, every time we suspect that QUIC packets from the server have arrived. We have curl applications that do hundreds of transfers in parallel, depending on the peer's capabilities many on the same QUIC connection.

Question: can curl somehow learn if QUIC stream need reads and, ideally, which streams need read? Or install a callback that is invoked with the stream id and either the data or an event identifier?

Thanks for your support.

[t8m]

I'll answer some of the questions leaving the rest for @hlandau.

Re (1): This is definitely something that is missing in our API and should be added. We could either add SSL_write_ex2() with a flag parameter or we could add a flag to be used with SSL_stream_conclude() such as SSL_STREAM_CONCLUDE_FLAG_ON_NEXT_WRITE. This should be a fairly simple API addition.

Re (5): Polling will be added with the server support at the latest. Please see this #21795 PR for the design document.

[t8m]

BTW thank you very much for this feedback, it is much appreciated.

[icing]

Thanks @t8m.

Re (1): seems a good way forward. Personally, I'd prefer the SSL_write_ex2().

Re (5): curl needs and does its own polling. It may be even done by the libcurl application using event frameworks. We therefore would not be a user of any poll/select whatever by OpenSSL itself. We are interested in a correct indication when the QUIC connection wants to send. I assume that SSL_net_write_desired() was made for this, but it does not seem to behave correctly (as we interpret it) on a blocked connection where it just needs to receive ACKs to progress.

[hlandau]

Thanks for the feedback. It is greatly appreciated and invaluable to our future efforts.

Answering your questions out of order:

1. Question: is there a way to write data to the SSL* together with the FIN?

• Not currently. This was viewed as an optimisation and thus not prioritised for our MVP release in 3.2. We definitely want to provide an API for optimised FIN in future, probably some kind of new SSL_write-like API which allows combining the write and a FIN.

5. Question: can curl somehow learn if QUIC stream need reads and, ideally, which streams need read? Or install a callback that is invoked with the stream id and either the data or an event identifier?

• This is a known gap in our API currently. We have plans to provide a polling API (similar to poll or epoll but in terms of SSL objects) which allows an application to learn what streams are ready. The proposed polling API can be found here — any feedback on the proposed design would be greatly appreciated. There is a lot to cover there, so we may start out only implementing a subset of this and growing from there.
• To be clear: It is absolutely and always our intention to provide an implementation where the application is able to control all network I/O and do its own polling. The polling API proposal above refers to future APIs for taking a set of stream objects (for example) and calling some libssl API call to learn which are ready to do work. (The API will also support polling to the OS as well for those that want a turnkey solution there, but that would be strictly optional.)
• The SSL_pending call does provide information on how much read data is buffered on a given stream, which may be of at least some use. Of course this is still suboptimal, but may be worth using for now.

2. Question: how can we manage this situation better?

• This is also a problem caused by the fact that we don't have a polling API yet, as discussed above.
  Using SSL_net_write_desired isn't the right solution, because it refers to whether QUIC packets are pending transmission to the network. So if writing to a stream is blocked by (e.g.) congestion or flow control, writing can be blocked even if this is not true.
• A stopgap solution is that we should probably provide at least the write equivalent of SSL_pending which allows the amount of available room in the write buffer to be determined. This would be a trivial feature addition.

3. Question: is there a way to set the local and query the remote idle time?

• Not currently — it is managed internally and there are no options to configure this. This is absolutely something we can offer in a future release and would be a simple addition. Note that our current implementation always negotiates an idle timeout of min(30s, peer requested idle timeout) — if that information is of any use.

4. Question: is there a facility in OpenSSL's QUIC to get that information?

• Can you clarify what you mean by "open"? Or rather, are you seeking an API to retrieve the number of new streams that can currently be created based on the MAX_STREAMS flow control budget? The latter would be quite simple for us to provide in a future API. If you are instead seeking information on the number of streams currently in existence in some state or another, that also does not seem hard to do — but would need clarification on the meaning of "open" — refer to the stream states in RFC 9000 s. 3. What states are you trying to learn about?

Thanks for your feedback and your work on integrating this into curl. This is invaluable for us and will definitely inform our future development efforts. Don't hesitate to drop us a line if you have other questions.

[hlandau]

PR #23360 has been raised to solve (3) and (4) — feedback welcomed

This is in addition to #23343 to provide optimised FIN in response to (1)

[icing]

Both look good. Looking forward to integrate use of that into curl!

[hlandau]

#23343 has now merged and is expected to be released in OpenSSL 3.3.

[icing]

excuse my ignorance, but when is 3.3 approximately going to happen?

[mattcaswell]

April

[hlandau]

@icing So, to reiterate the current status of things:

• A solution to (1) has merged and will be released in 3.3, which will happen in April 2024.
• A solution to (3) and (4) has a PR QUIC: Idle timeout configuration & stream count query APIs #23360 up awaiting review. Of course, any feedback on this PR or testing of it would be welcomed.

With regards to (2), it seems to me like there's a bit more going on here than I thought. You say that the CPU is getting pegged because SSL_want_write() is returning false too early. But of course, the CPU needs to be executing code to call SSL_want_write() in the first place; so I guess my question is how you're avoiding pegging the CPU in other cases where SSL_want_write() is true.

The basic idea here for an asynchronous application is to use SSL_net_write_desired(), SSL_net_read_desired() and SSL_get_event_timeout() to determine when the QUIC engine wants to do internal processing:

• SSL_net_write_desired() means some QUIC packets have been generated but have yet to be successfully sent (i.e., yet to be successfully passed to the kernel). Since the rate at which QUIC packets generated is obviously modulated by congestion control, this should not be true for anything but the briefest period.

• SSL_net_read_desired() means the QUIC engine is interested in incoming packets. This is basically always true unless the connection hasn't been initiated yet or the connection has been closed and finished going through the terminating states (RFC 9000 s. 10.2 closing/draining).

• SSL_get_event_timeout() determines when the QUIC engine next wants to handle some kind of timeout.

When any of these things happen, there is useful work for the QUIC engine to do and that internal processing might cause a situation where a stream which blocked on SSL_write() can now accept more data.

I assume you're already plugging whatever asynchronous I/O event handling system you have into SSL_net_read_desired(), SSL_net_write_desired() and SSL_get_event_timeout(), which is the idea. So the simple thing to do here is, when one of those events occurs, is call SSL_write() and see if it works, and
if not, go back to sleep. So it seems like a "SSL_write_buffer_space_available()" call, as it were, would only make sense if there is some kind of higher computational cost in an application getting the write buffer to pass to SSL_write() in case it is capable of doing work. Is this the case for your application? Even if that is the case, it seems like in most cases an application could just stash its write buffer and retry subsequent SSL_write() calls with the same buffer it used last time. Can you clarify these points on how your application is using the API?

As an aside - the 'want write' API is a little bit gnarly and this is a reflection of the existing libssl API here. SSL_want_write() is just a front for SSL_get_error(). SSL_get_error() is supposed to be updated with the result of the last API call, if

• the API call was an "I/O" (application datapath) call, like SSL_read or SSL_write;

• the API call was not an I/O call, but failed.

In other words, the current value of SSL_want_write() gets clobbered by making e.g. a SSL_read() call and vice versa. But other calls like SSL_get_stream_read_state() are "non-I/O" and only clobber the value of SSL_get_error()/SSL_want_read()/SSL_want_write() if they fail. Basically these APIs are only really intended to be used for retrieving information about the last API call immediately after it is made. Admittedly this subtlety might be quite confusing, especially the fact that some API calls modify SSL_get_error() and some don't. This is part of our existing TLS API, so we've tried to preserve the existing semantics here.

As always looking forward to your input.

[icing]

@hlandau, thanks for your explanations. I inspected our upload handling once again and try to explain our problems in using the existing API in more detail.

We use SSL_net_write_desired(), SSL_net_write_desired() and SSL_get_event_timeout() to populate our pollsets and timeout for the connection. When any of these triggers, we invoke our transfers' send and recv methods. This works well for downloads, as new data from the server will always give us a reliable POLLIN event.

The problem is with uploads. On what condition do we start to send the remaining data of a request body?

• POLLOUT is rarely a good trigger. It will always almost happen, should we ask for it, unless the UDP sending is really EAGAINed by the OS, which will be rare. Consequently, as you also describe, SSL_net_write_desired() will rarely return TRUE.
• This leaves us with POLLIN and event timeout to steer an upload. What happens, if we only rely on these, is a stall. The scenario is as follows:
  • while forwarding all of nghttp3 streams data to its SSL*, we get a SSL_ERROR_WANT_WRITE on one write, indicating that this stream is flow control blocked.
  • we call SSL_get_event_timeout() before returning from our send, meaning we rely on (POLLIN, timeout) to be called again.
  • often SSL_get_event_timeout() will tell us to run again in 20-24ms. I assume this is the timer waiting for server ACKs of the packets in flight. But sometimes, the timeout given to us is 15 seconds. This is, I assume again, the connection IDLE timeout negotiated.
  • our upload stalls. I assume that some call into OpenSSL triggered network IO, received the ACKs it was waiting for, and is totally fine and happy with the situation.

What we need is a trigger that the ACKs have been received and the flow control window has been opened again.

What our implementation does now, to prevent the stallings, is to check on SSL_want_write(streamssl) to unblock the nghttp3 processing and trigger our send/recv again. This is, as you described, not the way to do it since SSL_want_write() is not designed to give us the answer we need. It will almost always return 0, this will trigger our send/recv over and over again, avoiding the stalling but consuming 100% cpu.

To summarize:

1. SSL_write_ex() to a QUIC stream blocks
2. SSL_get_event_timeout() returns 20-24ms and sometimes 15 seconds
3. when do we unblock our sending of a blocked stream? (there might be 100 streams or more on the same connection)

[t8m]

I think this asks for being able to query the current stream and connection data limits.

[icing]

I do not think that is feasible. If some SSL call side-effects into IO, not noticed by the application, the status may change and we go into poll with the IDLE timeout.

[hlandau]

After SSL_get_event_timeout() expires, the idea is you call SSL_handle_events() either directly (or implicitly by making another API call such as SSL_read/SSL_write, etc.). This is when the state of the QUIC engine is updated. So it seems like logically you can just probe for if an SSL_write_ex call succeeds after handling such events. What I'm a little confused by is why this would lead to pegging the CPU. The intention behind the design is something like (pseudocode):

for (;;) {
  poll.fd = network_socket;
  poll.events = 0;
  poll.events |= SSL_net_read_desired() ? POLLIN : 0;
  poll.events |= SSL_net_write_desired() ? POLLOUT : 0;
  poll(&poll, SSL_get_event_timeout());
  if (poll.revents) SSL_handle_events();

  // state of QUIC engine has been updated, try to make progress
  ret = SSL_write_ex(...);
  if (!ret && SSL_get_error(..., ret) == SSL_ERROR_WANT_WRITE)
    continue; // go back to sleep until we can make progress
}

This should not be causing 100% CPU usage. Did you find that this approach caused 100% CPU usage? If so it is probably a bug. I may be able to spend some time investigating this if you can point me to a branch which exhibits the 100% CPU usage.

[icing]

The problem creeps in when you do another SSL call after the SSL_write_ex(). That one may trigger SSL_handle_events() internally (or something equivalent) that receives packets from the peer and changes the stream state.

The simplest way this happens is when you have 2 streams to write. When the first one returns SSL_ERROR_WANT_WRITE you assume it is blocked. Then you write the second, which may unblock the first, but curl does not know. Then we may stall.

What pegs the cpu now is that we try to detect if a stream is still being blocked before entering polling. This wrongly uses the SSL_want_write() and always unblocks. Which leads then to write attempts over and over again. But, as pointed out above, if we do not have the exact stream state before polling, we will stall eventually.

I hope this makes sense to you. We need a way to know the stream state before entering polling, no matter what SSL calls have been made before. The SSL_ERROR_WANT_WRITE is insufficient. For example, ngtcp2 has a callback when a stream's flow window is being increased. That we use there to unblock a stream again.

[bagder]

Another detail: BIO_new_dgram() takes the socket as an int, but on 64 bit Windows socket types are 64 bit...

curl/curl#12865

[hlandau]

Yeah. This is some unfortunate technical debt we have where OpenSSL throughout its codebase uses int for sockets. This has been the case for a long time and isn't constrained to one API. Fixing this would be desirable but the changes would be quite widespread, so considering our API compatibility requirements it would be hard.

In practice it does work. What seems to be going on here is that while on Windows SOCKET values are nominally NT handles, and NT handles are pointer-sized opaque values... it would appear that in practice Windows never issues SOCKET handles which are not truncatable to 32 bits. My guess is that Microsoft determined that there are so many applications out there using the wrong type for socket handles that when they added 64-bit support they determined that compatibility effectively required issuing 32-bit-truncatable socket handles in practice. (Similarly we use -1 all over the place for "no socket", which by pure coincidence happens to equal a sign-truncated version of Microsoft's INVALID_HANDLE_VALUE.)

So in practice, things work — but yes, it is horrible and definitely not an example of how things should be done.

Sorry for the delay in getting back to you on the other outstanding issue in this thread — I've been kept busy, including with some other development items that have already come out of this thread. I have some ideas for possible resolutions to the issue you're describing and will definitely get back to you.

[FdaSilvaYY]

Yeah. This is some unfortunate technical debt we have where OpenSSL throughout its codebase uses int for sockets. This has been the case for a long time and isn't constrained to one API. Fixing this would be desirable but the changes would be quite widespread, so considering our API compatibility requirements it would be hard.

There is a comment block in the code saying that this practice is acceptable and has no harm, even if not officially documented.

See socket.h

[bagder]

But it does cause harm. It makes users of this API everywhere forced to implement weird work-arounds and tricks to silence compilers and code analyzers that rightfully complain. Even if the code runs fine in the end.

[hlandau]

@icing

Statement of the problem (write blocking)

This is a summary of the problem as I understand it, which is a real issue:

• We internally may perform QUIC processing during any SSL I/O API call. This means that the states of a stream may change during any I/O API call, even if that call relates to a different stream.

• This essentially creates a “whack-a-mole” situation where you only have reliable information on the blocking status of the last API call:

                                                Application View
                                                ====================
                                                Stream A   Stream B
                                                ---------  ---------
    write(streamA) → SSL_ERROR_WANT_WRITE
                                                Blocked    ?
    write(streamB) → SSL_ERROR_WANT_WRITE
                                                ?          Blocked
    write(streamA) → SSL_ERROR_WANT_WRITE
                                                Blocked    ?
  

  The first call gives you a reliable indication that stream A is blocked, but once you try to confirm if stream B is blocked, you obtain reliable information on stream B but the status of stream A is now undefined, because internal processing could have unblocked it. As such, you cannot get accurate information on the state of all streams at a finite point in time.

• Faced with this incomplete information, your only choice is to either

  a) hang, or at least experience a long delay on the order of seconds, as in this example:

    for (;;) {
        poll(...);
        SSL_handle_events(conn);
  
        SSL_write(streamA) → SSL_ERROR_WANT_WRITE;
        SSL_write(streamB) → SSL_ERROR_WANT_WRITE; // Suppose stream A is now silently unblocked
        // Loop reenters poll and hangs (at least until more network traffic
        // occurs, which may take some time, e.g. PTO), when in actual fact
        // it is now possible to write to stream B.
    }

  b) peg the CPU by constantly calling SSL_write on every stream just in case it has silently unblocked.

Proposed solution

There are two solutions I propose to provide various options here:

• The polling lite API, which provides a non-blocking way to get a point-in-time view of the readiness an of arbitrary number of objects. The PR for this is now up: QUIC Polling Lite #23495

• Autotick control. This has been planned for some time but should now be given greater priority.

  Basically, there will be a boolean “disable implicit event processing” option which you can turn on for a QUIC connection. When you turn this on, only SSL_handle_events will do internal processing. So things like SSL_read and SSL_write will copy the stream buffer but won't actually do other QUIC processing, thus avoiding “silent” changes to stream states except at the explicit point where SSL_handle_events() is called. This places a greater burden on the application to ensure that it does call this in a timely manner, but of course sophisticated applications like curl will probably prefer having this control anyway.

The design of potential additional APIs which offer callbacks as an alternative to polling is also still under investigation and I will keep you informed. But would these two things be a start to giving you some viable options here?

Let me know what you think — all of this is based on the requirements you've articulated. As mentioned, this input has been really valuable — thanks again.

[icing]

@hlandau thanks for the reply. As to your question: no, I do not see the autotick function being useful in our case.

We want to progress a particular stream when an internal event for that stream triggers. If the "tick" is triggered implicit as part of an SSL_write() call or explicit does not make a difference for our problem in managing changes to stream states.

[hlandau]

Essentially, you'd call SSL_handle_events yourself at some central point and immediately afterwards either a) use SSL_poll or b) try to service the various stream objects.

a) is suboptimal to e.g. kqueue-style solutions as discussed above, and b) is suboptimal to (a), but both of these are better than "you can either hang sometimes, or peg the CPU".

But yes, better to use SSL_poll when it is available. In any case I will probably ship autotick control as I believe people should have this kind of control if they want it.

I'm still researching possible callback API designs as an option and will keep you informed.

[icing]

There is no "at some central point" in libcurl and we do not plan to introduce it. In regard to

IMO the performance issues intrinsic to poll()-like designs are less of a concern than they are for server-side applications

that does not align with reality for our power users. I would discourage making such assumptions when designing OpenSSL's API for the future. You are free to design the API any way you want, of course, but we try to give you feedback from one of the most common (and powerful) client side applications you'll probably encounter.

We have deployments where poll/select like scheduling on TCP based transfers is not good enough and better scaling event frameworks are in use. To work well in such environments, we will not iterate frequently over all active streams to check for activity or want to collect them all for a POLL-like api call into OpenSSL.

We will adapt our OpenSSL QUIC adaptation to the changes you make. The new POLL API will be better than what we currently have with cpu pegging, sure. But it will not scale sufficiently for larger applications, is my impression.

[bagder]

the performance issues intrinsic to poll()-like designs are less of a concern than they are for server-side applications

The performance struggle with a poll API design is mostly a function of number of concurrent transfers, not if the application is a server or a client.

[hlandau]

that does not align with reality for our power users. I would discourage making such assumptions when designing OpenSSL's API for the future. You are free to design the API any way you want, of course, but we try to give you feedback from one of the most common (and powerful) client side applications you'll probably encounter.

Much appreciated! And yeah, I was focusing too much on simple curl(1) invocations. But of course libcurl usage could and will become arbitrarily intense.

Again — a kqueue-style polling interface is absolutely planned and I believe it is essential. We're on the same page here. The sole question is the timeline on which it can be delivered — my immediate objective is to get you what you need for the 3.3 release in April, subject to what can be delivered in that timeframe. Delivering the planned kqueue-style API in that timeframe might be difficult — but I will look into it as a possibility.

We have deployments where poll/select like scheduling on TCP based transfers is not good enough and better scaling event frameworks are in use. To work well in such environments, we will not iterate frequently over all active streams to check for activity or want to collect them all for a POLL-like api call into OpenSSL.

We will adapt our OpenSSL QUIC adaptation to the changes you make. The new POLL API will be better than what we currently have with cpu pegging, sure. But it will not scale sufficiently for larger applications, is my impression.

Agreed.

While I know it is a big ask — the proposed design for the kqueue-style API is available in #23455 (quic-polling.md) under 'Sketch B', and any feedback you could give on the suitability of that design for your applications would be greatly welcomed.

After all, these APIs will still be around in 20 years time — so there's an opportunity to make a real difference here. (It's certainly something I'm always thinking about when designing new OpenSSL APIs.)

[hlandau]

#23360 has merged and will be in OpenSSL 3.3.

[icing]

@hlandau I just have build openssl master and will check the new API additions.

fyi: had some trouble building, since openssl puts the wrong libdir in the pkgconfig files (just the top level dir and not the /lib underneath).

[hlandau]

Thanks! I've opened an issue for the pkgconfig issue: #23569

[icing]

Thanks!

I can now set and retrieve the negotiated idle timeout on connections. Nice. Now, how would I best determine if that has been reached? Have a crutch now in place, which reports the time of the last successful stream I/O.

I integrated other changes in curl/curl#12933: idle time, available bidi and SSL_write_ex2.

Did I miss anything else?

[levitte]

fyi: had some trouble building, since openssl puts the wrong libdir in the pkgconfig files (just the top level dir and not the /lib underneath).

Which one are you looking at, $BLDDIR/libcrypto.pc or $BLDDIR/exporters/libcrypto.pc? It's the latter that should be installed, while the former is there if you want to use the build directory for linking other projects against.

... I should probably add some commentary in those files that explains what they are for (and likewise for the corresponding CMake config files)

[hlandau]

PRs coming out of this discussion which have merged and will ship in 3.3:

• QUIC: Optimised FIN API #23343
• QUIC Polling Lite #23495
• QUIC: Idle timeout configuration & stream count query APIs #23360

PRs undergoing review and expected to ship in 3.3:

• QUIC: Explicit Event Handling Control #23535
• QUIC: Allow stream write buffer stats to be queried #23584

Current status:

• Items (1), (3) and (4) are fully addressed by features which have now merged and will be in 3.3.
• Item (2) effectively concerns autoticking causing stream state to become indeterminate after an I/O call, preventing the caller from having full point-in-time knowledge of the readiness of multiple streams. This can be resolved via the use of SSL_poll() and/or the explicit event handling control feature proposed above. Of course, we can continue to discuss more convenient/performant ways of addressing these issues, though anything that comes out of that discussion is unlikely to make 3.3.
• Item (5) is partially addressed with the subset of SSL_poll() which has now merged and will be in 3.3. This provides a workable solution, though one which will not scale to large numbers of objects. An API which provides better performance will be delivered after 3.3.
• Though I think the discussion above moved on from it, "SSL_pending() but including FIN" is information which can now be obtained via SSL_poll() if desired. Probably not worth adding its own API for it.
• The new PR QUIC: Allow stream write buffer stats to be queried #23584 allows stream write buffer stats to be queried as I certainly think it's useful for people to be able to get this information. Whether it is a good idea for an application to use it as part of control flow is another matter, but it's there if it's of use.

I've done some investigation into the feasibility of delivering the registered polling API for 3.3, and come to the conclusion that it's not really plausible. There is relatively little opportunity for scope reduction in terms of the work required that would still deliver something useful while also offering greater performance than SSL_poll. Of course this API is guaranteed to happen, but will need to be scheduled post-3.3. (As you may appreciate from e.g. some of the design mistakes in epoll that we are stuck with, poller APIs are a complex and subtle matter and it is important to get them right, so it's not something to be rushed. Plus, since the objective of such an API is to offer greater performance than SSL_poll(), I don't think it makes sense to ship it without having the opportunity to test and validate that the design meets the performance objectives.)

The prospect of providing some kind of state transition callbacks is interesting, and I'm happy to have something like that if it makes things easier for specific applications, but there are a fair number of design considerations around the API and some of the related complications it creates. Realistically, that's not something to be rushed, and isn't going to be deliverable for 3.3.

I think that this basically concludes the business that comes out of this discussion which can make 3.3.

Nice. Now, how would I best determine if that has been reached? Have a crutch now in place, which reports the time of the last successful stream I/O.

Are you looking to determine if a connection has been terminated due to the QUIC idle timeout? You can identify this condition by calling SSL_get_conn_close_info and testing for an error code of UINT64_MAX (transport). Which really should be documented better — I've raised a PR to provide documentation and a proper define:

• QUIC: Clarify idle timeout and make error codes public #23598

Let me know if there's anything else I can do to help. I'll try to make some time to look at your branch some point, since it will definitely be useful input.

[talregev]

Any plans to have this discussion for openssl 3.4.0?

[talregev]

@levitte This discussion is sufficient, or a new discussion will be more helpful?

[liudongmiao]

For the 5. Efficient Stream Receives, I have the same problem.

If I use the fd read event, then I cannot know which stream should be called in SSL_read_ex.

Could there an API like SSL_read_ex2 to read on the quic connection ssl and return the stream id? (we still need lhash or similar to map stream_id to stream ssl.)

[talregev]

PRs coming out of this discussion which have merged and will ship in 3.3:

• QUIC: Optimised FIN API #23343
• QUIC Polling Lite #23495
• QUIC: Idle timeout configuration & stream count query APIs #23360

PRs undergoing review and expected to ship in 3.3:

• QUIC: Explicit Event Handling Control #23535
• QUIC: Allow stream write buffer stats to be queried #23584

Current status:

• Items (1), (3) and (4) are fully addressed by features which have now merged and will be in 3.3.
• Item (2) effectively concerns autoticking causing stream state to become indeterminate after an I/O call, preventing the caller from having full point-in-time knowledge of the readiness of multiple streams. This can be resolved via the use of SSL_poll() and/or the explicit event handling control feature proposed above. Of course, we can continue to discuss more convenient/performant ways of addressing these issues, though anything that comes out of that discussion is unlikely to make 3.3.
• Item (5) is partially addressed with the subset of SSL_poll() which has now merged and will be in 3.3. This provides a workable solution, though one which will not scale to large numbers of objects. An API which provides better performance will be delivered after 3.3.
• Though I think the discussion above moved on from it, "SSL_pending() but including FIN" is information which can now be obtained via SSL_poll() if desired. Probably not worth adding its own API for it.
• The new PR QUIC: Allow stream write buffer stats to be queried #23584 allows stream write buffer stats to be queried as I certainly think it's useful for people to be able to get this information. Whether it is a good idea for an application to use it as part of control flow is another matter, but it's there if it's of use.

I've done some investigation into the feasibility of delivering the registered polling API for 3.3, and come to the conclusion that it's not really plausible. There is relatively little opportunity for scope reduction in terms of the work required that would still deliver something useful while also offering greater performance than SSL_poll. Of course this API is guaranteed to happen, but will need to be scheduled post-3.3. (As you may appreciate from e.g. some of the design mistakes in epoll that we are stuck with, poller APIs are a complex and subtle matter and it is important to get them right, so it's not something to be rushed. Plus, since the objective of such an API is to offer greater performance than SSL_poll(), I don't think it makes sense to ship it without having the opportunity to test and validate that the design meets the performance objectives.)

The prospect of providing some kind of state transition callbacks is interesting, and I'm happy to have something like that if it makes things easier for specific applications, but there are a fair number of design considerations around the API and some of the related complications it creates. Realistically, that's not something to be rushed, and isn't going to be deliverable for 3.3.

I think that this basically concludes the business that comes out of this discussion which can make 3.3.

Nice. Now, how would I best determine if that has been reached? Have a crutch now in place, which reports the time of the last successful stream I/O.

Are you looking to determine if a connection has been terminated due to the QUIC idle timeout? You can identify this condition by calling SSL_get_conn_close_info and testing for an error code of UINT64_MAX (transport). Which really should be documented better — I've raised a PR to provide documentation and a proper define:

• QUIC: Clarify idle timeout and make error codes public #23598

Let me know if there's anything else I can do to help. I'll try to make some time to look at your branch some point, since it will definitely be useful input.

@icing Can you confirm that 1,3,4 are fully addressed, and if not, what missing on them?
@hlandau Any plans to addressed the 2,5 in 3.4.0 version?

Thank you all for your work.

[icing]

1 and 3 have been addressed. About 4 I am not sure how it addresses the situation. If QUIC packet processing/generating needs to be done "explicit", it is not clear to me what this means. Is there sample code somewhere that shows how a client is supposed to use that?

The proposed 5 seems to imply an iteration over all ongoing streams whenever the state of the connection needs to be assessed. If the current "partially addressed" proposal "will not scale to large numbers of objects" I am hesitant in investing time to evaluate it.

As I described before, we need to be able to quickly asses the POLL status of a QUIC connection. Is a stream blocked on writing or not? This is not to be confused with the total connection state. Stream A and B are sending. A is blocked, B may write. We have data for A, but no data for B right now. So we should not POLLOUT the socket. These kind of decisions we currently are unable to make in a cost efficient manner.

The current state of our QUIC implementation can be seen here: https://github.com/icing/blog/blob/main/curl-h3-performance.md. As you see in the second chart, OpenSSL QUIC is at 50% of the peformance of the other implementations.

We'd appreciate any advice on how that can be improved.

[liudongmiao]

@icing For (4), does SSL_get_quic_stream_uni_local_avail / SSL_get_quic_stream_uni_remote_avail work?
Ref: https://www.openssl.org/docs/man3.3/man3/SSL_get_value_uint.html

[nhorman]

Regarding item 5 above, does SSL_poll not provide the data needed here? That is to say, SSL_poll will tell you which SSL stream objects need reading via the SSL_EVENT_READ revent flag. Or is there something more you are looking for here?

[talregev]

@icing For (4), does SSL_get_quic_stream_uni_local_avail / SSL_get_quic_stream_uni_remote_avail work? Ref: https://www.openssl.org/docs/man3.3/man3/SSL_get_value_uint.html

@icing Is 4 is address by that?
Also there is a release of openssl 3.4.0
2,5 are address in 3.4.0 version?

Thank you for your time and your hard work on curl! 🙏🏻

[icing]

Looks like we can solve 4 with that. I have not have much time to work on openssl's quic recently.

[icing]

I updated the discussion intro with the information what item can be considered fixed.

[talregev]

Thank you! 🙏🏻

[nhorman]

@icing regarding item 2 above, why does SSL_poll on a stream object waiting for a SSL_POLL_EVENT_W event not meet your needs here? Just trying to get my head around what you need here.

[talregev]

@icing is very busy, but when he will have time, it will be great that if he can help us here, to create a path for http3 with openssl not experimental in curl. I think this is what we want here, and many users of openssl and curl will be loved too. 🙏🏻

[nhorman]

ok, but that still doesn't answer the question (regardless of weather it is answered by @icing or by someone else). Before we go adding a demo. is SSL_poll(SSL_POLL_EVENT_W) what you are looking for to address item 2 above?

[bagder]

Given the descriptions, it sounds like SSL_poll() should work.

A complaint mentioned by @icing above is that SSL_want_write() is/was returning TRUE "too easily" (basically lying). SSL_poll() only works if it does not have that same issue.

[nhorman]

I've submitted curl/curl#15909 to the curl project to handle this

[bagder]

That PR has been merged into @curl.

[talregev]

@nhorman
Thank you for your PR: curl/curl#15909 💪🏻
Will you try to solve item 5?

[bagder]

I believe the OpenSSL project needs to take a step back and consider QUIC performance in general. Maybe start measuring a little and think about where time is spent.

OpenSSL is significantly slower than the competition and the API it provides is part of the explanation. Step 5 is just one piece of that puzzle.