Do we want to add Connection Pools to the Architecture / API

[philsbln]

During the discussion of possible QUIC mappings for TAPS in Bangkok, we have seen that there might be two ways to map QUIC Streams / Connections to TAPS Messages / Connections / Connection groups – Mapping QUIC Streams to TAPS Message is more convenient for HTTP style applications, mapping QUIC streams to TAPS Connections looks more natural.

One way to solve this tussle is to introduce a new kind of object – a Connection Pool:

• An application can send a message using a connection pool and get events for a reply without touching a connection object
• A connection pool can represent things like a multi-streaming QUIC connection or a bunch of TCP connections between equivalent endpoints.
• To make this logic happen, a Connection Pool needs to be stuffed with framers to make this work
• A Send on a connection pool may take Selection Properties to aid selection between multiple connections within a pool.

Do we want to add Connection Pools to the Architecture / API ?

[csperkins]

Mapping a TAPS message to a QUIC stream seems odd to me. QUIC/HTTP streams can carry multiple requests, surely? I'd like to understand the mapping much better before introducing connection pools.

[philsbln]

The QUIC/HTTP mapping opens a new bi-directional stream for each request.
In reply, it can send the HTTP reply and multiple push-promises related to the request.
The pushed objects then go on a separate uni-directional server-initiated stream each.

[csperkins]

Wouldn't the reply and the push promises be separate TAPS messages? This sounds like several messages sent on a single stream.

[philsbln]

So Reply and Push-Promise would be received over the same stream, where a HTTP/3 framer can decide what to do:

• Separate them into separate TAPS messages.
• Append a list of all Push-Promises to the response message

Still there is a 1:1 relationship between requests/responses and QUIC streams.
The semantic if push-promise in HTTP/3 binds a push to a request while the unsolicited response comes on new separate stream (which the client would need to "listen" for if we map QUIC streams to TAPS connections)

[csperkins]

I view the framer as something that handles packetisation, not something that's involved in connection handling. The Reply and Push-Promise would be de-framed as TAPS messages, and when the server initiates a new stream, a ConnectionReceived<> event would fire. The logic is HTTP/3 is not in TAPS; just the framing.

[philsbln]

I view framers more as some kind of input filters that can do anything from looking at message boundaries to implementing protocol layers – but I guess this does not matter for this case.

In both cases above, the framer parses HTTP/3 – the only question is wether on a HTTP/3 frame or an HTTP/3 Request level. The latter has the advantage that Requests are neatly aligned with QUIC streams (except for server push that needs additional application logic) and allow to turn HTTP/3 Requests/Responses into TAPS messages.

[csperkins]

It seems we perhaps need to define the framer more clearly

[mwelzl]

I'm afraid that the framer is turning into a magic box that can potentially do whatever we currently can't agree on. Like @csperkins, I didn't expect a framer to do connection handling. However, it seems we do need a certain amount of functionality beyond "just framing", like changing them in-flight, as per #249.

Regarding Connection pools, I'm sceptical - it seems like a proposal to add a lot of extra just to support some possibly obscure types of messages-stream mapping (while we are trying to abstract streams away here, as much as possible). It reminds me a little of a discussion in NEAT, where we added stream numbers just because Michael Tuexen wanted to be able to talk to some existing SCTP servers that assume certain communication to go over a stream with a certain number. That was ok for the research project that it was - but I think here, we should minimize such things; this specific SCTP functionality would probably be a protocol-specific property for us here.

Back to the point: I think we should add as little as possible, while ensuring that we can communicate with an HTTP/3 peer. This means that, if there are 2 HTTP-QUIC mapping options allowed, optional to choose from, then locally, for a TAPS host, we should pick one, but we must be able to correctly communicate with an HTTP/3 peer that can be implemented with option 1 or 2. I'm not sure that this amounts to the same functionality?

In other words: if a TAPS host wants to correctly communicate with a non-TAPS HTTP/3 host, does it really need these Connection Pools?

[philsbln]

Connection pools are not needed for a TAPS host to communicate with a non-TAPS HTTP/3 hosts. This can always be resized on top of TAPS.

Connection Pools are needed to hide from the application whether HTTPS (HTTP/1.1 over TLS), HTTP/2 or HTTP/3 (HTTP over QUIC) are used. The logic around these protocols has become so tightly integrated with TLS, QUIC and Happy Eyeballs that using these three from TAPS while implementing the HTTP variants in the application may need many back and forth channels, e.g., to add information learned though ALT SRV to the Happy Eyeballs candidates and enable connection coalescing.
In the future, it will become even tighter and may also include DNS resolution (see quicwg/base-drafts#253).

So I fear without abstracting HTTP away in the connection pool, the tight interactions defined between HTTP variants and everything else will force HTTP clients to re-implement most of the TAPS functionality or require them to interact with the TAPS implementation internals to share all necessary information.

[mikkelfj]

If ordered messaging is important, you should map a TAPS connection to a stream. If low latency is important, you should map each TAPS messages to a single stream.

Latency that is normally 5ms on average can jump to 300ms out of nowhere.

[philsbln]

@mikkelfj I don't think this is the right abstraction, but a hack.

1. The mapping of Messages to Streams should be done by the application.
2. The only exception I would like to make is for pseudo-transports like HTTP. Here, we have a 1:1 mapping for HTTP/3 (by design, whereby multiple streams can share one connection) and arbitrary mapping for HTTP/1.1 and HTTP/2 (which can introduce head-of-line blocking by sharing a connection). For the latter case, your heuristic is useful.

To enable the application to choose between 1. and 2., we need the connection pool abstraction. Using it gives you 2., manually managing connections gives you 1.

[mwelzl]

Right now there are voices against the Connection Pools in PR #295. I agree with the sentiment of @tfpauly and @britram there that we shouldn't add such functionality unless we must (and they don't seem to see that necessity), i.e. Connections should give you all functionality you need anyway.

Looking at this issue again, @philsbn writes: "So I fear without abstracting HTTP away in the connection pool, the tight interactions defined between HTTP variants and everything else will force HTTP clients to re-implement most of the TAPS functionality or require them to interact with the TAPS implementation internals to share all necessary information."

That draws a worrisome picture. My take-away is that, if Connection Pools are not added, then the implementation draft should explain how Connections can work atop HTTP/1.1, /2 and /3.

[philsbln]

@mwelzl I am not sure wether we really need a connection pool object or can merge its functionality into the Connection object. The functionality I see is the following:

• Merge several transport protocol connection into one TAPS connection (possibly also across different protocols like HTTP/1.1, HTTP/2, HTTP/3…).
• Link Requests to Responses.
• Allow per-message path- and endpoint selection (which implies automatically spawning transport protocol connection including racing)

As this semantic requires a big explicit switch, I am not sure this is "just" a selection property.

[mwelzl]

I understand. But if we can do this more easily, with lots of "Require" classifiers, that would be nicer IMO.

[theri]

I agree that abstracting some more connection/stream/message handling away from the application is a good idea. I think it would be best to have this functionality provided by a TAPS Connection, instead of adding a separate Connection Pool.
However, this is not "just" a selection property, but a bigger switch. Actually I'd like it if the application does not explicitly have to set a property that it is okay with having multiple underlying TCP connections instead of one QUIC connection, for example.

As I see it, this goes back to our definition of TAPS Connection and how it relates to Protocol Instances and Stacks, maybe touching upon the "What is a Connection" discussion we had a year ago:

Reading the architecture draft, I find that Connection is defined as "state pertaining to the underlying transport protocol instance and any ongoing data transfer", and Protocol instance is right now defined a "a single instance of one protocol". From this I conclude that one TAPS Connection can maybe abstract one QUIC connection with multiple streams, but it cannot abstract multiple TCP connections. Thus, an application would have to check whether the Connection is HTTP/1.1 HTTP/2, or HTTP/3, and in the two former cases, perhaps it has to create more TAPS Connections. I agree this is a problem.

To solve this, I think we could define a TAPS Connection as "state pertaining to one or multiple underlying transport protocol instances". Then the application would have just one TAPS Connection and be able to use it like a Connection Pool in PR #295, just send Messages and not care about the HTTP version or any underlying TCP connection or HTTP/2 or QUIC stream handling.

However, this has architecture implications. For example, it implies that Initiate() is not the only action that can spawn a TCP connection, but a transport system can also create new TCP connections (or new QUIC streams) on Send(). So if the application calls Send(), it might result in a new underlying TCP connection for the same TAPS Connection. I'm not sure our current architecture covers this.
Similar with Listen() and Receive(): Does an incoming TCP connection always create a new TAPS Connection for the application, or can it also just be an incoming Received<> event?

I'm wondering if defining a Connection as potentially multiple protocol instances could create problems for Connection Properties, which might be different for different protocol stack instances, i.e., different TCP connections.

Also, I'm wondering about multipath/multistreaming protocols - right now, Section 4.2 in the architecture document reads as if any "multiplexing transport protocol" MUST be represented as multiple TAPS Connections in a Connection Group to the application. But maybe we want to further abstract this in some cases, i.e., hide the multiple streams from the application, and just provide one TAPS Connection to the application, on which it receives multiple Messages? Then we would again be close to the Connection Pool proposed in the PR.

So, all in all, if we want this functionality (which I think we really want), but not add Connection Pools but let the Connection provide the functionality, I think we have to go back and revisit the definition of Connection and how it relates to protocol stack instances.
I suggest we talk about this in-person in Prague.

[philsbln]

I don't see any of this land today anyway – will make two PRs for discussion in Prague – one with explicit Connection Pool object and one with extended Connection object and selection Property with default forbid.

@theri do you think the time for connection pools on the agenda is sufficient?

[mwelzl]

Thanks @theri for this very good analysis of the situation. Exactly the scope for a discussion to be had IMO.

[gorryfair]

I can see benefits in adding this (thanks Theresa for the review of issues) - but I still don't know that we have to do this right now - so why would I change from: "we shouldn't add such functionality unless we must"?

[philsbln]

Status update for this issue:

• PR Merge Message References and Message Contexts and enable their use for Requests/Response protocols #321 adds a way to link requests and responses to the API.
• Once Merge Message References and Message Contexts and enable their use for Requests/Response protocols #321 landed, I will make another PR to the API that describes connection coalescing and connection pooling

[philsbln]

With #321 merged, there is only the connection coalescing and connection pooling part missing, which needs a bit of discussion in Montreal before turning it into text.

[philsbln]

Closing this issue as all remaining work moved to Implementation and will be tracked in Issue #339