UDP based transaction ingestion is too simple for current usage patterns

[jstarry]

Problems

Each transaction needs to be sent to the current leader over UDP to the leader's TPU port.

• Because packet fragmentation is slow, transactions must be sent in a single small datagram (currently no more than 1232 byte payload).
• Since UDP is unreliable, clients don't know if their sent transaction has been received and so they repeatedly send txs until they eventually observe their tx included in a block or the tx expires.
• Due to IP spoofing, a leader cannot reliably enforce rate limiting for clients

Proposed Solution

Add support for the QUIC protocol for ingesting transactions to get

1. Support for larger transactions
2. Reliable transmission
3. Enforceable rate limiting
4. standardized flow control which can be potentially offloaded to something outside the validator

[riptl]

@jstarry @sakridge @ryleung-solana What is the proposed way to submit transactions using QUIC?

The PR seems to use both datagrams and streams.

Some additional context:

• QUIC datagrams cannot be fragmented so they cannot support larger transactions.
• QUIC datagrams are ACK-eliciting so senders can detect if the recipient has ingested their DATAGRAM.
• Benchmarks have shown that each CPU core can handle about 10000 QUIC stream creations per second using the quinn server.
• Rate-limiting for DATAGRAM can be enforced through bandwidth using MAX_DATA frames and rate limiting for stream creation and per-stream bandwidh through MAX_STREAMS and MAX_STREAM_DATA frames.

[jstarry]

Streams seem well suited for intra-validator tx forwarding, RPC nodes, program deploys, oracles, and bots. But I'm not sure what the plan is for rate limiting unknown clients, I imagine we'd use datagrams for that and then maybe leaders could have both a global rate limiter and a rate limit per tx signer.

(Edited because I misunderstood that acks don't require bidirection streams)

[jstarry]

This issue isn't valuable anymore because the migration to QUIC is well underway. There are other issues tracking migration progress.