document transport read/write pause, buffer sizes, low/high water marks, etc.

[vstinner]

Asyncio performance and concurrency depends on parameters which are badly documented in the reference documentation:

• 'limit' of StreamReader/open_connection(): https://docs.python.org/dev/library/asyncio-stream.html#streamreader
• set_write_buffer_limits() low/high water mark on writing for transports, https://docs.python.org/dev/library/asyncio-protocol.html#asyncio.WriteTransport.set_write_buffer_limits
• etc.

[vstinner]

cc @1st1 @GMLudo @methane @asvetlov etc.

[vstinner]

Maybe we can do deeper and mention kernel parameters, Linux tcp sysctls:

• /proc/sys/net/ipv4/tcp_mem
• /proc/sys/net/core/rmem_default and /proc/sys/net/core/rmem_max: The default and maximum amount for the receive socket memory
• /proc/sys/net/core/wmem_default and /proc/sys/net/core/wmem_max: The default and maximum amount for the send socket memory
• /proc/sys/net/core/optmem_max : The maximum amount of option memory buffers
• net.ipv4.tcp_no_metrics_save
• net.core.netdev_max_backlog: Set maximum number of packets, queued on the INPUT side, when the interface receives packets faster than kernel can process them.
• etc.

[vstinner]

Ah, another useful information for performance: since Python 3.6, asyncio now sets the TCP_NODELAY option on newly created sockets.

"TCP_NODELAY is intended to disable/enable segment buffering so data can be sent out to peer as quickly as possible, so this is typically used to improve network utilisation."

(Windows doc) "Disables the Nagle algorithm for send coalescing. This socket option is included for backward compatibility with Windows Sockets 1.1"

Ah, I also found "TCP_QUICKACK is used to send out acknowledgements as early as possible than delayed under some protocol level exchanging, and it's not stable/permanent, subsequent TCP transactions (which may happen under the hood) can disregard this option depending on actual protocol level processing or any actual disagreements between user setting and stack behaviour." on the Internet. I never saw this option before.

[vstinner]

It seems like aiohttp uses set_writer_buffer_limits(0), see:

• Graceful shutdown can (I think) truncate connections that it shouldn't aio-libs/aiohttp#1369
• https://github.com/aio-libs/aiohttp/pull/1478/files

And https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/

• "Unfortunately, this isn't enough. As the docs warn us, drain doesn't actually block until all data is written; it only guarantees that the unwritten data is less than the "high water mark"
• "To really fix the problem, we need to get rid of this high-water mark thing. This can be done by calling transport.set_write_buffer_limits(0) on the underlying transport object; then drain will only return once the send buffer is completely empty."

[ludovic-gasc]

For me, it's a try-error approach to find the right parameters for the kernel.

[asvetlov]

It seems like aiohttp uses set_writer_buffer_limits(0)
Yes, we disabled it for backpressure support and implemented own buffering.

[asvetlov]

Perhaps the projects need a new page for information about performance tuning.
Albeit I didn't see such page in Twisted/Tornado docs :)

[vstinner]

I added a new performance page: commit d91afb3.