TCP comms could be faster

[pitrou]

This issue is the distributed-specific side of tornadoweb/tornado#2147. We are using IOStream for the framed Comm protocol, which incurs additional copies.

A benchmark script is available at https://gist.github.com/pitrou/245e24de52dec34e03cfc4148c001466

[pitrou]

Some results for the above benchmark:

• with Tornado master: 370 MB/s (same for raw IOStream)
• with [WIP] Fix #2147: mostly zero-copy writes and reads in IOStream tornadoweb/tornado#2166: 680 MB/s (same for raw IOStream)
• with experimental limited IOStream: more than 900 MB/s (raw iOStream is a few % faster)

All this on Ubuntu 16.04 with Python 3.6.

[TomAugspurger]

Was this issue essentially fixed by the use of read_into, or are there additional steps we should take?

distributed/distributed/comm/tcp.py

Lines 190 to 193 in 90758dc

	if PY3 and self._iostream_has_read_into:
	frame = bytearray(length)
	n = yield stream.read_into(frame)
	assert n == length, (n, length)

[pitrou]

Both read_into and the write improvements. It should probably be fixed (out of curiosity, did you run the benchmark script and which numbers did you get?).

[TomAugspurger]

633 MB/s with tornado 5.1.1

I'm working on UCX-backed comms (#2344) and will post the performance results once I get things working.

[pitrou]

Which CPU and system, out of curiosity?

[TomAugspurger]

That's on an Nvidia DGX system, and I believe the CPU is

model name      : Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz

[pitrou]

I see, thanks. My benchmark numbers above had been obtained with a Core i5-2500K.

[jakirkham]

Should we close this issue? Sounds like this was already addressed (though please correct me if not)

[GenevieveBuckley]

Was this issue essentially fixed by the use of read_into, or are there additional steps we should take?

distributed/distributed/comm/tcp.py

Lines 190 to 193 in 90758dc

	if PY3 and self._iostream_has_read_into:
	frame = bytearray(length)
	n = yield stream.read_into(frame)
	assert n == length, (n, length)

@pitrou did this sufficiently fix your issues?

[jakirkham]

Yeah I think this was fixed a while ago. Let’s go ahead and close this. There are already newer issues on improving communication. We can follow up on additional tasks in those issues or in a new issue