Wish: Increase IO_BUFFER_SIZE #5501
Original bug ID: 5501
I think IO_BUFFER_SIZE=4096 (for channel I/O) is nowadays a bit low. For current machines and OS, this should be increased, maybe to 64K. (It's a constant in byterun/io.c).
A few indications why this is reasonable:
Recently I ran into a case where the producer of a pipe slowed the consumer down, because only 4K were written at a time. The pattern was especially bad because the producer wrote the results of computations and used the CPU to 100%, and the 4K blocks were emitted with a certain frequency. The consumer was poll-based, and the poll loop was running way to often, also at the same frequency. The result was that the consumer also used another core to almost 100%. The lesson to learn for me was that the consumer cannot do anything against this when the producer drives it the wrong way. Using more of the pipe buffer finally solved the problem (by running the pipe like "producer | dd bs=1M iflags=fullblock | consumer")