Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wish: Increase IO_BUFFER_SIZE #5501

Closed
vicuna opened this issue Feb 5, 2012 · 2 comments

Comments

Projects
None yet
2 participants
@vicuna
Copy link

commented Feb 5, 2012

Original bug ID: 5501
Reporter: gerd
Assigned to: @damiendoligez
Status: closed (set by @damiendoligez on 2012-04-10T14:09:52Z)
Resolution: fixed
Priority: normal
Severity: trivial
Version: 3.12.1
Fixed in version: 4.00.0+dev
Category: runtime system and C interface
Monitored by: mehdi @ygrek

Bug description

I think IO_BUFFER_SIZE=4096 (for channel I/O) is nowadays a bit low. For current machines and OS, this should be increased, maybe to 64K. (It's a constant in byterun/io.c).

A few indications why this is reasonable:

  • Certain OS increased their pipe buffer size (e.g. Linux to 64K, but it
    grows beyond that if needed; FreeBSD and OSX to 16K, with the option to
    grow to 64K)
  • RAM bandwidth is continously increasing
  • Systems have more RAM than ever

Additional information

Recently I ran into a case where the producer of a pipe slowed the consumer down, because only 4K were written at a time. The pattern was especially bad because the producer wrote the results of computations and used the CPU to 100%, and the 4K blocks were emitted with a certain frequency. The consumer was poll-based, and the poll loop was running way to often, also at the same frequency. The result was that the consumer also used another core to almost 100%. The lesson to learn for me was that the consumer cannot do anything against this when the producer drives it the wrong way. Using more of the pipe buffer finally solved the problem (by running the pipe like "producer | dd bs=1M iflags=fullblock | consumer")

@vicuna

This comment has been minimized.

Copy link
Author

commented Apr 10, 2012

Comment author: @alainfrisch

I'm also in favor of increasing the default value for IO_BUFFER_SIZE (to something between 16kB and 64kB). Who's against?

@vicuna

This comment has been minimized.

Copy link
Author

commented Apr 10, 2012

Comment author: @damiendoligez

Fixed in version 4.00 (commit 12331) and trunk (commit 12332).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.