controlling channel buffer maximum I/O size

[jhh67]

Issue #18913 modifies the channel buffer so that there is a limit to the size of the I/Os between the buffer and the underlying file. This behavior is not controllable by the application, however. Should we:

1. Modify channel.init with an optional parameter that sets the maximum I/O size? If so, what should its name be?
2. And/or should there be channel methods that allow the application to set and get the maximum I/O size? If so, what should their names be?
3. What should the default maximum I/O size be? Note that one possibility is "unlimited" which is backwards compatible with the current behavior.

[mppf]

Issue #18913 modifies the channel buffer so that there is a limit to the size of the I/Os between the buffer and the underlying file.

Did you mean to mention a PR?

I thought this issue would be about the size of the buffer chunks that the qio system allocates, which is currently 64KiB, and not configurable (except by undocumented means). But, it sounds like it is talking about the amount of memory that can be buffered with mark etc. Am I understanding this correctly?

[jhh67]

I created draft PR #19967 for this issue, although tbh i'm never sure when to create a PR and when not to. In this case there are still unresolved design decisions so a PR seems premature but I went ahead and created one anyway. I made it a draft because it's not ready for review until these design decisions have been finalized.

The issue here is not the size of the buffer chunks, but the size of the I/Os between the buffer and the underlying file, which in turn affects the size of the copies between the application and the channel buffer, and therefore the total amount of channel buffer required. Previously, doing a read of N bytes into a string or bytes required 3N bytes of memory, while doing a write of N bytes of a string or bytes required 2N bytes of memory. With these changes reads and writes require N+d bytes of memory where d is the size of the I/O (rounded up to a whole number of buffer chunks).

[mppf]

I don't think that it needs to be configurable at all. I think choosing a value between 4KB and 64KB would probably be fine.

[jhh67]

Closing this as there isn't currently a desire to make this configurable by the application. It shouldn't be hard to add this functionality if we subsequently decide it's needed. The default is currently 64KB.