_io.TextIOWrapper._CHUNK_SIZE ignores underlying buffer size, defaults 8192

[tom93]

Bug report

Bug description:

If I try to read a line from a file containing a very long line using open(path, buffering=n).readline(), it reads in chunks of 8192 regardless of the value of n. I would like it to respect n so that I can reduce the number of syscalls.

I can manually work around this by setting the undocumented attribute _CHUNK_SIZE, but I think this should be done automatically. The number 8192 is hard-coded in _io_TextIOWrapper___init___impl; I propose automatically setting it to the buffer size of the underlying _io.BufferedReader / _io.BufferedWriter / _io.BufferedRandom (if the underlying object is one of those).

docker run --rm -it python:3.14.6 bash
apt-get update && apt-get install -y strace
python3 -c 'print("x"*100000)' > f
# f.readline ignores the buffer size
strace -e trace=read python3 -c 'open("f", buffering=100000).readline()' # calls `read(3, ..., 8192)`
# workaround: set f._CHUNK_SIZE
strace -e trace=read python3 -c 'f = open("f", buffering=100000); f._CHUNK_SIZE = 1000000; f.readline()' # calls `read(3, ..., 1000000)`
# aside: f.buffer.readline respects the buffer size
strace -e trace=read python3 -c 'f = open("f", buffering=100000); f.buffer.readline()' # calls `read(3, ..., 100000)`

CPython versions tested on:

3.14

Operating systems tested on:

Linux

[picnixz]

The docs say:

buffering is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable when writing in text mode), and an integer > 1 to indicate the size in bytes of a fixed-size chunk buffer. Note that specifying a buffer size this way applies for binary buffered I/O, but TextIOWrapper (i.e., files opened with mode='r+') would have another buffering.

I am not entirely sure then this is a bug.

cc @serhiy-storchaka @cmaloney

[serhiy-storchaka]

Yes, they are unrelated.

The number of syscalls should be reduced if specify large buffering value. The binary buffered reader will use large buffer, and TextIOWrapper will read from it by smaller chunks without syscalls.

[tom93]

@picnixz Thanks, I didn't see that. I still think the current behaviour is undesirable.

@serhiy-storchaka That sounds logical, but it doesn't match the behaviour that I'm seeing. Specifically, open("f", buffering=100000).readline() makes lots of syscalls with size 8192. Maybe I misdiagnosed and there is a bug elsewhere?

[serhiy-storchaka]

This is worth investigation.

[tom93]

Here is a quick analysis of the call trace:

    _io_TextIOWrapper_readline_impl
    => _textiowrapper_readline
    => textiowrapper_read_chunk(self, size_hint = 0)
    => buffer_callmethod_onearg(self, read1 or read, chunk_size), where chunk_size = max(self->chunk_size, size_hint)
    => buffer.read1, which doesn't use buffering.

I feel the overall result is undesirable, but I'm not sure where the change should happen.

[sreehariannam]

I looked into this. Confirmed the hardcoded 8192 at Modules/_io/textio.c:1268 (self->chunk_size = 8192; in _io_TextIOWrapper___init___impl), and confirmed _io.BufferedReader/Writer/Random already track the real buffer size internally (buffer_size field in the buffered struct in bufferedio.c) but never expose it to Python.

Proposed approach, happy to send a PR if this direction sounds right:

1. Add a read-only buffer_size property to BufferedReader/BufferedWriter/BufferedRandom in bufferedio.c, using the same Argument Clinic @getter pattern as the existing closed property, just returning self->buffer_size.
2. In textio.c's __init__, inside the existing if (Py_IS_TYPE(buffer, state->PyBufferedReader_Type) || ...) block (which already fetches raw from the buffer the same way), also fetch buffer_size and use it to set self->chunk_size instead of the hardcoded 8192.
3. Tests + docs (Doc/library/io.rst, versionadded:: 3.16) for the new public buffer_size attribute, plus a NEWS entry since this is a behavior change.

Since this adds new public API surface (buffer_size becoming a documented attribute on the buffered I/O classes) rather than just fixing an internal bug, wanted to check the direction before writing the full patch.

[cmaloney]

I'm 👍 for using self->chunk_size as buffer size if it makes a measurable difference on a I/O microbenchmark. I'm not sure how much it will actually change things. The list that is then coalesced of TextIOWrapper is very distinct from the BufferedIO fixed size buffer always copied into.

👎 for any new buffer_size API: That TextIOWrapper does its own buffering and has special modes like "write_through" on top of BufferedIO is fairly confusing and to me suggests a couple refctorings. I don't see a good reason why users of TextIOWrapper should care about its internal buffer size, and exposing it restricts what we could do in the future.

For a more comprehensive on some of my thoughts re: BufferedIO and TextIOWrapper which I'm hoping to get into 3.16: https://discuss.python.org/t/reworking-buffered-i-o-in-cpython/90019/16

[sreehariannam]

Ran the microbenchmark @cmaloney asked for. Used the existing private `_CHUNK_SIZE` attribute to simulate the effect (no C changes, so this is purely data on whether the value helps, independent of the public-API question):

Syscall count (10,000 lines x 2000 bytes, ~20MB file, `buffering=1_000_000`), via `strace -c -e trace=read,pread64`:

`_CHUNK_SIZE`	`read()` calls	time in read()
default (8192)	2487	75.9ms
matched to buffering (1,000,000)	65	5.0ms

~38x fewer syscalls.

Wall-clock time (median of repeated runs, built from current `main`):

Scenario	default 8192	matched to buffering
single 100,000-byte line, `buffering=100_000`	961.5µs	555.6µs (~42% faster)
20MB / 10,000 lines, `buffering=1_000_000`	136.6ms	111.2ms (~19% faster)

So the underlying mechanism does produce a measurable difference, at least for large-buffering + long-line/large-file workloads. Whatever the eventual API shape turns out to be (new attribute, internal-only default change, or something from the broader rework), the direction of the fix looks worth pursuing. Happy to help however's useful once the design direction settles — benchmark scripts available if anyone wants to reproduce or extend them.