Add file-like reader/(maybe writer) class for memoryview

[abebus]

Feature or enhancement

Proposal:

This is a WIP draft.

Rationale

At present, the only standard way to wrap a bytes-like buffer in a file-like object is io.BytesIO.
However, BytesIO always copies the input.

For some use cases this is perfectly fine, but in other cases you already have the data in memory (e.g. bytes, bytearray, memoryview, or array.array) and just need a thin file-like reader on top of it. Creating another copy seems unnecessary.

I think a MemoryviewReader class could fill this gap. It would act like a lightweight, zero-copy BytesIO, wrapping any buffer that supports the buffer protocol.

That said, maybe this is too niche, or maybe there are drawbacks I haven’t considered — I could be wrong.

Notes on Implementation

The following is just a proof of concept in pure Python to illustrate the idea. If there is interest, a C implementation might be worthwhile, since in Python 2 there was both BytesIO and cBytesIO, with the latter being noticeably faster.

import io
from typing import Self

class MemoryviewReader:
    """
    File-like reader over internal buffer of bytes or bytearray using memoryview.

    Basic read:
    >>> r = MemoryviewReader(b"hello world")
    >>> r.read(5), r.read()
    (b'hello', b' world')

    Read lines:
    >>> r = MemoryviewReader(b"a\\nb\\nc")
    >>> r.readline(99), r.readline(1), r.readline(), r.readline(), r.readline(99)
    (b'a\\n', b'b', b'\\n', b'c', b'')

    Iterate lines:
    >>> list(MemoryviewReader(b"x\\ny"))
    [b'x\\n', b'y']

    Seek/tell:
    >>> r = MemoryviewReader(memoryview(b"abcdef"))
    >>> r.read(3), r.tell()
    (b'abc', 3)
    >>> r.seek(0), r.read(2)
    (0, b'ab')
    >>> r.seek(2, io.SEEK_CUR), r.read(2)
    (4, b'ef')
    >>> r.seek(-3, io.SEEK_END), r.read(99)
    (3, b'def')

    Errors:
    >>> r.seek(0, 99)
    Traceback (most recent call last):
    ...
    ValueError: Invalid whence
    >>> r.seek(-10, io.SEEK_SET)
    Traceback (most recent call last):
    ...
    ValueError: Seek out of range
    >>> r.seek(10, io.SEEK_SET)
    Traceback (most recent call last):
    ...
    ValueError: Seek out of range
    """  # noqa: D301

    __slots__ = ("_mv", "_pos", "size")

    def __init__(self, buf: bytes | bytearray | memoryview):
        if not isinstance(buf, memoryview):
            buf = memoryview(buf)
        self._mv = buf
        self._pos = 0
        self.size = len(self._mv)

    def read(self, amount: int = -1, /) -> bytes:
        if amount < 0:
            end = self.size
        else:
            end = self._pos + amount
            if end > self.size:
                end = self.size
        start = self._pos
        self._pos = end
        return self._mv[start:end].tobytes()

    def readline(self, amount: int = -1, /) -> bytes:
        if self._pos >= self.size:
            return b""
        if amount < 0:
            end = self.size
        else:
            end = self._pos + amount
            if end > self.size:
                end = self.size
        start = self._pos
        i = start
        while i < end:
            if self._mv[i] == 10:  # ord('\n')
                i += 1
                break
            i += 1
        self._pos = i
        return self._mv[start:i].tobytes()

    def __iter__(self) -> Self:
        return self

    def __next__(self) -> bytes:
        line = self.readline()
        if not line:
            raise StopIteration
        return line

    def seek(self, offset: int, whence: int = 0) -> int:
        if whence == io.SEEK_SET:
            new = offset
        elif whence == io.SEEK_CUR:
            new = self._pos + offset
        elif whence == io.SEEK_END:
            new = self.size + offset
        else:
            raise ValueError("Invalid whence")
        if not (0 <= new <= self.size):
            raise ValueError("Seek out of range")
        self._pos = new
        return self._pos

    def tell(self) -> int:
        return self._pos

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

[serhiy-storchaka]

Did you know that in CPython, BytesIO does not copy the input data if it is a bytes object and you are only reading it?

[abebus]

@serhiy-storchaka oh, no, I didn't... I assumed all the references in the documentation to the underlying raw stream were about the internal buffer of BytesIO and BufferedReader, not the original bytes object that was passed in. Thanks for clarifying! Maybe it would be helpful to mention this explicitly in the docs?

[abebus]

So, this can be closed as irrelevant?

[uhx]

Did you know that in CPython, BytesIO does not copy the input data if it is a bytes object and you are only reading it?

@serhiy-storchaka I would disagree, as BytesIO never provided zero-copy for reading (at least for me?):

import tracemalloc
import io


data = b'A' * 16894
tracemalloc.start()
d = io.BytesIO(data)
print(tracemalloc.get_traced_memory()[0])  # 80
bytesio_read = d.read(4096)
print(tracemalloc.get_traced_memory()[0])  # 4209
tracemalloc.stop()
tracemalloc.start()
m = memoryview(data)
print(tracemalloc.get_traced_memory()[0])  # 312
memoryview_read = m[0:4096]
print(tracemalloc.get_traced_memory()[0])  # 496
tracemalloc.stop()

where tracemalloc.get_traced_memory()[0] is the current allocated memory since tracemalloc.start()

As we can see, it allocates 80 bytes for BytesIO itself on init, and then allocates 4129 (4209-80) on read.

Python 3.11.2
UPD: same for Python 3.13.7

[abebus]

Writing this comment for documenting our discussion in other medium

@uhx I assume that currently reading without copying can only be achieved if we will return new memoryview (or its slice) on the initial bytes. My sketch copies by calling .tobytes(), so here it matches BytesIO behaviour

As you, @Eclips4 and @sergey-miryanov discussed, semantically it is doable to return bytes during read by making a slice and increasing the refcount on the parent/initial buffer, that will eventually increase the lifetime of the parent object (which is not preferred).
But it is also possible to do something with weakref.finalize, so bytes from the initial buffer will be copied to the slices it(reader class) produced only if initial buffer needs to be freed

[uhx]

Thank you, this is very smart of you to document all the thoughts here

I will add something as well:

Actually, I was supposing that a memoryview slice is essentially some sort of zero-allocation bytes. But in reality, it's just a sliced... memoryview, or, if we do .tobytes(), just a copied bytes from the input buffer, which doesn't make much sense in this context, since it will behave the same as io.BytesIO.

[abebus]

Yes, this does not make much sense since this is how BytesIO actually behaves. I wrote this snippet because I wrongly thought that BytesIO copies input data

[serhiy-storchaka]

Maybe it would be helpful to mention this explicitly in the docs?

This is just a CPython implementation detail. A detailed description of when a copy is made and when it is not would complicate the documentation. Most users don't need this. They just got the pleasant surprise that their program ran a little faster when this optimization was added in Python 3.5.

Other implementations may not have such optimizations. But if it is for simplicity or technical reasons, then their implementation of MemoryviewReader will simply be a read-only wrapper over BytesIO, so MemoryviewReader will have no advantage over BytesIO in these implementations too.

[abebus]

Well, that makes sense... But what would be a good place to record this so it’s documented somewhere, so curious users like me don’t end up reinventing the wheel?