FileIO size limit

[woutdenolf]

FileIO.read and FileIO.write may read/write less bytes than requested, in which case you need multiple calls.

For example:

import os
from io import FileIO

MAX = 2 ** 31 - 4096

filename = "test.dat"
nbytes = MAX + 1
print("Data size bytes:", nbytes)

data = b"0" * nbytes
for openctx in (open, FileIO):
    with openctx(filename, "wb") as f:
        nwritten = 0
        n = 0
        while nwritten < nbytes:
            nwritten += f.write(data[nwritten:])
            n += 1
    print(f"\nFile size bytes: {os.stat(filename).st_size} (written in {n} times)")

    with openctx(filename, "rb") as f:
        nbytesread = 0
        nmissing = nbytes
        n = 0
        while nmissing:
            nbytesread += len(f.read(nmissing))
            nmissing = nbytes - nbytesread
            n += 1
        print(f"Read size bytes: {nbytesread} (read in {n} times)")

In other words, Fabio cannot handle images larger that 2GB.

[woutdenolf]

See https://docs.python.org/3/library/io.html#io.RawIOBase

[kif]

The snippet of code produces the same number of bytes written and read regardless the value of nbytes (1, 2 or 3 GB) on linux x86_64.
Maybe this bug is platform specific ?

[woutdenolf]

The nbytes for one read and write operation are equal yes, but that issue is that you cannot save an EDF image larger than 2GB for example.

[kif]

Indeed I managed to reproduce the bug with an image large enough ...

[kif]

In [13]: (2**31/4)**0.5
Out[13]: 23170.47500592079

In [14]: raw = numpy.random.randint(0,100000, size=(23200,23200)).astype("uint32")

In [15]: raw.nbytes, numpy.log(raw.nbytes)/numpy.log(2)
Out[15]: (2152960000, 31.003674369804592)

In [16]: fabio.edfimage.edfimage(data=raw).write("/tmp/big2g.edf")

In [17]: os.stat("/tmp/big2g.edf")
Out[17]: os.stat_result(st_mode=33188, st_ino=1911, st_dev=66308, st_nlink=1, st_uid=1000, st_gid=1000, st_size=2147479552, st_atime=1641829820, st_mtime=1641830092, st_ctime=1641830092)

In [18]: re = fabio.open("/tmp/big2g.edf").data
ERROR:fabio.edfimage:Data stream is incomplete: 2147479040 < expected 2152960000 bytes

[kif]

It would be nice to have such test ... there are very large tests which are disabled most of the time (like the ones searching for memory-leaks)

[woutdenolf]

Ok sure, I'll add a test