io.FileIO cannot write more than 2GB (-4096) bytes??? must be documented (if not fixed)

[YaroslavHalchenko]

BPO	31651
Nosy	@benjaminp, @serhiy-storchaka, @eryksun
Files	longwrite.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-10-01.01:53:54.757>
created_at = <Date 2017-09-30.20:58:07.538>
labels = ['type-bug', 'invalid', 'expert-IO']
title = 'io.FileIO cannot write more than 2GB (-4096) bytes??? must be documented (if not fixed)'
updated_at = <Date 2017-10-01.08:30:32.789>
user = 'https://bugs.python.org/YaroslavHalchenko'

bugs.python.org fields:

activity = <Date 2017-10-01.08:30:32.789>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2017-10-01.01:53:54.757>
closer = 'benjamin.peterson'
components = ['IO']
creation = <Date 2017-09-30.20:58:07.538>
creator = 'Yaroslav.Halchenko'
dependencies = []
files = ['47182']
hgrepos = []
issue_num = 31651
keywords = []
message_count = 5.0
messages = ['303429', '303443', '303447', '303448', '303452']
nosy_count = 4.0
nosy_names = ['benjamin.peterson', 'Yaroslav.Halchenko', 'serhiy.storchaka', 'eryksun']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31651'
versions = ['Python 2.7', 'Python 3.5']

[YaroslavHalchenko]

originally detected on python 2.7, but replicated with python 3.5.3 -- apparently io.FileIO, if given a bytestring of 2GB or more, cannot write it all at once -- saves (and returns that size) only 2GB - 4096.

I found no indication for such behavior anywhere in the documentation. And it is surprising to me especially since regular file.write does it just fine! attached is the code snippet which I list below and which demonstrates it

$> python3 --version; python3 longwrite.py
Python 3.5.3
Written 2147479552 out of 2147483648
4096 bytes were not written
Traceback (most recent call last):
  File "longwrite.py", line 28, in <module>
    assert in_digest == out_digest, "Digests do not match"
AssertionError: Digests do not match
python3 longwrite.py  7.03s user 5.80s system 99% cpu 12.848 total
1 11365 ->1.....................................:Sat 30 Sep 2017 04:56:26 PM EDT:.
smaug:/mnt/btrfs/scrap/tmp
$> cat longwrite.py
# -*- coding: utf-8 -*-
import io
import os
import hashlib

s = u' '*(256**4//2)  #+ u"перфекто"
s=s.encode('utf-8')
#s=' '*(10)

in_digest = hashlib.md5(s).hexdigest()
fname = 'outlong.dat'

if os.path.exists(fname):
    os.unlink(fname)

with io.FileIO(fname, 'wb') as f:
#with open(fname, 'wb') as f:
     n = f.write(s)

#n = os.stat(fname).st_size
print("Written %d out of %d" % (n, len(s)))
if n != len(s):
    print("%d bytes were not written" % (len(s) - n))

# checksum
with open(fname, 'rb') as f:
    out_digest = hashlib.md5(f.read()).hexdigest()
assert in_digest == out_digest, "Digests do not match"
print("all ok")

[benjaminp]

The docs to look at here are https://docs.python.org/3/library/io.html#io.RawIOBase.write, which points out that short writes can happen.

[eryksun]

Additionally, the FileIO documentation states the following:

The read() (when called with a positive argument), readinto() and 
write() methods on this class will only make one system call.

The Linux man page for write() in turn states this:

On Linux, write() (and similar system calls) will transfer at most 
0x7ffff000 (2,147,479,552) bytes, returning the number of bytes 
actually transferred.  (This is true on both 32-bit and 64-bit 
systems.)

[YaroslavHalchenko]

Thank you for the follow-ups!

Wouldn't it be better if Python documentation said exactly that

On Linux, write() (and similar system calls) will transfer at most 0x7ffff000 (2,147,479,552) bytes, returning the number of bytes
actually transferred. (This is true on both 32-bit and 64-bit
systems.)

Also, it might be nice to add a note on top, that this module is for 'low level' IO interface, and that it is recommended to use regular file type for typical file operations (not io.FileIO) to avoid necessity of dealing limitations such as the one mentioned.

[serhiy-storchaka]

On Linux, write() (and similar system calls) will transfer at most 0x7ffff000 (2,147,479,552) bytes, returning the number of bytes
actually transferred. (This is true on both 32-bit and 64-bit
systems.)

This is platform-depending limitation. It can be be changed in future. In addition, there are other causes of writing not all data (see man 2 write).

Also, it might be nice to add a note on top, that this module is for 'low level' IO interface, and that it is recommended to use regular file type for typical file operations (not io.FileIO) to avoid necessity of dealing limitations such as the one mentioned.

This is not true for the module overall. And this is already documented for io.RawIOBase:

"""
Raw binary I/O typically provides low-level access to an underlying OS device or API, and does not try to encapsulate it in high-level primitives (this is left to Buffered I/O and Text I/O, described later in this page).
"""