os.lseek() and FileIO.seek() does not support offset larger than 2^63-1

[KuberanNaganathan]

BPO	12545
Nosy	@jcea, @pitrou, @vstinner
Files	lseek_negative.diff lseek_unsigned.patch lseek_unsigned_test.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2011-07-13.03:29:18.643>
labels = ['3.10', '3.9', 'expert-IO']
title = 'os.lseek() and FileIO.seek() does not support offset larger than 2^63-1'
updated_at = <Date 2020-11-17.21:36:19.610>
user = 'https://bugs.python.org/KuberanNaganathan'

bugs.python.org fields:

activity = <Date 2020-11-17.21:36:19.610>
actor = 'neologix'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['IO']
creation = <Date 2011-07-13.03:29:18.643>
creator = 'Kuberan.Naganathan'
dependencies = []
files = ['22719', '30487', '30490']
hgrepos = []
issue_num = 12545
keywords = ['patch']
message_count = 21.0
messages = ['140222', '140223', '140250', '140251', '140253', '140261', '140270', '140272', '140289', '140605', '140715', '140721', '140723', '140807', '140904', '140923', '190724', '190725', '190727', '190728', '190729']
nosy_count = 4.0
nosy_names = ['jcea', 'pitrou', 'vstinner', 'Kuberan.Naganathan']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue12545'
versions = ['Python 3.9', 'Python 3.10']

[KuberanNaganathan]

The lseek function can legitimately return a code less then zero ( except for -1 ) when seeking beyond an offset of 2^63. This behavior should be supported in order to permit the python interpreter to seek in files with valid data at locations greater than or equal to 2^63. This can happen in a sparse file or in the /proc file system address space file.

The fix is simple. In the posix_lseek function check for result != -1 instead of checking for result < 0 in return code checks of the value returned by lseek.

[KuberanNaganathan]

In addition I would like the posix_lseek function to accept a value larger than 2^63 as a seek offset. Currently I seem to need to seek multiple times within a file to achieve an offset larger than 2^63 ( assuming the return type check on the return value of lseek is fixed. )

[pitrou]

In addition I would like the posix_lseek function to accept a value
larger than 2^63 as a seek offset

How would it work? The C lseek() takes a signed (64-bit) offset as argument, so we would have to call it multiple times anyway.

[vstinner]

"... A negative file offset may be valid for some devices in some implementations.

The POSIX.1-1990 standard did not specifically prohibit lseek() from returning a negative offset. Therefore, an application was required to clear errno prior to the call and check errno upon return to determine whether a return value of ( off_t)-1 is a negative offset or an indication of an error condition. The standard developers did not wish to require this action on the part of a conforming application, and chose to require that errno be set to [EINVAL] when the resulting file offset would be negative for a regular file, block special file, or directory."

Extract of:
http://pubs.opengroup.org/onlinepubs/009695399/functions/lseek.html

[neologix]

In the posix_lseek function check for result != -1 instead of checking
for result < 0 in return code checks of the value returned by lseek.

+1
This can be useful when parsing /dev/mem on x86_64, for example.

In addition I would like the posix_lseek function to accept a value
larger than 2^63 as a seek offset

Like Antoine, I'm a little bit puzzled here.

[vstinner]

How would it work? The C lseek() takes a signed (64-bit) offset
as argument, so we would have to call it multiple times anyway.

Python does already call multiple times the same function if the input is larger than the type used by the function. Some examples:

• mbcs codec: work on chunk on INT_MAX bytes (input size type: Py_ssize_t)
• zlib: crc32() works on chunk on UINT_MAX bytes (input size type: Py_ssize_t)

We have also functions processing fewer data if the function cannot handle all data: FileIO.write() clamps the buffer size of INT_MAX on Windows for example.

If we call lseek() multiple times, the implementation will depend on the whence value: SEEK_SET will use SEEK_SET and then SEEK_CUR. For SEEK_CUR, all calls can use SEEK_CUR. For SEEK_END... hum? SEEK_END and then SEEK_CUR?

I vote -1 for this feature because I bet that the behaviour of lseek() with an file position > 2^63 or offset > 2^63 highly depend on the platform (kernel, libc) version.

You can implement it in Python for your usecase. I prefer to keep a thin wrapper with an known and reliable behaviour.

[KuberanNaganathan]

Im most familiar with solaris. Atleast on solaris lseek interprets its
signed arg as the unsigned value with the same bit pattern. I cannot be
certain that this is common across other operating systems.
On Jul 13, 2011 8:09 AM, "Antoine Pitrou" <report@bugs.python.org> wrote:

Antoine Pitrou <pitrou@free.fr> added the comment:

> In addition I would like the posix_lseek function to accept a value
> larger than 2^63 as a seek offset

How would it work? The C lseek() takes a signed (64-bit) offset as
argument, so we would have to call it multiple times anyway.

----------
nosy: +neologix, pitrou
versions: +Python 3.2, Python 3.3

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue12545\>

---

[pitrou]

Im most familiar with solaris. Atleast on solaris lseek interprets its
signed arg as the unsigned value with the same bit pattern.

Only with SEEK_SET, I assume?

[KuberanNaganathan]

I did some testing on this. Solaris seems to treat files as a large
circular buffer of size 2^64 with respect to seeks in the /proc file system
address space file. No error results with seeks of greater than 2^63 with
any of SEEK_SET, SEEK_CUR, SEEK_END. This is not true of a regular file
where I get errno=22 when seeking to an offset greater than or equal to
2^63.

So even on solaris the behavior seems to be filesystem dependent.

[neologix]

So even on solaris the behavior seems to be filesystem dependent.

So I'd suggest forgetting about this part.
Would you like to write a patch for the first point?

[KuberanNaganathan]

Hi. I can submit a patch for the first part. Should I submit on this issue tracker item?

[neologix]

Hi.  I can submit a patch for the first part.  Should I submit on this issue tracker item?

Sure.

[vstinner]

So I'd suggest forgetting about this part.

If someone really wants this feature (relative seek of more than 2^63), he/she should open a new issue.

This issue remembers me a mktime() issue: mktime() may return -1 even if it is not an error. time_mktime() uses now a sentinel to detect an error:

buf.tm_wday = -1;  /* sentinel; original value ignored */
tt = mktime(&buf);
/* Return value of -1 does not necessarily mean an error, but tm_wday
 * cannot remain set to -1 if mktime succeeded. */
if (tt == (time_t)(-1) && buf.tm_wday == -1) /* OverflowError */

For lseek, we can rely on errno. Try something like that:

errno = 0;
offset = lseek(...);
if (offset == (off_t)-1 && errno) /* error */

We can write a test using a sparse file... Or maybe a mmap object?

[KuberanNaganathan]

Hi. I'm unable ( or have to jump through lots of hoops ) to submit this patch myself for regulatory reasons. Can someone else submit this please? Thanks.