Truncate __len__() at sys.maxsize

[abalkin]

BPO	2723
Nosy	@rhettinger, @gpshead, @abalkin, @pitrou, @vstinner, @benjaminp
Files	len.diff: patch against py3k revision 62564 len_message.patch: Change OverflowError message when len > sys.maxsize (py3k r62990)

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-01-14.00:01:49.964>
created_at = <Date 2008-04-30.04:36:29.421>
labels = ['interpreter-core', 'type-feature']
title = 'Truncate __len__() at sys.maxsize'
updated_at = <Date 2009-01-14.00:01:49.962>
user = 'https://github.com/abalkin'

bugs.python.org fields:

activity = <Date 2009-01-14.00:01:49.962>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2009-01-14.00:01:49.964>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2008-04-30.04:36:29.421>
creator = 'belopolsky'
dependencies = []
files = ['10143', '10270']
hgrepos = []
issue_num = 2723
keywords = ['patch']
message_count = 12.0
messages = ['65989', '65994', '66001', '66013', '66046', '66459', '66573', '66687', '66688', '72198', '78285', '79810']
nosy_count = 9.0
nosy_names = ['rhettinger', 'gregory.p.smith', 'belopolsky', 'pitrou', 'vstinner', '_doublep', 'benjamin.peterson', 'rbp', 'hagen']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue2723'
versions = ['Python 3.0']

[abalkin]

On Tue, Apr 29, 2008 at 10:36 PM, Guido van Rossum <guido@python.org>
wrote:
..

Let's also fix __len__() so that it returns sys.{maxint,maxsize} when
the result doesn't fit in a Py_ssize_t.

http://mail.python.org/pipermail/python-3000/2008-April/013343.html

With attached patch given

class x:
    def __len__(self):
        return 2**100

len(x()) and len(range(2**100)) will return sys.maxsize.

[rhettinger]

Wouldn't it be better to raise OverflowError or somesuch?

[abalkin]

On Wed, Apr 30, 2008 at 2:32 AM, Raymond Hettinger
<report@bugs.python.org> wrote:

Wouldn't it be better to raise OverflowError or somesuch?

Thats what the current code does. I don't know what Guido's full
rationale is, but I guess the idea is that len(..) is not supposed to
raise an exception on sizeable objects.

Here is a quote from another message:

"""
__len__ will always be problematic when there are more values than can
be counted in a signed C long; maybe we should do what the Java
collections package does: for once, Java chooses practicality over
purity, and simply states that if the length doesn't fit, the largest
number that does fit is returned (i.e. for us that would be
sys.maxsize in 3.0, sys.maxint in 2.x).
"""
-- Guido van Rossum, 2008-04-30
http://mail.python.org/pipermail/python-3000/2008-April/013340.html

I suspect, however, that part of Java's motivation for this behavior
is that exceptions need to be declared and declaring the length method
as throwing OverflowError would make many programmers very unhappy.

[pitrou]

Gasp, having len() return something else than the true container size
sounds horrible. At least raising OverflowError makes it clear that
something wrong is going on...

[benjaminp]

If you're interested I asked a Java news group:
http://groups.google.com/group/comp.lang.java.programmer/browse_thread/thread/fddbc3b1f9fec125#

[pitrou]

Well apparently the Java guys think raising an exception would have been
a much better idea than the behaviour they are stuck with.
There's also in interesting proposal there:

""" The ReturnValueTooBigException could even have a method declared as
"long size()" that reports the actual size of the collection. """

[rbp]

I think returning sys.{maxint,maxsize} in this case is a plain lie.
That's not practicality, that's giving back false information.

Barring drastic language changes (such as having objects representing
"infinity" or "greater than" - which, of course, won't happen), I think
the current behaviour of raising an exception is the correct one. But,
although I think OverflowError is good enough, the current exception
message is a bit cryptic, especially for anyone who doesn't know C:

"""OverflowError: Python int too large to convert to C ssize_t"""

I've attached a simple patch (modified from Alexander's) to raise:

"""OverflowError: Length too large"""

(I thought about "Object too large", but our problem is actually that
the *length* itself is too large)

[gpshead]

Agreed, having it lie about the size is the WORST possible behavior
because it will silently hide problems. Lets not do that.

But I must've missed something, why can't __len__ return the correct
value? Merely because range() is broken and might use it as input?
Thats no excuse. Fix range().

[abalkin]

On Sun, May 11, 2008 at 6:38 PM, Gregory P. Smith
<report@bugs.python.org> wrote:
..

But I must've missed something, why can't __len__ return the correct
value?

The problem is the C signature of the sq_length slot:

typedef Py_ssize_t (*lenfunc)(PyObject *);

[doublep]

I'm also absolutely against having len() lying to me. This would be a
welcome to bump into some other hideous error later, e.g. discarding
part of data as I'd think it wasn't there. Better raise an exception as
now, at least then programmers will know something is wrong and have a
chance to workaround, etc.

[vstinner]

Most people disagree with the original idea (len.diff: truncate the
length to sys.maxsize).

I don't like rbp's patch: replace verbose error message ("Python int
too large to convert to C ssize_t") by a shorter message ("Length too
large"). If I want to debug my program, I prefer longer error
messages. The original message contains the C type: ssize_t, useful
information.

I dislike both patches. Can we close this issue with
resolution=invalid?

[vstinner]

Initial proposition (len.diff) was rejected. If you want to change
len() behaviour, reopen this issue or open another one.