Make IndexError messages for list more informative

[miguendes]

BPO	44166
Nosy	@rhettinger, @aroberge, @serhiy-storchaka, @vedgar, @TeamSpen210, @pablogsal, @remilapeyre, @miguendes
PRs	bpo-44166: Make IndexError messages for lists more informative #26207

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = <Date 2021-12-06.02:22:29.951>
created_at = <Date 2021-05-18.07:33:39.043>
labels = ['interpreter-core', 'type-feature', '3.11']
title = 'Make IndexError messages for list more informative'
updated_at = <Date 2021-12-06.02:33:35.609>
user = 'https://github.com/miguendes'

bugs.python.org fields:

activity = <Date 2021-12-06.02:33:35.609>
actor = 'rhettinger'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2021-12-06.02:22:29.951>
closer = 'rhettinger'
components = ['Interpreter Core']
creation = <Date 2021-05-18.07:33:39.043>
creator = 'miguendes'
dependencies = []
files = []
hgrepos = []
issue_num = 44166
keywords = ['patch']
message_count = 11.0
messages = ['393858', '393861', '393866', '393905', '407707', '407737', '407744', '407747', '407748', '407752', '407766']
nosy_count = 9.0
nosy_names = ['rhettinger', 'aroberge', 'serhiy.storchaka', 'veky', 'Spencer Brown', 'pablogsal', 'remi.lapeyre', 'miguendes', 'mishra.akash.myself']
pr_nums = ['26207']
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue44166'
versions = ['Python 3.11']

[miguendes]

I've noticed that in people will often ask for help on forums or stackoverflow to better understand the causes of
IndexErrors [1][2][3].

The error message line for an IndexError doesn’t give you great information.[1]

Currently, when we access a out of bounds position we get something along the lines:
Traceback (most recent call last):

>>> a = [0, 10, 20, 30, 40]
>>> a[5]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range
>>> a.pop(6)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: pop index out of range
>>> a[6] = 7
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list assignment index out of range
>>> a = []
>>> a[2]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range

These error messages are too broad and doesn't inform the current length of the list, nor the index that was fed to it.

To improve the debugging experience in both interactive and non-interactive code,
I propose to offer a richer and more informative error messages. I think this can help users, especially newer and less
experienced folks, debug their code.

>>> a = [0, 10, 20, 30, 40]
>>> a[5]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range, the len is 5 so index must be in -5..4, got 5
>>> a[-6]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range, the len is 5 so index must be in -5..4, got -6
>>> a.pop(6)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: pop index out of range, the len is 5 so index must be in -5..4, got 6
>>> a[6] = 7
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list assignment index out of range, the len is 5 so index must be in -5..4, got 6
>>> a = []
>>> a[2] = 0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list assignment index out of range, got index 2 but the list is empty

These proposed messages are inspired by other languages, with the difference that in Python we can have negative index.
So informing the allowed ranges is a plus.

These improvements are not restricted to list, so it can be applied to strings and tuples as well, or any other indexable
object.

I have a branch ready, looking forward to hearing from you!

[1] https://stackoverflow.com/questions/1098643/indexerror-list-index-out-of-range-and-python
[2] https://stackoverflow.com/questions/16005707/index-error-list-index-out-of-range-python
[3] https://realpython.com/python-traceback/#indexerror

[serhiy-storchaka]

The problem is that such exceptions are often used with the EAFP style. For example:

        while stop is None or i < stop:
            try:
                v = self[i]
                if v is value or v == value:
                    return i
            except IndexError:
                break
            i += 1

You caught IndexError and do something, but do not use the error message. Formatting error message takes a time. I am sure that error messages of a wast majority of IndexError exceptions are not used, just the type of the raised exception matters.

[pablogsal]

I share the same concern as Serhiy, see my comment in the PR:

#26207 (review)

[miguendes]

Thanks for your comments, folks! I really appreciate it.

I left a comment about this issue in the PR thread: #26207 (comment)

[mishraakashmyself]

While browsing some traceback error reporting i land on this solution threads.

Specially beginner face traceback error for

• KeyError
• IndexError (As mentioned in current message thread)
• ValueError
  ..

It is sometime self explanatory from error reports but may be it can help to design some error document in python to more generalize errors.

Found some of traceback error with some possible cause explanations. Sharing if this is useful.

https://github.com/pablorgarcia/Learn-to-program-with-Python-guide/blob/master/Understanding%20errors/IndexError-list-index-out-of-range.py

https://www.quizcure.com/python/traceback-most-recent-call-last-in-python#Keyerror-dictionary

https://stackoverflow.com/questions/64428188/deep-learning-chatbot-specific-index-error-list-index-out-of-range

[TeamSpen210]

One potential solution would be to add two Py_ssize_t to IndexError, storing the index and length along with the existing exception value. Then __str__() can append that to the message if set, perhaps having len be negative to signal they're not passed.

An issue though is that it might be backwards incompatible with classes trying to multiply inherit from other exceptions too. To deal with that it could put the values in the args tuple, though then it'd have to allocate the tuple and ints which might hurt perf.

[rhettinger]

I also share Serhiy's concerns and prefer that it be left as-is.

Conceptually, adding more information in the error message would make it more useful for debugging, but in practice, it would rarely be helpful. In teaching and coaching Python, I've learned that the intended, dominant effect of seeing "IndexError: list index out of range" is that a person understands that index is invalid because it is out of range. That is a helpful dominant impression. IMO it would be distracting and rarely helpful to focus on the specific index and length. The important point is that a nonsense lookup was being made.

[rhettinger]

Here's another way to think of it:

The call s[i] raising an IndexError isn't a numerical error, it is a conceptual error. Knowing that i==15 and len(s)==10 doesn't usually help resolve the problem. The fix typically isn't replacing s[i] with s[i - 5]. Usually the issue is usually deeper than that.

Note, this is different than KeyError where misspellings are common and where the fix can be indicated by knowing the key: d['missteak'] -> d['mistake'].

[vedgar]

fix typically isn't replacing s[i] with s[i - 5]

... especially since that will still raise IndexError (in case when i==15 and len(s)==10). ;-P