[easy C] array.array.index() method downcasts Py_ssize_t to long

[WildCard65]

BPO	41085
Nosy	@pfmoore, @vstinner, @tjguk, @zware, @zooba, @WildCard65, @miss-islington
PRs	bpo-41085: Fixed 'array.array.index' downcasting 'Py_ssize_t' to 'long'. #21071 [3.9] bpo-41085: Fix array.array.index() on 64-bit Windows (GH-21071) #21076 [3.8] bpo-41085: Fix array.array.index() on 64-bit Windows (GH-21071) #21077
Files	test_output.log: Log of tests that were ran at time of upload.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-06-23.13:52:54.522>
created_at = <Date 2020-06-23.00:44:44.317>
labels = ['easy', 'type-bug', '3.8', '3.9', '3.10', 'library']
title = '[easy C] array.array.index() method downcasts Py_ssize_t to long'
updated_at = <Date 2020-06-23.14:37:17.548>
user = 'https://github.com/WildCard65'

bugs.python.org fields:

activity = <Date 2020-06-23.14:37:17.548>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-06-23.13:52:54.522>
closer = 'WildCard65'
components = ['Library (Lib)']
creation = <Date 2020-06-23.00:44:44.317>
creator = 'WildCard65'
dependencies = []
files = ['49257']
hgrepos = []
issue_num = 41085
keywords = ['patch', 'easy (C)', 'newcomer friendly']
message_count = 9.0
messages = ['372135', '372140', '372153', '372155', '372156', '372164', '372166', '372167', '372175']
nosy_count = 8.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'SilentGhost', 'zach.ware', 'steve.dower', 'WildCard65', 'miss-islington']
pr_nums = ['21071', '21076', '21077']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41085'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[WildCard65]

Here's the verbose stack trace of the failing test:

======================================================================
FAIL: test_index (test.test_array.LargeArrayTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "L:\GIT\cpython\lib\test\support\__init__.py", line 837, in wrapper
    return f(self, maxsize)
  File "L:\GIT\cpython\lib\test\test_array.py", line 1460, in test_index
    self.assertEqual(example.index(11), size+3)
AssertionError: -2147483645 != 2147483651

[SilentGhost]

This looks like an overflow failure in a bigmem test (though the given limit 24Gb should accommodate that?). Do bigmem tests work on Windows?

William, you've made some sort of modifications to the versioned files, what are those?

[WildCard65]

The only modification I made was to "rt.bat" to have the value of '-u' work properly.

[vstinner]

It's Python 3.10 on Windows built in 64-bit with Visual Studio.

Extract of test_output.log:

L:\GIT\cpython\PCbuild>"L:\GIT\cpython\PCbuild\amd64\python_d.exe" -u -Wd -E -bb -m test -u all,-curses -v -M 24Gb --header

== CPython 3.10.0a0 (heads/master-dirty:c96d00e88e, Jun 22 2020, 19:15:48) [MSC v.1926 64 bit (AMD64)]
== Windows-10-10.0.19041-SP0 little-endian
== cwd: L:\GIT\cpython\build\test_python_20756
== CPU count: 8
== encodings: locale=cp1252, FS=utf-8
(...)
test_index (test.test_array.LargeArrayTest) ...
... expected peak memory use: 4.2G
FAIL
(...)
======================================================================
FAIL: test_index (test.test_array.LargeArrayTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "L:\GIT\cpython\lib\test\support\__init__.py", line 837, in wrapper
    return f(self, maxsize)
  File "L:\GIT\cpython\lib\test\test_array.py", line 1460, in test_index
    self.assertEqual(example.index(11), size+3)
AssertionError: -2147483645 != 2147483651

[vstinner]

That's a bug in array_array_index() which downcasts "Py_ssize_t i" to long:

static PyObject *
array_array_index(arrayobject *self, PyObject *v)
/*[clinic end generated code: output=d48498d325602167 input=cf619898c6649d08]*/
{
    Py_ssize_t i;

    for (i = 0; i < Py_SIZE(self); i++) {
        PyObject *selfi;
        int cmp;

        selfi = getarrayitem((PyObject *)self, i);
        if (selfi == NULL)
            return NULL;
        cmp = PyObject_RichCompareBool(selfi, v, Py_EQ);
        Py_DECREF(selfi);
        if (cmp > 0) {
            return PyLong_FromLong((long)i); // <===== HERE ===
        }
        else if (cmp < 0)
            return NULL;
    }
    PyErr_SetString(PyExc_ValueError, "array.index(x): x not in array");
    return NULL;
}

[vstinner]

New changeset 1d3dad5 by WildCard65 in branch 'master':
bpo-41085: Fix array.array.index() on 64-bit Windows (GH-21071)
1d3dad5

[miss-islington]

New changeset c6e24e7 by Miss Islington (bot) in branch '3.8':
bpo-41085: Fix array.array.index() on 64-bit Windows (GH-21071)
c6e24e7

[miss-islington]

New changeset 92f8b48 by Miss Islington (bot) in branch '3.9':
bpo-41085: Fix array.array.index() on 64-bit Windows (GH-21071)
92f8b48

[vstinner]

Thanks William Pickard for the bug report and the fix!