array compare is hideously slow #68888
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
assignee = None closed_at = <Date 2017-08-17.12:46:35.899> created_at = <Date 2015-07-23.23:01:14.149> labels = ['3.7', 'library', 'performance'] title = 'array compare is hideously slow' updated_at = <Date 2017-08-17.12:46:35.898> user = 'https://bugs.python.org/swanson'
activity = <Date 2017-08-17.12:46:35.898> actor = 'pitrou' assignee = 'none' closed = True closed_date = <Date 2017-08-17.12:46:35.899> closer = 'pitrou' components = ['Library (Lib)'] creation = <Date 2015-07-23.23:01:14.149> creator = 'swanson' dependencies =  files = ['46675'] hgrepos =  issue_num = 24700 keywords =  message_count = 8.0 messages = ['247234', '247236', '288661', '299811', '299860', '299900', '300415', '300416'] nosy_count = 6.0 nosy_names = ['rhettinger', 'pitrou', 'josh.r', 'swanson', 'Adrian Wielgosik', 'lou1306'] pr_nums = ['3009'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'performance' url = 'https://bugs.python.org/issue24700' versions = ['Python 3.7']
The text was updated successfully, but these errors were encountered:
Comparing two array.array objects is much slower than it ought to be.
The whole point of array.array is to be efficient:
But comparing them is orders of magnitude less efficient than comparing tuples or lists of numbers. It would seem that array's __eq__ is converting every single number into an int, float, etc. first instead of just comparing the arrays in their native format.
from timeit import timeit setup = ''' from array import array a = array("I", %s) ac = a.__copy__ b = ac() t = tuple(a) u = t[:1] + t[1:] ''' for init in ("*10", "[0xffffffff]*10", "*1000", "[0xffffffff]*1000"): print("\n", init) for action in ("ac()", "a == b", "a == ac()", "t == u"): print(("%6.2f" % timeit(action, setup % init)), action)
You're correct about what is going on; aside from bypassing a bounds check (when not compiled with asserts enabled), the function it uses to get each index is the same as that used to implement indexing at the Python layer. It looks up the getitem function appropriate to the type code over and over, then calls it to create the PyLongObject and performs a rich compare.
The existing behavior is probably necessary to work with array subclasses, but it's also incredibly slow as you noticed. Main question is whether to keep the slow path for subclasses, or (effectively) require that array subclasses overriding __getitem__ also override he rich comparison operators to make them work as expected.
For cases where the signedness and element size are identical, it's trivial to acquire readonly buffers for both arrays and directly compare the memory (with memcmp for EQ/NE or size 1 elements, wmemcmp for appropriately sized wider elements, and simple loops for anything else).
I noticed the issue is still there in Python 3.6.
I would argue that _integerness_ sholud also be identical: otherwise
Added a PR with a fast path that triggers when compared arrays store values of the same type. In this fast path, no Python objects are created. For big arrays the runtime reduction can reach 50-100x.
It's possible to optimize the comparison loop a bit more - for example array('B') comparison could use memcmp and become extra 10x faster, and other types could receive similar treatment - but I wanted to keep the code relatively simple.
Test Before After % of old time