Comparison of memoryview

[finswimmer]

BPO	20399
Nosy	@pitrou, @MojoVampire, @remilapeyre, @ironage

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-01-26.20:37:37.194>
labels = ['interpreter-core', 'type-feature', '3.8']
title = 'Comparison of memoryview'
updated_at = <Date 2019-04-02.19:16:07.171>
user = 'https://bugs.python.org/finswimmer'

bugs.python.org fields:

activity = <Date 2019-04-02.19:16:07.171>
actor = 'james stone'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2014-01-26.20:37:37.194>
creator = 'fin.swimmer'
dependencies = []
files = []
hgrepos = []
issue_num = 20399
keywords = []
message_count = 9.0
messages = ['209351', '209373', '209421', '209449', '333985', '333987', '334009', '334013', '339363']
nosy_count = 5.0
nosy_names = ['pitrou', 'fin.swimmer', 'josh.r', 'remi.lapeyre', 'james stone']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20399'
versions = ['Python 3.8']

[finswimmer]

Comparison by using memoryview seems not to work completely.

This works:
>>> memoryview(bytearray(range(5))) != memoryview(bytearray(range(5)))
False
>>> memoryview(bytearray(range(5))) == memoryview(bytearray(range(5)))
True

But:
>>> memoryview(bytearray(range(5))) < memoryview(bytearray(range(5)))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unorderable types: memoryview() < memoryview()

So memoryview cannot be used as a key-value for sorting

[pitrou]

This sounds reasonable, at least when the two memoryviews have the same shape.

[BreamoreBoy]

Some discussion here https://groups.google.com/forum/#!topic/dev-python/1D_iExlsva8

[skrah]

For integer sequences I think non-equality comparisons will be somewhat
confusing. Are the sequences little or big endian?

If we start to view bytes more like latin-1 again (PEP-461), it would
make sense for the 'B' and 'c' formats.

[MojoVampire]

The lack of support for the rich comparison operators on even the most basic memoryviews (e.g. 'B' format) means that memoryview is still a regression from some of the functionality buffer offered back in Python 2 ( https://stackoverflow.com/a/13574862/364696 ); you either need to convert back to bytes (losing the zero-copy behavior) or hand-write a comparator of your own to allow short-circuiting (which thanks to sort not taking cmp functions anymore, means you need to write it, then wrap it with functools.cmp_to_key if you're sorting, not just comparing individual items).

While I'll acknowledge it gets ugly to try to support every conceivable format, it seems like, at the very least, we could provide the same functionality as buffer for 1D contiguous memoryviews in the 'B' and 'c' formats (both of which should be handleable by a simple memcmp).

[pitrou]

Josh, could you say what your use case is?

[MojoVampire]

Not my use case specifically, but my link in the last post (repeated below) was to a StackOverflow answer to a problem where using buffer was simple and fast, but memoryview required annoying workarounds. Admittedly, in most cases it's people wanting to do this with strings, so in Python 3 it only actually works if you convert to bytes first (possibly wrapped in a memoryview cast to a larger width if you need to support ordinals outside the latin-1 range). But it seems a valid use case.

Examples where rich comparisons were needed include:

Effcient way to find longest duplicate string for Python (From Programming Pearls) - https://stackoverflow.com/a/13574862/364696 (which provides a side-by-side comparison of code using buffer and memoryview, and memoryview lost, badly)

strcmp for python or how to sort substrings efficiently (without copy) when building a suffix array - https://stackoverflow.com/q/2282579/364696 (a case where they needed to sort based on potentially huge suffixes of huge strings, and didn't want to end up copying all of them)

[pitrou]

If not wanting to support the whole gamut of PEP-3118 types, one could start by only providing ordered comparisons when format == 'B'.

[ironage]

I encountered this issue as well when using python 3.6.7 and psycopg2. Postgres tries to sort rows by the primary key field and if the returned type is a memoryview an the error is thrown: "TypeError: '<' not supported between instances of 'memoryview' and 'memoryview'." It may be more on the postgres guys to use a proper type with full comparison support. But I wanted to mention it here as a datapoint for a valid use case in the wild. More details on the specific issue at modoboa/modoboa-amavis#115