Add sorting helpers for collections containing None values

[ncoghlan]

BPO	20630
Nosy	@tim-one, @rhettinger, @ncoghlan, @pitrou

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-02-15.00:06:16.265>
labels = ['type-feature']
title = 'Add sorting helpers for collections containing None values'
updated_at = <Date 2014-02-25.05:50:44.745>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2014-02-25.05:50:44.745>
actor = 'rhettinger'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2014-02-15.00:06:16.265>
creator = 'ncoghlan'
dependencies = []
files = []
hgrepos = []
issue_num = 20630
keywords = []
message_count = 9.0
messages = ['211251', '212103', '212121', '212123', '212144', '212145', '212146', '212147', '212164']
nosy_count = 6.0
nosy_names = ['tim.peters', 'rhettinger', 'ncoghlan', 'pitrou', 'Arfrever', 'cvrebert']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20630'
versions = ['Python 3.5']

[ncoghlan]

Currently, it's a bit annoying to sort collections containing "None" values in Python 3. While the default behaviour isn't going to change, it would be good to offer standard "none_first" and "none_last" helps (inspired by the SQL NULL FIRST and NULL LAST ordering control).

Suggested home: functools (since that is where the total_ordering class decorator already lives), but collections would also be a reasonable choice (as this feature mostly relates to sorting containers)

The minimal option (suggested by Peter Otten):

    def none_first(v):
        return v is not None, v

    def none_last(v):
        return v is None, v

A more complex alternative would be to provide general purpose SortsFirst and SortsLast singletons:

    @functools.total_ordering
    class _AlwaysLesser:
        def __eq__(self, other):
            return isinstance(other, _AlwaysLesser):
        def __lt__(self, other):
            return not isinstance(other, _AlwaysLesser):

    @functools.total_ordering
    class _AlwaysGreater:
        def __eq__(self, other):
            return isinstance(other, _AlwaysGreater):
        def __gt__(self, other):
            return not isinstance(other, _AlwaysGreater):

    SortsFirst = _AlwaysLesser()
    SortsLast = _AlwaysGreater()

    def none_first(v):
        return SortsFirst if v is None else v
    def none_last(v):
        return SortsLast if v is None else v

The advantage of the latter more complex approach is that you can embed the SortsFirst and SortsLast values inside a tuple as part of a more complex key, whereas the simple solution only handles the case where the entire value is None.

(Inspired by Chris Withers's python-dev thread: https://mail.python.org/pipermail/python-dev/2014-February/132332.html)

[rhettinger]

Currently, it's a bit annoying to sort collections
containing "None" values in Python 3

I think we should seriously consider whether to restore None's ability to compare with other entries. Removing this capability has been a major PITA and is an obstacle for people converting code to Python 3.

The need to create helper function work-arounds is a symptom, not a cure.

[tim-one]

I've haven't yet seen anyone complain about the inability to compare None except in the specific context of sorting. If it is in fact specific to sorting, then this specific symptom and "the problem" are in fact the same thing ;-)

[pitrou]

Both Nick's proposals look ok to me.

[ncoghlan]

It occurred to me the current names are a bit misleading when using
"reverse=True", so low/high is likely a better naming scheme than
first/last.

I think I'll propose a patch for six before doing anything to the standard
library - this is already an issue for some forward ports, so at least
adding a "none_low" sort key that is a no-op on Py2 makes sense.

[ncoghlan]

And in case that last comment worried anyone - I won't commit *anything*
related to this to the standard library until after creating a PyPI
"sortlib" module that includes both this and an "order_by_key" class
decorator, and we have consensus that the proposed changes are reasonable.

[rhettinger]

If it is in fact specific to sorting, then this specific symptom
and "the problem" are in fact the same thing ;-)

The first rule of tautology club is the first rule of tautology club ;-)

FWIW, we had to add a work-around for this in pprint._safe_key class. Without that work-around, it was difficult to work with JSON-style data hierarchies:

# wouldn't pprint() without the _safe_key() work-around:
temperatures = {'Jan': 25.2, 'Feb': 22.3, 'Mar': None, 'Apr': 19.1,
                'May': 22.2, 'Jun': None, 'July': 22.3}

I think this will be typical for the kind of issue people will encounter when using None as a placeholder for missing data.

FWIW, if None stays non-comparable, Nick's additions look fine to me. I just think it easier for everyone to restore None's universal comparability rather than adding work-arounds for the problems caused by removing that capability.

[ncoghlan]

I suspect if we'd thought of it back in the 3.0 or 3.1 time frame then
giving the Py3 None a consistent "sorts low" behaviour would have been more
likely.

At this stage of the Py3 life cycle, though, it seems simpler overall to
remain consistent with earlier Py3 releases.

[rhettinger]

At this stage of the Py3 life cycle, though,
it seems simpler overall to remain consistent
with earlier Py3 releases.

Given that so few users have converted, it is
simpler to become consistent with Py2.7 and
to not introduce compensating features.