hash(-1) is not behaving as expected

[Dyr-El]

Bug report

Bug description:

Of all integers I have tested (looped through a couple of millions) all tend to have the same value of hash as themselves (hash(i) == i) except for i == -1. (There might be some reason for this but it makes for some strange behaviour. For me it meant that hash((4, -1)) == hash((4, -2)) which was rather unexpected.)

assert hash(-1) == -2

CPython versions tested on:

3.9, 3.12

Operating systems tested on:

Linux, Windows

[TeamSpen210]

This is because in the C API, -1 is used to indicate the __hash__() method raised an exception. It's not a problem, since hash values aren't really guaranteed to be anything specific, other than if the values are equal the hash is equal. Note that very large values also aren't correct, since it wraps around. In general you should never assume that because hashes match the values are equal, you need to do that additional check.

[skirpichev]

Of all integers I have tested (looped through a couple of millions) all tend to have the same value of hash

It's not true actually. Check enough big integers.

except for i == -1.

It's not the only documented exception. But it's documented, please read: https://docs.python.org/3/library/stdtypes.html#hashing-of-numeric-types

[Dyr-El]

But in that case wouldn't it be more reasonable to have a totally different value? You want similar things to hash apart since that reduces the risk of a hash collision and gives you better performance. Normally it would be enough since it is close to even in distribution, but the values -1 and -2 are so commonly used that you really would like them to hash apart for performance reasons.

[skirpichev]

In principle, maybe it's not a bad idea. Though, this will introduce huge backward incompatibility and should be discussed first on https://discuss.python.org/c/ideas/6

but the values -1 and -2 are so commonly used

That point still require some justification;-) I.e. why -2 is worse than some "big enough" number, that fits to Py_hash_t. How much worse?

PS: I would guess, that -1 was chosen as an error indicator for __hash__() for consistency with other "numerical" parts of C-API (take look on PyLong_* or PyFloat_* methods). // CC @tim-one

[terryjreedy]

The primary use of hashes in Python is for the set items and dict keys. I believe that negative ints are less common in such uses than non-negative ints. Moreover, hash values are used mod k, where k is the current size of the internal array. -2 % k is k-2. If larger remainders are less common than smaller values and hash(-1)==-1 is excluded, -2 leading to k-2 is the negative hash value least likely to collide and hence the best choice.