Skip to content

behavior of cmp() on NaNs? #5290

Open
jiahao opened this Issue Jan 3, 2014 · 7 comments

4 participants

@jiahao
The Julia Language member
jiahao commented Jan 3, 2014

1. Inconsistent logic between lexcmp and nextfloat

julia> nextfloat(Inf)
NaN

julia> lexcmp(Inf,NaN)
ERROR: InexactError()
 in lexcmp at operators.jl:32

julia> lexcmp(Inf,Inf)
ERROR: InexactError()
 in lexcmp at operators.jl:32

Given the definition of lexcmp according to the help text, throwing an error is the right thing to do given that >, == and < are all supposed to be false when either operand is a NaN and so none of the standard return values (1, 0, -1) can be correct. However, it probably shouldn't throw an InexactError, but something else instead (probably DomainError).

In contrast, I don't see a meaningful sense in which NaN is the "next" number after Inf. The standard rule for computing with floating-point Infs is to interpret it as a limit, in which case the correct result of nextfloat(Inf) should be Inf and not NaN (and similarly prevfloat(-Inf) returning -Inf. (Going by this rule, lexcmp(Inf,Inf) does correctly fail since there is a coalescence of x<x+eps(), x==x+0 and x>x-eps() as x->Inf, so that there is no single limiting value.)

2. Inconsistent behavior of nextfloat and prevfloat at different precisions

If we take the argument in the preceding paragraph to be correct, then in the following, only nextfloat(Inf16) is correct:

 julia> nextfloat(Inf) #should be Inf
NaN

julia> nextfloat(Inf32) #should be Inf32
NaN32

julia> nextfloat(Inf16) #should be Inf16
Inf16

julia> prevfloat(-Inf) #should be -Inf
NaN

julia> prevfloat(-Inf32) #should be -Inf32
NaN32

julia> prevfloat(-Inf16) #should be -Inf16
Inf16
@JeffBezanson JeffBezanson added a commit that referenced this issue Jan 3, 2014
@JeffBezanson JeffBezanson fix Float16 inf and nan testing. ref #5290
correct help for frexp
33bdd92
@JeffBezanson
The Julia Language member

In my view, lexcmp implements a total order, so ordinary NaN semantics don't apply. And, for example, we sort NaNs after everything else.

@JeffBezanson
The Julia Language member

Also, currently cmp and lexcmp are the same for types with a canonical total order, including floating point types. But maybe cmp should require the arguments to be non-NaN, leaving the more permissive behavior to lexcmp.

@jiahao
The Julia Language member
jiahao commented Jan 3, 2014

What you just wrote seems to be the opposite of what is implemented, since it sounds like you want lexcmp(Inf,NaN) == -1. In contrast nextfloat and prevfloat have names that suggest standards-compliant floating-point behavior, but don't behave like logical consequences of < and > on floats.

I'm fine with having non-canonical behavior, but at the very least the functions that will do so should be documented.

@JeffBezanson
The Julia Language member

IEEE 754 defines a standard total order on floats where NaN is greater than Inf, so isless(Inf,NaN)==true and lexcmp(Inf,NaN)==-1 are correct.

Unfortunately the IEEE 754 order also puts -NaN less than everything, which is not how we prefer to sort things. Normally the sign of NaN is not interpreted, but has specified behavior for copy, negate, abs, copySign, and totalOrder. I would argue that making all NaNs isequal is much more useful though; the only functions that might distinguish them are the comparison functions themselves.

IEEE 754 does specify that nextUp(Inf) == Inf, as you say, so we should probably change that.

@jiahao
The Julia Language member
jiahao commented Jan 3, 2014

So is the claim that lexcmp on two floating point operands should implement the totalOrder of IEEE 754-2008, Sec. 5.10? It sounds like the ordering you would get by casting all the bits to a signed integer of the same bitsize and ordering those.

If the answer to my question is 'yes', then there are further behaviors that need fixing:

julia> lexcmp(-0.0,0.0) #c.f. IEEE 754-2008 5.10.c.1: "totalOrder(−0, +0) is true."
0

julia> lexcmp(-0.0,float32(-0.0)) #??? I don't think the standard says anything about comparing different precisions - just let type promotion handle this?
0

And then there is the business of total ordering numerical equivalent bit patterns...

@JeffBezanson JeffBezanson referenced this issue Jan 3, 2014
@JeffBezanson JeffBezanson fix isless and cmp/lexcmp for floating point
for now cmp() uses the total ordering, but we might change it to give a
DomainError for NaN arguments
2343ba0
@StefanKarpinski
The Julia Language member

This should be addressed at the same time as we change isless and hash for numbers. Any changes before then are just likely to have to change yet again.

@simonbyrne

FWIW, future C standards TS 18661-1:2014 (free draft) will offer a totalorder function that matches the IEEE-754 totalOrder predicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.