in lexcmp at operators.jl:32
in lexcmp at operators.jl:32
Given the definition of lexcmp according to the help text, throwing an error is the right thing to do given that >, == and < are all supposed to be false when either operand is a NaN and so none of the standard return values (1, 0, -1) can be correct. However, it probably shouldn't throw an InexactError, but something else instead (probably DomainError).
In contrast, I don't see a meaningful sense in which NaN is the "next" number after Inf. The standard rule for computing with floating-point Infs is to interpret it as a limit, in which case the correct result of nextfloat(Inf) should be Inf and not NaN (and similarly prevfloat(-Inf) returning -Inf. (Going by this rule, lexcmp(Inf,Inf) does correctly fail since there is a coalescence of x<x+eps(), x==x+0 and x>x-eps() as x->Inf, so that there is no single limiting value.)
If we take the argument in the preceding paragraph to be correct, then in the following, only nextfloat(Inf16) is correct:
julia> nextfloat(Inf) #should be Inf
julia> nextfloat(Inf32) #should be Inf32
julia> nextfloat(Inf16) #should be Inf16
julia> prevfloat(-Inf) #should be -Inf
julia> prevfloat(-Inf32) #should be -Inf32
julia> prevfloat(-Inf16) #should be -Inf16
fix Float16 inf and nan testing. ref #5290
correct help for frexp
In my view, lexcmp implements a total order, so ordinary NaN semantics don't apply. And, for example, we sort NaNs after everything else.
Also, currently cmp and lexcmp are the same for types with a canonical total order, including floating point types. But maybe cmp should require the arguments to be non-NaN, leaving the more permissive behavior to lexcmp.
What you just wrote seems to be the opposite of what is implemented, since it sounds like you want lexcmp(Inf,NaN) == -1. In contrast nextfloat and prevfloat have names that suggest standards-compliant floating-point behavior, but don't behave like logical consequences of < and > on floats.
lexcmp(Inf,NaN) == -1
I'm fine with having non-canonical behavior, but at the very least the functions that will do so should be documented.
IEEE 754 defines a standard total order on floats where NaN is greater than Inf, so isless(Inf,NaN)==true and lexcmp(Inf,NaN)==-1 are correct.
Unfortunately the IEEE 754 order also puts -NaN less than everything, which is not how we prefer to sort things. Normally the sign of NaN is not interpreted, but has specified behavior for copy, negate, abs, copySign, and totalOrder. I would argue that making all NaNs isequal is much more useful though; the only functions that might distinguish them are the comparison functions themselves.
IEEE 754 does specify that nextUp(Inf) == Inf, as you say, so we should probably change that.
nextUp(Inf) == Inf
So is the claim that lexcmp on two floating point operands should implement the totalOrder of IEEE 754-2008, Sec. 5.10? It sounds like the ordering you would get by casting all the bits to a signed integer of the same bitsize and ordering those.
If the answer to my question is 'yes', then there are further behaviors that need fixing:
julia> lexcmp(-0.0,0.0) #c.f. IEEE 754-2008 5.10.c.1: "totalOrder(−0, +0) is true."
julia> lexcmp(-0.0,float32(-0.0)) #??? I don't think the standard says anything about comparing different precisions - just let type promotion handle this?
And then there is the business of total ordering numerical equivalent bit patterns...
fix isless and cmp/lexcmp for floating point
for now cmp() uses the total ordering, but we might change it to give a
DomainError for NaN arguments
make nextfloat/prevfloat saturate at Inf, as per the IEEE standard
This should be addressed at the same time as we change isless and hash for numbers. Any changes before then are just likely to have to change yet again.
throw a DomainError for cmp on unordered floats. ref #5290
FWIW, future C standards TS 18661-1:2014 (free draft) will offer a totalorder function that matches the IEEE-754 totalOrder predicate.