You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
CmpTotal() does not properly compare equivalent Decimal values when the coefficient and exponent representations differ
Example:
package main
import (
"github.com/cockroachdb/apd""log"
)
funcmain() {
a:=apd.New(10, 0)
b:=apd.New(1, 1)
log.Println(a) // prints 10log.Println(b) // prints 1E+1log.Println(a.CmpTotal(b)) // prints -1log.Println(a.Cmp(b)) // prints 0log.Println(b.CmpTotal(a)) // prints 1log.Println(b.Cmp(a)) // prints 0
}
The text was updated successfully, but these errors were encountered:
If two representations have the same numerical value then the exponent is taken into account; larger (more positive) exponents are higher in the ordering.
I do think that the documentation of Cmp and CmpTotal needs to be improved to discuss this, though, because that's clearly not obvious to users.
CmpTotal()
does not properly compare equivalentDecimal
values when the coefficient and exponent representations differExample:
The text was updated successfully, but these errors were encountered: