Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CmpTotal Bug #65

Closed
dhui opened this issue Jul 20, 2017 · 2 comments
Closed

CmpTotal Bug #65

dhui opened this issue Jul 20, 2017 · 2 comments

Comments

@dhui
Copy link
Contributor

dhui commented Jul 20, 2017

CmpTotal() does not properly compare equivalent Decimal values when the coefficient and exponent representations differ

Example:

package main

import (
    "github.com/cockroachdb/apd"
    "log"
)

func main() {
    a := apd.New(10, 0)
    b := apd.New(1, 1)
    log.Println(a)             // prints 10
    log.Println(b)             // prints 1E+1
    log.Println(a.CmpTotal(b)) // prints -1
    log.Println(a.Cmp(b))      // prints 0
    log.Println(b.CmpTotal(a)) // prints 1
    log.Println(b.Cmp(a))      // prints 0
}
@maddyblue
Copy link
Contributor

maddyblue commented Jul 20, 2017

This is working as intended (see the specification at http://speleotrove.com/decimal/damisc.html#refcotot, point number 2):

If two representations have the same numerical value then the exponent is taken into account; larger (more positive) exponents are higher in the ordering.

I do think that the documentation of Cmp and CmpTotal needs to be improved to discuss this, though, because that's clearly not obvious to users.

I've opened #66 to address the docs issue.

@dhui
Copy link
Contributor Author

dhui commented Jul 20, 2017

Thanks for the clarification! Time to update all of my code to use Cmp() instead of CmpTotal()!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants