Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

How do you specify precision correctly? #2

Closed
pixelfreak opened this Issue Mar 13, 2013 · 3 comments

Comments

Projects
None yet
2 participants

How do you specify precision correctly?

Num(1.5, 2) returns 0.15. Shouldn't it be 1.50?

What I'm trying to accomplish is this, when I do Num(1).div(Num(3)), I get 0. It seems to be rounding it based on the precision of the numbers being divided. But what I want is 0.33 or 0.3333 depending on the precision specified.

Thanks!

Owner

defunctzombie commented Mar 13, 2013

The precision argument might be broken.

num(1.5) will result in 1.5
num('1.00').div(3) == 0.33

I will check out the precision issue. And yes, it isn't so much as rounding as an issue with precision for the division operation. Unlike the other operations, the precision has to be limited.

From glancing through the code, it looks like the number should be pre-padded with zeros before the decimal point is applied? Just a guess, still mulling...

Owner

defunctzombie commented Mar 14, 2013

I believe the use of the two argument constructor (with precision) was
intended for when you have an integer that represents a decimal number. i.e.

1000 -> 1.000
num(1000, 3);

But I am not sure that type of api really makes much sense honestly haha. I
will think about it. Patches for what you think should be correct are
welcome for review :)

On Wed, Mar 13, 2013 at 6:24 PM, William Khoe notifications@github.comwrote:

From glancing through the code, it looks like the number is not padded
before the decimal point is applied? Maybe the num_str length should
always be precision + 1?


Reply to this email directly or view it on GitHubhttps://github.com/shtylman/node-num/issues/2#issuecomment-14872578
.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment