Fixed float maths #19

Merged
merged 1 commit into from Jan 18, 2012

Conversation

Projects
None yet
2 participants
Contributor

vicvega commented Dec 23, 2011

I fixed another case of imprecise maths with float numbers (remember #15)
This time is in the method significant (in lib/gruff/base.rb)
The original code

def significant(inc) # :nodoc:
  return 1.0 if inc == 0 # Keep from going into infinite loop
  factor = 1.0
  while (inc < 10)
    inc *= 10
    factor /= 10
  end
  while (inc > 100)
    inc /= 10
    factor *= 10
  end
  res = inc.floor * factor
  if (res.to_i.to_f == res)
    res.to_i
  else
    res
  end
end

doesn't work fine. Try it in console

ruby-1.9.2-p290 :150 > significant 1.2
=> 1.2000000000000002 

If you use BigDecimal instead of Float

def significant(i) # :nodoc:
  return 1.0 if i == 0 # Keep from going into infinite loop
  inc = BigDecimal(i.to_s)
  factor = BigDecimal('1.0')
  while (inc < 10)
    inc *= 10
    factor /= 10
  end

  while (inc > 100)
    inc /= 10
    factor *= 10
  end

  res = inc.floor * factor
  if (res.to_i.to_f == res)
    res.to_i
  else
    res
  end
end

Everything goes right

ruby-1.9.2-p290 :174 > r = significant 1.2
=> #<BigDecimal:9f0b7f8,'0.12E1',8(16)> 
ruby-1.9.2-p290 :175 > r.to_f
=> 1.2 

topfunky added a commit that referenced this pull request Jan 18, 2012

@topfunky topfunky merged commit 777cd05 into topfunky:master Jan 18, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment