Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Fixed float maths #19

Merged
merged 1 commit into from

2 participants

@vicvega

I fixed another case of imprecise maths with float numbers (remember #15)
This time is in the method significant (in lib/gruff/base.rb)
The original code

def significant(inc) # :nodoc:
  return 1.0 if inc == 0 # Keep from going into infinite loop
  factor = 1.0
  while (inc < 10)
    inc *= 10
    factor /= 10
  end
  while (inc > 100)
    inc /= 10
    factor *= 10
  end
  res = inc.floor * factor
  if (res.to_i.to_f == res)
    res.to_i
  else
    res
  end
end

doesn't work fine. Try it in console

ruby-1.9.2-p290 :150 > significant 1.2
=> 1.2000000000000002 

If you use BigDecimal instead of Float

def significant(i) # :nodoc:
  return 1.0 if i == 0 # Keep from going into infinite loop
  inc = BigDecimal(i.to_s)
  factor = BigDecimal('1.0')
  while (inc < 10)
    inc *= 10
    factor /= 10
  end

  while (inc > 100)
    inc /= 10
    factor *= 10
  end

  res = inc.floor * factor
  if (res.to_i.to_f == res)
    res.to_i
  else
    res
  end
end

Everything goes right

ruby-1.9.2-p290 :174 > r = significant 1.2
=> #<BigDecimal:9f0b7f8,'0.12E1',8(16)> 
ruby-1.9.2-p290 :175 > r.to_f
=> 1.2 
@topfunky topfunky merged commit 777cd05 into topfunky:master
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Dec 23, 2011
  1. @vicvega

    fixed float accuracy

    vicvega authored
This page is out of date. Refresh to see the latest.
Showing with 6 additions and 5 deletions.
  1. +6 −5 lib/gruff/base.rb
View
11 lib/gruff/base.rb
@@ -699,8 +699,8 @@ def draw_line_markers
@d = @d.fill(@marker_color)
@d = @d.line(@graph_left, y, @graph_right, y)
- marker_label = (BigDecimal(index.to_s) * BigDecimal(@increment.to_s) +
- BigDecimal(@minimum_value.to_s)).to_f
+ marker_label = BigDecimal(index.to_s) * BigDecimal(@increment.to_s) +
+ BigDecimal(@minimum_value.to_s)
unless @hide_line_numbers
@d.fill = @font_color
@@ -979,9 +979,10 @@ def min(data_point, index) # :nodoc:
data_point
end
- def significant(inc) # :nodoc:
- return 1.0 if inc == 0 # Keep from going into infinite loop
- factor = 1.0
+ def significant(i) # :nodoc:
+ return 1.0 if i == 0 # Keep from going into infinite loop
+ inc = BigDecimal(i.to_s)
+ factor = BigDecimal('1.0')
while (inc < 10)
inc *= 10
factor /= 10
Something went wrong with that request. Please try again.