Fixing RangeError with code from 1.7 branch

With my previous commit, RangeError: bignum too big to convert into `long' was the result when running (0...2**64).max

This commit ports handling for non-RubyFixnums from the 1.7.20 branch.

Addresses, jruby#3118

core/src/main/java/org/jruby/RubyRange.java

@@ -659,7 +659,11 @@ public IRubyObject max(ThreadContext context, Block block) {
             if (!(begin instanceof RubyInteger)) {
                 throw context.runtime.newTypeError("cannot exclude end value with non Integer begin value");
             }
-            rangeEnd = RubyFixnum.newFixnum(context.runtime, ((RubyInteger) end).getLongValue() - 1);
+            if (end instanceof RubyFixnum) {
+                rangeEnd = RubyFixnum.newFixnum(context.runtime, ((RubyFixnum)end).getLongValue() - 1);
+            else {
+                rangeEnd =  end.callMethod(context, "-", RubyFixnum.one(context.runtime));
+            }
         } else {
             rangeEnd = end;
         }