New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Abort in rshift #3620

Closed
clayton-shopify opened this Issue Apr 19, 2017 · 0 comments

Comments

Projects
None yet
1 participant
@clayton-shopify
Contributor

clayton-shopify commented Apr 19, 2017

Dinko Galetic & Denis Kasak (https://hackerone.com/dgaletic) reported that the following input demonstrates a crash (when MRuby is built with 32-bit integer size):

0 << -2147483648

This occurs because fix_lshift negates width, which leaves it negative if it happens to be -2^31:

return rshift(val, -width);

They proposed the following patch:

diff --git a/src/numeric.c b/src/numeric.c
index 2117fcc3..125ce80f 100644
--- a/src/numeric.c
+++ b/src/numeric.c
@@ -996,6 +996,9 @@ fix_lshift(mrb_state *mrb, mrb_value x)
   }
   val = mrb_fixnum(x);
   if (width < 0) {
+    if (width < -NUMERIC_SHIFT_WIDTH_MAX) {
+        width = -NUMERIC_SHIFT_WIDTH_MAX;
+    }
     return rshift(val, -width);
   }
   return lshift(mrb, val, width);

@matz matz closed this in 262fbaf Apr 20, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment