Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bigarray reshape wrong for 64 bit systems #5511

vicuna opened this issue Feb 20, 2012 · 2 comments


None yet
1 participant
Copy link

commented Feb 20, 2012

Original bug ID: 5511
Reporter: gerd
Status: closed (set by @xavierleroy on 2013-08-31T10:48:35Z)
Resolution: fixed
Priority: normal
Severity: minor
Version: 3.12.1
Fixed in version: 3.13.0+dev
Category: otherlibs
Related to: #4457

Bug description

Just found these lines in bigarray_stubs.c, in caml_ba_reshape:

if (dim[i] < 0 || dim[i] > 0x7FFFFFFFL)
caml_invalid_argument("Bigarray.reshape: negative dimension");

The second condition, dim[i] > 0x7FFFFFFFL, can never be true on 32 bit platforms (because dimensions are signed), and is apparently wrong on 64 bit platforms. If I'm not totally blind, this test can just be deleted.

Steps to reproduce

On a 64 bit system:

open Bigarray;;

let g = Genarray.create char c_layout [| 0x80000000 |];;

val g :
(char, Bigarray.int8_unsigned_elt, Bigarray.c_layout) Bigarray.Genarray.t =

reshape_1 g 0x80000000;;

Exception: Invalid_argument "Bigarray.reshape: negative dimension".


This comment has been minimized.

Copy link

commented Feb 20, 2012

Comment author: gerd

On the second glance, I think the test "size > 0x7FFFFFFFL" (for 32 bit only) belongs into caml_ba_alloc, after multiplying the dimensions. The real problem is that "size" (and num_elts) is unsigned, but the dimensions are signed. So, when reshaping, it may happen that a positive size is interpreted as a negative dimension (e.g. when a 1000 * 3_000_000 char array is reshaped into a one-dimension 3_000_000_000 array). IMHO, this should better be caught in caml_ba_alloc, by just disallowing that large arrays.

I'm not absolutely sure whether such a case actually exists, because one has to pass the new dimensions in to reshape, and is limited by max_int anyway.


This comment has been minimized.

Copy link

commented Feb 21, 2012

Comment author: @xavierleroy

The intent of this test was indeed to guarantee that bigarray dimensions always fit in 32-bit signed, for compatibility between 32- and 64-bit platforms. As discussed in the related PR, this guarantee is useless anyway, and the test in Bigarray.reshape should have been removed. Now it is (SVN commit 12180).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.