Bigarray reshape wrong for 64 bit systems #5511
Original bug ID: 5511
Just found these lines in bigarray_stubs.c, in caml_ba_reshape:
if (dim[i] < 0 || dim[i] > 0x7FFFFFFFL)
The second condition, dim[i] > 0x7FFFFFFFL, can never be true on 32 bit platforms (because dimensions are signed), and is apparently wrong on 64 bit platforms. If I'm not totally blind, this test can just be deleted.
Steps to reproduce
On a 64 bit system:
let g = Genarray.create char c_layout [| 0x80000000 |];;
val g :
reshape_1 g 0x80000000;;
Exception: Invalid_argument "Bigarray.reshape: negative dimension".
Comment author: gerd
On the second glance, I think the test "size > 0x7FFFFFFFL" (for 32 bit only) belongs into caml_ba_alloc, after multiplying the dimensions. The real problem is that "size" (and num_elts) is unsigned, but the dimensions are signed. So, when reshaping, it may happen that a positive size is interpreted as a negative dimension (e.g. when a 1000 * 3_000_000 char array is reshaped into a one-dimension 3_000_000_000 array). IMHO, this should better be caught in caml_ba_alloc, by just disallowing that large arrays.
I'm not absolutely sure whether such a case actually exists, because one has to pass the new dimensions in to reshape, and is limited by max_int anyway.
Comment author: @xavierleroy
The intent of this test was indeed to guarantee that bigarray dimensions always fit in 32-bit signed, for compatibility between 32- and 64-bit platforms. As discussed in the related PR, this guarantee is useless anyway, and the test in Bigarray.reshape should have been removed. Now it is (SVN commit 12180).