Integer overflow issue with large image (copy and FL_RGB_SCALING_NEAREST)

[fire-eggs]

I'm working with a large JPEG image [30208 x 30208 x 3]. I have built fltk 1.3.5 in 64-bit mode using Visual Studio 2019.

In FL_Image.cxx, `Fl_RGB_Image::copy', there is an integer overflow problem that occurs with said image, when using FL_RGB_SCALING_NEAREST.

Namely, in this line:

for (dx = W, xerr = W, old_ptr = array + sy * line_d; dx > 0; dx --) {

I used ulong to fix it, as follows:

for (dx = W, xerr = W, old_ptr = array + ((ulong)sy * line_d); dx > 0; dx --) {

I see the problem still exists in the 1.4 code.

[fire-eggs]

Further analysis suggests that (unsigned int) would work as well, at least for this image.

[Albrecht-S]

Thanks for the report and the patch, this appears to be a bug. However, the real solution might be more involved, at least in FLTK 1.4.

Note 1: Regarding FLTK 1.3 and 1.4 (patch given above): I'd suggest to keep the signedness (sy and line_d are both int's) and use (long) for the cast rather than (ulong).

Note 2: I'm afraid that there are similar overflows in bilinear scaling mode below (lines 506 etc. in current 1.4 (master) code):

  memcpy(left, array + lefty * line_d + leftx * d(), d());
  ...

If this is true then this should be fixed as well. It's just an observation, I don't have enough time right now to dig deeper.

[fire-eggs]

Thank you for the follow-up. When I had a chance to try bilinear scaling mode, I would report any bug there as well.