BUG: preventing crash due to overflow for large int vectors by circumventing VNL dot_product

[romangrothausmann]

ITK-Snap crashed when opening large datasets. It seems this was caused by an integer overflow when calculating a pointer index. The function used dot_product from VNL, which does not check for overflows (http://itk.org/gitweb?p=ITK.git;a=blob;f=Modules/ThirdParty/VNL/src/vxl/core/vnl/vnl_sse.h;h=04f930ef05bb49036e1fa69d6fe9ae9a697e2d45;hb=HEAD#l189). Doing the calculation avoiding VNL allows to use a simple cast to size_t before multiplication.
This solved the issue for me. I can now load a grey image of about 10GB plus a 40GB segmentation into ITK-Snap, modify it (even in gWS mode) and then render the segmentation without any problems.
Perhaps using size_t for the vectors used for index calculation (instead of Vector3i) might be a cleaner approach. I'm not sure, but the casting solution might slow down ITK-Snap and is only needed for big datasets. Perhaps a pre-check for overflow could be introduced and dot_product used as long as possible.
I'm not sure if it is a good idea to encourage users to use ITK-Snap for such big datasets. If they shouldn't, ITK-Snap should at least through an exception that the dataset to be loaded is too big instead of crash.

[romangrothausmann]

Closing PR as the changes were merged at SF (http://sourceforge.net/p/itk-snap/src/ci/20152435b5be3371fcb71e9d6d8b37d17cafc91b/)