Incorrect integer comparisons and buffer overflows

[gvanas]

In some parts of the code, the comparison between a caller-provided size and a small value is incorrectly written. For instance, in SpongeAbsorb() (see KeccakSponge.inc), the comparison is written as partialBlock + instance->byteIOIndex > rateInBytes, where partialBlock depends on the caller-provided size, rateInBytes is a (small) constant and instance->byteIOIndex is never greater than rateInBytes. If partialBlock is close to the limit, the problem is that partialBlock+instance->byteIOIndex can overflow, yielding a number smaller than rateInBytes and the comparison is flawed. Instead, it should be written as partialBlock > rateInBytes - instance->byteIOIndex, where the right hand side never underflows since instance->byteIOIndex is never greater than rateInBytes.

Additionally, there are a couple of incorrect castings, like (unsigned int)(dataByteLen - i) that can cause an overflow when size_t is 64-bit.

Consequently, a buffer overflow can occur. For instance, in SpongeAbsorb(), the flawed comparison with a large value for partialBlock causes an overflow in the subsequent call to SnP_AddBytes.