fixing GCM troubles at HP-UX/IA64

[karel-m]

I have experienced GCM failures on:

• HP-UX 11.31 / 64bit
• CPU: Itanium 2 / IA64
• Compiler: HP C/aC++ B3910B A.06.26 [Apr 12 2011]

It compiled fine but GCM gives wrong results. Interesting is that all other test vectors (for hashes, ciphers, macs, rsa, dsa, ecc ..) passed, just GCM was failing.

After some experimenting I have come to the enclosed patch. I admit that it seems a bit unbelievable that this might be the cause of the failure. But according my empirical observations this change fixed it.

[sjaeckel]

wouldn't it make sense to leave x and y unsigned and only make z an unsigned char?
I think in the direction of compilers always truncating the counter variables to 8bit where it wouldn't be required.
can you test that please?

[karel-m]

This:

   unsigned x, y;
   unsigned char z;

fails as well.

Unfortunately I do not have permanent access to a HP-UX/IA64 machine, so experimenting is a bit complicated.

[sjaeckel]

Alright, no worries, then we'll leave it like that