RSA signatures are sometimes shorter than expected

[iay]

I've noticed occasional problems in verifying eduGAIN aggregates which seem to be caused by RSA signature values which are shorter than expected. For a 2048-bit modulus, we'd always expect the resulting signature value to be 256 bytes, but sometimes it is only 255 bytes. This is rejected by (e.g.) code based on the Java Santuario library >= v1.5. I believe that rejection is per the specification, see below.

Here are two examples of the failing case (about 14MB each): bad1 bad2

Here is a document whose signature is valid: good

The certificate containing the relevant public key is here: cert

The XML DSIG specification requires the implementation of RSASSA-PKCS1-V1_5-SIGN per RFC 2437 section 8.1. In step 3, this applies the RSASP1 operation from section 5.2.1. This results in a large integer signature representative but the length of this representative in terms of octets will vary: in 1 in 256 signatures on average, it will be 1 octet shorter than the modulus if "leading 00s" are not taken into account, in 1 in 65536 signatures it will be 2 octets short, and so on.

Step 4 therefore applies the I2OSP primitive to generate a fixed-length signature which is always the same length as the modulus; the code does not appear to be performing this step.

The code also appears to omit the required check on signature length in the RSASSA-PKCS1-V1_5-VERIFY algorithm in section 8.1.2:

 1. If the length of the signature S is not k octets, output "invalid
   signature" and stop.

[leifj]

Confirmed. I believe I have tracked this sucker down but I am still not clear if the fix is to drop the native python RSA library or weather to fix it.

[leifj]

fixed some time ago but forgot to close