SymCryptRsaPkcs1Sign rejects valid unhashed messages

[qmuntal]

I'm using SCOSSL to sign an unhashed message using a 2048 bits RSA key with PKCS1 padding. The message lengths is the maximum allowed for this key size and padding combination: 2048/8-11.

I would expect that message to be correctly signed, but SCOSSL returns the following error (which I've formatted a bit): SymCryptRsaPkcs1Sign failed - SYMCRYPT_INVALID_ARGUMENT (0x800e).

Debugging a bit I've found that SymCrypt chokes on this check:

SymCrypt/lib/rsa_padding.c

Lines 709 to 712 in 769e0a3

	if (cbEncoding > 0x80)
	{
	scError = SYMCRYPT_INVALID_ARGUMENT;
	goto cleanup;

AFAIU, cbEncoding should only be lower or equal than 0x80 when the RSA_PKCS1_NO_ASN1 flag is NOT set, else the encoding length is not added to the output, therefore it doesn't need to fit in a single byte.

Reproducer using the Microsoft Go toolchain with GOEXPERIMENT=systemcrypto on AZL3:

package main

import (
  "crypto/rand"
  "crypto/rsa"
)

func main() {
  priv, _ := rsa.GenerateKey(rand.Reader, 2048)
  msg := make([]byte, 2048/8 - 11)
  _, err := rsa.SignPKCS1v15(rand.Reader, priv, 0, msg)
  if err != nil { panic(err) }
}

[samuel-lee-msft]

Good catch - this is almost certainly not going to cause real world interoperability issues, but it is an easy thing to fix to make SymCrypt a little more permissive.

We have a fix for this internally.