fix verifying wii dev signatures

[shuffle2]

allows verifying dev-signed discs and wads

also fixes a memleak because mbedtls_rsa_free wasn't called before.

[Shizmob]

Update on this: the non-verifying signatures I've encountered actually do have correct PKCS#1 v1.5 padding, but they're of block type 2 (public-key padding) instead of the usual and recommended 1 (private-key padding). Possibly because apparently the dev keys are supposed to be known to developers anyway?

Either way, I've implemented the type 2 padding checks in my own tooling and all the signatures seem to verify correctly.

[shuffle2]

Good point! I should have looked into it a bit more :) I've seen this elsewhere as well - dev sigs encrypt the hash using d=0x10001, e={big num}. i.e. it is technically encrypted and not signed.

afaict mbedtls makes this quite annoying since it requires "private" rsa operations to have both d and e set in the context. So just setting d=e then retrying with mbedtls_rsa_pkcs1_decrypt (which checks the "type 2" format) fails because the rsa context is missing some fields. It seems like using mbedtls to "properly" check the padding would require dolphin embedding the dev e (which people can use to sign things 🙂), otherwise dolphin could implement it's own check...but eh.

fwiw in python you should just be able to use this https://www.pycryptodome.org/src/cipher/pkcs1_v1_5

[Shizmob]

Yeah, it's a bit of a mess! I opted to not use the PyCrypto PKCS#1 primitives because they have (correct, mind you) opinions about when you should use RSA encrypt versus decrypt operations, and about unpadding stuff properly (both of which the Wii at various points of course, can violate :>).

It seems mbedtls sadly doesn't cater out-of-the-box to non-standard use-cases like this either nor is its padding API usable by itself, which is a bummer...