DH keys were 16 bits longer

[eduardsui]

While implementing TLS using libtomcrypt I noticed that MS Schannel (msie, edge) doesn't work with DHE keys if the size isn't exactly 2048/4096 bits. "DH-2048" and "DH-4096" are 16 bits longer.

[sjaeckel]

Why was this closed?

[eduardsui]

I managed to make an workaround, and I’m not sure that is really a bug. However, the DH prime is 16-24 bits longer for all structures.

For example, P length for a 256 byte (2048) key is 259 bytes (2072 bits). However, p is correct, but when testing TLS with MS SChannel, if the key and P are not exactly 2048 bits, will cause a "generic error". With any other implementation it works (openssl, nss, etc).

I solved it by modifying dh_make_key, to accept custom prime and generator. See https://github.com/eduardsui/tlslayer/blob/master/tlslayer.c, private_tls_dh_make_key (line 2490).

I could push the new function into libtomcrypt, if it helps.

The bottom line is that my changes removed the extra bits from he beginning of P, but I'm not sure that after update P is prime anymore (seems to be, but I'm not sure).

[karel-m]

Ad

modifying dh_make_key, to accept custom prime and generator

it would be IMO nice enhancement to libtomcrypt. I did something similar for EC in my branch https://github.com/libtom/libtomcrypt/tree/miko-ecc-enhancements (support for custom curve parameters). I am only using more ECC than DSA so improving dh_make_key was not such a big priority for me.

Maybe worth creating a separate issue, at least for not losing the link to Eduard's private_tls_dh_make_key idea.