rand_bn_bits returns numbers with too many or too few bits

[friedrichsenm]

Prerequisites

• Checked the developer manual
• Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=repo%3Alibtom%2Flibtomcrypt
• Checked that your issue isn't related to TomsFastMath's limitation that PK operations can by default only be done with max. 2048bit keys

Description

For a bit length that is not a multiple of 8, rand_bn_bits can return numbers with either more or less bits that you expect. For example, if the number of bits specified is 1 mod 8, the return value can have up to 6 more bits than expected. If the number is 7 mod 8, the return value will have a minimum of 6 fewer bits than expected.

Steps to Reproduce

#include "tomcrypt.h"

int main(void)
{
  ltc_mp = ltm_desc;

  void *p;
  int prng_idx;
    
  mp_init(&p);

  register_prng(&sprng_desc);
  prng_idx = find_prng("sprng");

  rand_bn_bits(p, 9, NULL, prng_idx);
  printf("Number of bits when expecting around 9: %d\n", mp_count_bits(p));

  rand_bn_bits(p, 7, NULL, prng_idx);
  printf("Number of bits when expecting around 7: %d\n", mp_count_bits(p));

  mp_clear(p);

  return 0;
}

Version

• v1.18.2
• gcc
• LTM
• Ubuntu 18.04

Additional Information

You should be able to fix the issue by changing the following line

mask = 0xff << (8 - bits % 8);

to

mask = 0xff >> (bits % 8 == 0 ? 0 : 8 - bits % 8);

and

/* mask bits */
   buf[0] &= ~mask;

to

/* mask bits */
   buf[0] &= mask;