Hash_gen fails with DRBG_CONT_FAILURE

[vuvova]

We're seeing RAND_bytes() failing (returning 0) rather often. Adding printfs in various places we traced it down to this check in Hash_gen()

        if (ret == 0) {
            XMEMCPY(&checkBlock, digest, sizeof(word32));
            if (drbg->reseedCtr > 1 && checkBlock == drbg->lastBlock) {
                if (drbg->matchCount == 1) {
                    fprintf(stderr,"HERE: in Hash_gen: 4: drbg->matchCount is 1, drbg->reseedCtr is %i, drbg->lastBlock is %i, returning DRBG_CONT_FAILURE %i\n", drbg->reseedCtr, drbg->lastBlock, DRBG_CONT_FAILURE);
                    return DRBG_CONT_FAILURE;
                }

this was printing, for example,

HERE: in Hash_gen: 4: drbg->matchCount is 1, drbg->reseedCtr is 543, drbg->lastBlock is 2037908499, returning DRBG_CONT_FAILURE 3
HERE: in Hash_DRBG_Generate: Hash_gen didn't return SUCCESS: 3

What is the logic behind this check? I don't see it in the NIST DRBG specs (as from http://dx.doi.org/10.6028/NIST.SP.800-90Ar1). It seems to be saying that the first 32 bit of the SHA256 cannot have the same value three times in a row. But it's a RNG, it seems that this coincidence can happen sometimes, even if it's unlikely

[vuvova]

Ok, so it's a FIPS thing. But

• why would it work in non-FIPS mode?
• isn't 32 bit too little to compare two SHA256 values?

[vuvova]

more info, initGlobalRNG seems to be 1 in wolfSSL_RAND_bytes, so concurrent invocations use the same shared state.

[embhorn]

Hello @vuvova

Could you please try the wolfcrypt test test.c and send us the output? If that passes, we will need to dig in deeper in the FIPS code. Otherwise there could be an issue in with the RNG entropy.

What is the RNG (hardware based, other)? How is it being seeded?

[vuvova]

@embhorn see my last comment, the first two turned out to be misleading, sorry. It's not about entropy or FIPS check, it's about many concurrent threads using the same shared WC_RNG globalRNG state without any mutex protection.

We're calling RAND_bytes() from different threads concurrently and they all use globalRNG because initGlobalRNG == 1

[embhorn]

@vuvova Thanks for the clarification. I've worked up a fix that I'm still testing. Please give it a review if you have an opportunity.
#3669

[vuvova]

Yes, that looks correct. Thanks!