PKITS 4.10.7 and 4.10.8 hang on Windows in OpenSSL 3.0

[brucestephens]

Presuming TA.pem is the trust anchor from PKITS, the following works fine on GNU/Linux but hangs on Windows:

% openssl cms -verify -CAfile TA.pem -policy anyPolicy -in SignedInvalidMappingFromanyPolicyTest7.eml
CMS Verification failure
0046881801000000:error:17000064:CMS routines:cms_signerinfo_verify_cert:certificate verify error:../crypto/cms/cms_smime.c:289:Verify error: invalid or inconsistent certificate policy extension

Similarly for the next test.

git bisect suggests the problem is 9aa4be6 so I presume the difference is something to do with the way locking works on Windows compared to pthreads.

[mattcaswell]

The issue is that there is an attempt to obtain a lock which is already held. The function in question is ossl_policy_cache_set_mapping which is only ever called by policy_cache_new which is only ever called by ossl_policy_cache_set.

ossl_policy_cache_set obtains the X509 lock before policy_cache_new is called and releases it afterwards. ossl_policy_cache_set_mapping then additionally tries to get the lock again.

Using pthreads the second attempt to obtain the lock simply fails and we skip the line that sets the EXFLAG_INVALID_POLICY flag. But that doesn't matter because policy_cache_new sets it anyway when ossl_policy_cache_set_mapping returns -1.

Under Windows it seems that attempting to obtain the lock a second time hangs.

This is all just wrong and the change from 9aa4be6 is just wrong. In fact the whole setting of the EXFLAG_INVALID_POLICY flag can be removed from that function completely because policy_cache_new will do it anyway.

The fix itself is easy - but setting "help wanted" because a test should be constructed for this.

[kroeckx]

It seems that in the case of USE_RWLOCK, we don't set PTHREAD_MUTEX_NORMAL or PTHREAD_MUTEX_ERRORCHECK. We probably should.

[t8m]

It seems that in the case of USE_RWLOCK, we don't set PTHREAD_MUTEX_NORMAL or PTHREAD_MUTEX_ERRORCHECK. We probably should.

Hmm, perhaps only in debug builds?

[kroeckx]

In case of !USE_RWLOCK, debug builds get ERRORCHECK, other builds NORMAL. In case of USE_RWLOCK, it seems we get DEFAULT because we don't set it.

[kroeckx]

So properly reading the manpages, only for mutexs can you set the things like recursive or not. For an rwlock it's just undefined, and seems to depend on the architecture. So maybe it can be useful for a debug build to not uwe rwlocks, but instead use the mutex in error checking mode.

[bernd-edlinger]

Are you saying we lock the write-lock while already holding the read-lock ?

[paulidale]

It's a wrtie lock while holding a write lock.

[anton-kazlouski]

Is there any ETA for releasing this fix?