asn1: add ASN1_STRING_set() check result

[nv-dmd]

Function asn1_str2type() (crypto\asn1\asn1_gen.c) uses function ASN1_STRING_set() to set the ASN1 string (line 702), but the result of ASN1_STRING_set() execution is not checked, although it may fail.
In the same function asn1_str2type() on line 648 the result of ASN1_STRING_set() is checked.

Added ASN1_STRING_set() check result.

Found by Linux Verification Center (linuxtesting.org) with SVACE.

[openssl-machine]

This pull request is ready to merge

[bernd-edlinger]

other places do a ERR_raise(ERR_LIB_ASN1, ERR_R_MALLOC_FAILURE);
suggest to do the same here

[nv-dmd]

other places do a ERR_raise(ERR_LIB_ASN1, ERR_R_MALLOC_FAILURE); suggest to do the same here

In the master branch is always done a ERR_raise(..., ERR_R_ASN1_LIB), but in the branches 3.0 and 3.1 done a ERR_raise(..., ERR_R_MALLOC_FAILURE)

What is better to done in such a case?

[bernd-edlinger]

oh, thanks for pointing that out,
I've only looked at 3.0 3.1 and did not expect master to be different.
but in that case it would be better to have a different PR for 3.0/3.1
where ERR_R_MALLOC_FAILURE is used instead.

[tmshort]

I can remove the (openssl-3.0) and (openssl-3.1) tags, and merge this just to master, @bernd-edlinger?

[bernd-edlinger]

yes this is certainly okay for master

This will be merged to master only; @nv-dmd will make a new PR for 3.1/3.0

[tmshort]

Adding (tests: exempted) because the external-tests are failing for master.

[tmshort]

Merged to master. Thank you for your contribution!