check the return value of EVP_MD_CTX_create(EVP_MD_CTX_new)

[x2018]

check the return value of EVP_MD_CTX_create(i.e. EVP_MD_CTX_new) in lib/sha256.c:87.
If some memory allocation errors occur, EVP_ MD_ CTX_ New() will return null, then EVP_DigestInit_ex(), EVP_DigestUpdate() and EVP_DigestFinal_ex() will make wrong memory access. So it is better to check the return value of it.

[bagder]

Shouldn't the init function rather return an error to then have Curl_sha256it() also return an error for this case so that curl can bail out all the way up?

[x2018]

Yes, it is best for this case. However, I hesitated to do that yesterday as there are other implementations under different macro conditions and I am not familiar with them. So if we suppose to do that, some definitions and statements need to be also updated.

[x2018]

Sorry. I made mistakes that commit to this branch. later I will rollback it.

[bagder]

please make Curl_sha256it return error/success instead and check that instead of relying on the output data. Yes, that's a bigger change but is much cleaner and nicer code.

[MarcelRaad]

Thanks! That was an oversight in 94696e1.

I agree with @bagder , the error should be handled by the caller. That would mean changing the signature of HMAC_hinit_func. Which would be good anyway as other implementations and the corresponding MD5 functions can also fail.

However, I hesitated to do that yesterday as there are other implementations under different macro conditions and I am not familiar with them. So if we suppose to do that, some definitions and statements need to be also updated.

The other implementations could just always return CURLE_OK for now. The CI system should tell us if anything was missed.

[x2018]

I changed the return type of Curl_sha256it() and Curl_md5it(). To be cautious, I still just add a check for the return value of EVP_MD_CTX_create(), but the other functions in *_init, *_update and *_final may also fail in some way. As I do not familiar with them and I'm not sure what status they should return when they have an error, so I just retain the original shape(or directly return CURLE_OK).

[MarcelRaad]

I can't comment on line 527, but the HMAC_hinit_func signature (and the MD5 init function) would have to be updated too to return CURLcode. Unfortunately the CURLX_FUNCTION_CAST meant for the parameter hides that. Looks good to me otherwise!

[x2018]

Sorry, I forgot to update it though you reminded me. I will do it after work...

[MarcelRaad]

Thanks, looks good to me now! Just waiting if @bagder wants to have another look too.

[bagder]

Thanks!