Fix -4 return code when expected BAD_FUNC_ARG(-173)

[kaleb-himes]

Description

A PR had introduced a specific error when building with configuration --enable-trackmemory --enable-smallstack CFLAGS="-DALT_ECC_SIZE" some code in ssl_certman.c was over-writing an expected error value -173 (BAD_FUNC_ARG) with the error code WOLFSSL_BAD_FILE (-4) causing api.c to fail an expected BAD_FUNC_ARG check.

Testing

Using the above configuration with commit: 3af87f6 that introduced the error and running ./tests/unit.test

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[SparkiDev]

Hang on. Why are we not enabling the check of ret above instead of doing the malloc when we are already failing.

[kaleb-himes]

We can go with that approach instead and do the initialization to satisfy the compiler:

diff --git a/src/ssl_certman.c b/src/ssl_certman.c
index 9f227be15..46011595f 100644
--- a/src/ssl_certman.c
+++ b/src/ssl_certman.c
@@ -782,7 +782,7 @@ int wolfSSL_CertManagerVerify(WOLFSSL_CERT_MANAGER* cm, const char* fname,
 #ifndef WOLFSSL_SMALL_STACK
     byte   staticBuffer[FILE_BUFFER_SIZE];
 #endif
-    byte*  buff;
+    byte*  buff = NULL;
     long   sz = 0;
     XFILE  file = XBADFILE;
 
@@ -814,16 +814,15 @@ int wolfSSL_CertManagerVerify(WOLFSSL_CERT_MANAGER* cm, const char* fname,
      * small. */
 #ifndef WOLFSSL_SMALL_STACK
     if ((ret == WOLFSSL_SUCCESS) && (sz > (long)sizeof(staticBuffer)))
+#else
+
+    if (ret == WOLFSSL_SUCCESS)
 #endif
     {
         WOLFSSL_MSG("Getting dynamic buffer");
         buff = (byte*)XMALLOC((size_t)sz, cm->heap, DYNAMIC_TYPE_FILE);
         if (buff == NULL) {
-            /* If some other error has already occurred preserve that code
-             * return WOLFSSL_BAD_FILE if that is the only error condition */
-            if (ret == WOLFSSL_SUCCESS) {
-                ret = WOLFSSL_BAD_FILE;
-            }
+            ret = WOLFSSL_BAD_FILE;
         }
     }
     /* Read all the file into buffer. */

[kaleb-himes]

Fixed.

[JacobBarthelmeh]

retest this please jenkins