Fix for CVE-2016-4332 (#3406)

This CVE issue was previously listed as fixed (via HDFFV-9950) back in
2016, but with no confirmation test. Now that test files exist for
the 2016 Talos CVE issues, we found that CVE-2016-4332 can raise an
assert in debug builds.

This fix replaces the assert with pointer checks that don't raise
errors or asserts. Since the function is in cleanup code, we do our
best to close and free things, even when presented with partially-
initialized structs.

Fixes CVE-2016-4332 and HDFFV-9950 (confirmed via the cve_hdf5 repo)

release_docs/RELEASE.txt

@@ -283,6 +283,18 @@ Bug Fixes since HDF5-1.14.0 release
 ===================================
     Library
     -------
+    - Fixed an assertion in a previous fix for CVE-2016-4332
+
+      An assert could fail when processing corrupt files that have invalid
+      shared message flags (as in CVE-2016-4332).
+
+      The assert statement in question has been replaced with pointer checks
+      that don't raise errors. Since the function is in cleanup code, we do
+      our best to close and free things, even when presented with partially
+      initialized structs.
+
+      Fixes CVE-2016-4332 and HDFFV-9950 (confirmed via the cve_hdf5 repo)
+
     - Fixed a file space allocation bug in the parallel library for chunked
       datasets
 

src/H5Omessage.c

@@ -619,13 +619,12 @@ H5O__msg_free_mesg(H5O_mesg_t *mesg)
 } /* end H5O__msg_free_mesg() */
 
 /*-------------------------------------------------------------------------
- * Function:	H5O_msg_free_real
+ * Function:    H5O_msg_free_real
  *
- * Purpose:	Similar to H5O_msg_reset() except it also frees the message
- *		pointer.
+ * Purpose:     Similar to H5O_msg_reset() except it also frees the message
+ *              pointer
  *
- * Return:	Success:	NULL
- *		Failure:	NULL
+ * Return:      NULL (always)
  *
  *-------------------------------------------------------------------------
  */
@@ -634,16 +633,15 @@ H5O_msg_free_real(const H5O_msg_class_t *type, void *msg_native)
 {
     FUNC_ENTER_NOAPI_NOINIT_NOERR
 
-    /* check args */
-    assert(type);
+    /* Don't assert on args since this could be called in cleanup code */
 
     if (msg_native) {
         H5O__msg_reset_real(type, msg_native);
-        if (NULL != (type->free))
+        if (type && type->free)
             (type->free)(msg_native);
         else
             H5MM_xfree(msg_native);
-    } /* end if */
+    }
 
     FUNC_LEAVE_NOAPI(NULL)
 } /* end H5O_msg_free_real() */