Report cert verify failure with MD5

[embhorn]

Description

Added a verify-mode guard to the CTC_MD5wRSA case in HashForSignature(), mirroring the existing MD2 sign/verify precedent. MD5-signed certificates now return HASH_TYPE_E during chain verification.

Fixes zd21597

Testing

Added test_wolfSSL_CertManagerRejectMD5Cert

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR makes certificate chain verification reject MD5-signed certificate signatures by returning HASH_TYPE_E when HashForSignature() is invoked in verify mode for CTC_MD5wRSA, and adds a regression test to ensure this behavior.

Changes:

• Add a verify-mode guard for CTC_MD5wRSA in HashForSignature() (blocked unless WOLFSSL_ALLOW_MD5_CERT_SIGS is defined).
• Add a new CertManager API test that generates an MD5-signed leaf cert and asserts verification fails with HASH_TYPE_E.
• Register WOLFSSL_ALLOW_MD5_CERT_SIGS in the known macro extras list and wire the new test into the certman test group.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
wolfcrypt/src/asn.c	Adds verify-mode rejection path for MD5 certificate signatures (unless explicitly allowed).
tests/api/test_certman.h	Declares and registers the new MD5-rejection certman test.
tests/api/test_certman.c	Implements a regression test generating an MD5-signed leaf cert and verifying it is rejected.
.wolfssl_known_macro_extras	Adds WOLFSSL_ALLOW_MD5_CERT_SIGS to macro discovery list.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10222

Scan targets checked: wolfcrypt-api_misuse, wolfcrypt-bugs, wolfcrypt-compliance, wolfcrypt-concurrency, wolfcrypt-consttime, wolfcrypt-defaults, wolfcrypt-mutation, wolfcrypt-portability, wolfcrypt-proptest, wolfcrypt-src, wolfcrypt-zeroize

No new issues found in the changed files. ✅

[github-actions]

MemBrowse Memory Report

No memory changes detected for:

• gcc-arm-cortex-m4- gcc-arm-cortex-m4-baremetal- gcc-arm-cortex-m4-min-ecc- gcc-arm-cortex-m4-tls12