schannel: check schannel_sha256sum() success, and more

[vszakats]

Also:

• support 4GiB+ SHA-256 digest inputs.
• check CryptGetHashParam() output size.
• avoid overwriting existing digest when new digest calculation fails.
• avoid adding digest hash element on failure.

---

https://github.com/curl/curl/pull/21739/files?w=1

[Copilot]

Pull request overview

This PR updates the Schannel SHA-256 helper path to propagate failures from the underlying WinCrypt hashing routines, and uses that status when computing digests for CA cache validation.

Changes:

• Change schannel_checksum() to return a CURLcode and report success/failure.
• Make schannel_sha256sum() return the checksum result instead of always CURLE_OK.
• Check schannel_sha256sum() return value when comparing/storing CAinfo blob digests for the Schannel cert-store cache.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[vszakats]

augment review
@aisle-analyzer

[aisle-research-bot]

🔒 Aisle Security Analysis

We found 1 low security issue(s) in this PR:

See details in the comment below.

Analyzed PR: #21739 at commit 1cbdb37

_{Last updated on: 2026-05-26T23:13:50Z}

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

[augmentcode]

🤖 Augment PR Summary

Summary: This PR tightens SChannel SHA-256 digest handling so failures are detected and don’t silently produce/propagate incorrect hashes.

Changes:

• Make schannel_checksum() return a CURLcode and propagate success/failure to callers
• Validate the returned CryptGetHashParam(HP_HASHVAL) output length matches the algorithm hash size
• Update schannel_sha256sum() to return the actual status instead of always CURLE_OK
• In cert-store caching, compute the CA-blob digest into a temporary buffer so an existing cached digest isn’t overwritten on failure
• Avoid adding/updating cached state (or leaking ownership assumptions) when digest calculation fails

Technical Notes: The cache validation path now fails closed if hashing fails, preventing reuse of a store keyed by an invalid/zeroed digest.

_{🤖 Was this summary useful? React with 👍 or 👎}

[augmentcode]

Review completed. No suggestions at this time.

Comment augment review to trigger a new review at any time.

[vszakats]

@aisle-analyzer

[aisle-research-bot]

🔒 Aisle Security Analysis

✅ Aisle scan complete — no security issues detected.

Previously reported findings

🔵 Size truncation in SChannel CAinfo blob hashing can cause incorrect certificate-store cache hits

Low severity CWE-197 at lib/vtls/schannel.c:2668-2672

Description

In schannel_checksum(), size_t inputlen is cast to a 32-bit DWORD when passed to CryptHashData(). On 64-bit builds, if inputlen > 0xFFFFFFFF, the cast truncates the length and the hash is computed over only the low 32 bits (or the call fails), so the resulting digest does not uniquely represent the full input.

This digest is used to key the cached certificate store for CURLOPT_CAINFO_BLOB (share->CAinfo_blob_digest). If an application supplies two different CA blobs with the same >4GiB length and the same first (DWORD) bytes, the cache comparison can incorrectly treat them as identical and reuse a certificate store built from different trust material.

Vulnerable code:

if(!CryptHashData(hHash, input, (DWORD)inputlen, 0))
  goto out;

Notes:

• The blob is application-provided, but this is still a correctness/security footgun in core TLS code; other backends explicitly guard against oversized blobs.

Recommendation

Reject or safely handle inputs larger than DWORD_MAX.

Option A (simplest/safest): fail if inputlen > DWORD_MAX.

if(inputlen > (size_t)DWORD_MAX)
  return CURLE_BAD_FUNCTION_ARGUMENT;
if(!CryptHashData(hHash, input, (DWORD)inputlen, 0))
  goto out;

Option B: hash in chunks to support arbitrary sizes.

size_t off = 0;
while(off < inputlen) {
  DWORD chunk = (DWORD)CURLMIN(inputlen - off, (size_t)DWORD_MAX);
  if(!CryptHashData(hHash, input + off, chunk, 0))
    goto out;
  off += chunk;
}

Also consider adding a similar size guard at the call site using ca_info_blob->len (like other TLS backends do).

_{Aisle supplements but does not replace security review.}

_{Last updated on: 2026-05-26T23:52:28Z}

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.