XMSS custom cbPrefix can trigger stack-buffer-overflow

[Wowblk]

Hi SymCrypt team,

I found a potential memory-corruption issue in the XMSS custom-parameter path. SymCryptXmssSetParams accepts a custom cbPrefix value larger than the internal 64-byte XMSS prefix buffer, and a later public API call can then write past that stack buffer.

Summary

When custom XMSS parameters are used, SymCryptXmssSetParams(..., cbPrefix=65) returns SYMCRYPT_NO_ERROR. A subsequent SymCryptXmsskeyGenerate() call reaches the XMSS PRF/public-key generation path and ASan reports a stack-buffer-overflow on the fixed-size prefix stack buffer.

This seems to be caused by missing upper-bound validation for cbPrefix: the API rejects cbPrefix == 0, but it does not reject values larger than the internal prefix buffer size. The value is later used by XMSS code that operates on a 64-byte stack buffer.

Minimal PoC

#include <stddef.h>
#include <stdio.h>
#include <symcrypt.h>

int main(void)
{
    SYMCRYPT_MODULE_INIT();

    SYMCRYPT_XMSS_PARAMS params;
    SYMCRYPT_ERROR err = SymCryptXmssSetParams(
        &params,
        0x78706f62,
        SymCryptSha256Algorithm,
        16,
        4,
        2,
        1,
        65);

    printf("SetParams=0x%08x\n", err);
    if (err != SYMCRYPT_NO_ERROR) return 1;

    PSYMCRYPT_XMSS_KEY key = SymCryptXmsskeyAllocate(&params, 0);
    if (key == NULL) return 1;

    puts("calling Generate; ASan should report stack-buffer-overflow");
    err = SymCryptXmsskeyGenerate(key, 0);
    printf("Generate=0x%08x\n", err);

    SymCryptXmsskeyFree(key);
    return 0;
}

Observed result

Built as a release ASan-instrumented SymCrypt module, the PoC prints:

SetParams=0x00000000
calling Generate; ASan should report stack-buffer-overflow

Then ASan reports:

ERROR: AddressSanitizer: stack-buffer-overflow
WRITE of size 1
...
#4 SymCryptXmsskeyGenerate
#5 main poc_dynamic_min.c:26

Address ... is located in stack of thread T0
  This frame has 3 object(s):
    [32, 96) 'prefix' <== Memory access at offset 96 overflows this variable

Expected result

SymCryptXmssSetParams should reject custom cbPrefix values larger than the maximum internal prefix buffer size, instead of accepting them and allowing later XMSS operations to corrupt stack memory.

A fix could add an explicit upper-bound check for cbPrefix, for example rejecting values greater than SYMCRYPT_XMSS_MAX_PREFIX_SIZE, and avoid silently storing externally supplied custom parameter values into smaller internal fields without range validation.

Thanks!