[Bug]: Inconsistent Integer-to-Byte Encoding of idx_sig

[soloblueice]

Contact Details

solomon.liu@armchina.com

Version

wolfSSL Release 5.8.4 (Nov. 20, 2025)

Description

Spec Reference (Section 2.4 – Integer-to-Byte Conversion):

If x and y are non-negative integers, we define Z = toByte(x, y) to be the y-byte string containing the binary representation of x in big-endian byte order.

Observed Behavior in wc_xmss_sign / wc_xmssmt_sign (wc_xmss_impl.c +3412):

/* byte[n] r = PRF(SK_PRF, toByte(idx_sig, 32)); */
wc_idx_copy(sig, params->idx_len, state->buf, XMSS_PRF_M_LEN);
wc_xmss_prf(state, sk_prf, state->buf, sig_r);
Here, idx_sig is encoded only into the first params->idx_len bytes of the buffer.

Observed Behavior in wc_xmss_hash_message (wc_xmss_impl.c +1275):

XMEMSET(idx_sig, 0, params->n - idx_len);
XMEMCPY(idx_sig + params->n - idx_len, idx, idx_len);

In this case, the encoding follows the spec definition more closely: the index is right-aligned within an n-byte buffer, padded with leading zeros to ensure big-endian representation.

Reproduction steps

No response

Relevant log output

[kareem-wolfssl]

Hi @soloblueice,

Thanks for the report. My colleague @SparkiDev who did our XMSS implementation will review this.

[zulfff]

hey @kareem-wolfssl just wanted to chime in on this

looked at the code and @soloblueice is right - there's definitely an inconsistency here. checked wc_idx_copy and it's doing:

XMEMCPY(d, s, sl);
XMEMSET(d + sl, 0, dl - sl);

so it's sticking the index at the front of the buffer. but RFC 8391 says toByte(x, y) should be big-endian (right-aligned with zeros).

the weird thing is wc_xmss_hash_message already does this correctly at line 1275:

XMEMSET(idx_sig, 0, params->n - idx_len);
XMEMCPY(idx_sig + params->n - idx_len, idx, idx_len);

fix would be simple - just flip the order in wc_idx_copy:

static void wc_idx_copy(const unsigned char* s, word8 sl, unsigned char* d,
    word8 dl)
{
    XMEMSET(d, 0, dl - sl);
    XMEMCPY(d + dl - sl, s, sl);
}

that way both PRF(SK_PRF, toByte(idx_sig, 32)) and H_msg(...toByte(idx_sig, n)...) use the same encoding. otherwise the PRF is getting garbage input lol

lmk if you need a PR or anything, happy to help

[SparkiDev]

Hi @AryzXploit,

I've made a PR with the change required: #9978
Please take a look and let me know if you are happy with the change.

Thanks,
Sean

[zulfff]

Sup @SparkiDev,

Looks good to me. The change aligns the encoding with the RFC 8391
definition of toByte(x, y) where the integer is right-aligned in the
byte string.

Thanks for the quick fix!