CKR_DATA_LEN_RANGE when trying to sign with ECDSA

[chern042]

I had some code that would succesfully sign data from an HSM, but for some reason randomly started giving the "CKR_DATA_LEN_RANGE" error on sign.final() even when I switched to a previously working copy.

        if (!params.data) {
            params.data = '1234567a1234567a'; //I have tested with simple string data, hex, and more
        }
        var sign = session.createSign(alg,privObj);
        sign.update(Buffer.from(params.data,'hex'));
        var signature = sign.final();
        console.log(signature.toString('hex'));

This crashes when trying to complete sign.final(), giving the error:

Error CKR_DATA_LEN_RANGE:33
    at Error (native) crypto_final:637

The length function returns a length of 8 for this specific hex string when converted to a buffer.

[microshine]

@chern042
What HSM do you use?
What algorithm do you use?

https://github.com/PeculiarVentures/graphene/blob/master/src/crypto/sign.ts#L35-L37

[chern042]

I'm using the enum provided:

        let alg: graphene.MechanismType;
        alg = graphene.MechanismEnum.ECDSA;

I'm not exactly sure what you mean by what HSM I use, if it helps I know I am using a SafetNet HSM.

[microshine]

ECDSA mechanism is using for Hash result signing. So you must compute a digest first and sign it by using once method. If you want to sign/verify data via update method you must use mechanisms like ECDSA_SHA1, ECDSA_SHA256, etc.

Have you seen node-webcrypto-p11 module? It's based on graphene-pk11 and implements WebCrypto interface

[chern042]

ECDSA_SHA256 gives an invalid mechanism type error. It worked fine with the ECDSA mechanism in all use cases before it started throwing this error.

[microshine]

SoftHSM supports CKM_ECDSA only https://github.com/opendnssec/SoftHSMv2/blob/develop/src/lib/SoftHSM.cpp#L4166

try this code

const digest = crypto.session.createDigest("sha256");
digest.update(data);
const hash = digest.final();

// or (depends on message size)

const hash2 = crypto.session.createDigest("sha256").once(data);

const signature = session.createSign(alg,privObj).once(hash); // or hash2

[chern042]

I see. It's expecting a 32-byte hashed value. Just a few days ago it would work fine with any data inputted into the sign.update Is there a reason this randomly changed?

[microshine]

I've got CKR_OPERATION_NOT_INITIALIZED error on update. I'm using SoftHSM v.2.5.0

const signer = session.createSign("ECDSA", keys.items(0));
signer.update(Buffer.from("1234567890"));
const signature = signer.final();

But it works with the wrong data (length less than 20 bytes) if I'm using once

const signature = crypto.session.createSign("ECDSA", keys.items(0)).once("1234567890");

[chern042]

Weird, when i set it up like you have it:
var sign = session.createSign(alg,privObj).once(Buffer.from(params.data,'hex'));
It gives me the error unless params.data is the right length.