RSA-PSS setParameter: trailerField != 1 throws unchecked IllegalArgumentException and half-applies the rejected spec

[Arpan0995]

Summary

Signature.setParameter on the RSA-PSS services rejects a PSSParameterSpec whose trailerField is not 1, which is correct, but it does so with an unchecked IllegalArgumentException where the JCA contract is InvalidAlgorithmParameterException. The spec's fields are also stored before the trailer field is checked, so the rejected spec is partially applied. After initSign, the live signer keeps its previous parameters while getParameters() reports the rejected ones, so a verifier or AlgorithmIdentifier built from getParameters() no longer matches the signature. Before initSign, the next initialisation picks up the half-applied fields, so a caller that was told the call failed signs with parameters it believes were refused; on the RSASSA-PSS service that yields a signature with a SHA-1 content digest and a SHA-256 mask generation function, reported as SHA-256 throughout. The unchecked type is also reachable from data: BC's own AlgorithmParameters("PSS") decodes a certificate's RSASSA-PSS-params with trailerField 2 as-is, and X509Certificate.verify(pub, "BC") then throws the IllegalArgumentException. This is the same shape as #2396 and #2412, an unchecked exception out of setParameter, and here the engine state is left inconsistent as well. The JDK's own provider throws InvalidAlgorithmParameterException for the same spec and rejects the certificate encoding at decode.

Environment

• bcprov 1.86.0.20698 (current 1.86 beta), main at commit 402aed6
• JDK 27 (27-ea+32)

Steps to reproduce

Steps 1 to 4 use SHA256withRSAandMGF1. Step 5 builds a self-signed certificate whose signature AlgorithmIdentifier declares trailerField 2 (the signature itself is made with trailer 1, which is the only encoding PSS defines, so the certificate is otherwise valid). Step 6 uses the RSASSA-PSS service and the lightweight PSSSigner to identify which digests the resulting signature was actually made with.

import java.io.ByteArrayInputStream;
import java.security.*;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.security.interfaces.RSAPublicKey;
import java.security.spec.*;
import java.util.Date;
import org.bouncycastle.asn1.ASN1EncodableVector;
import org.bouncycastle.asn1.ASN1Integer;
import org.bouncycastle.asn1.DERBitString;
import org.bouncycastle.asn1.DERNull;
import org.bouncycastle.asn1.DERSequence;
import org.bouncycastle.asn1.nist.NISTObjectIdentifiers;
import org.bouncycastle.asn1.pkcs.PKCSObjectIdentifiers;
import org.bouncycastle.asn1.pkcs.RSASSAPSSparams;
import org.bouncycastle.asn1.x500.X500Name;
import org.bouncycastle.asn1.x509.AlgorithmIdentifier;
import org.bouncycastle.asn1.x509.SubjectPublicKeyInfo;
import org.bouncycastle.asn1.x509.TBSCertificate;
import org.bouncycastle.asn1.x509.Time;
import org.bouncycastle.asn1.x509.V3TBSCertificateGenerator;
import org.bouncycastle.crypto.Digest;
import org.bouncycastle.crypto.digests.SHA1Digest;
import org.bouncycastle.crypto.digests.SHA256Digest;
import org.bouncycastle.crypto.engines.RSAEngine;
import org.bouncycastle.crypto.params.RSAKeyParameters;
import org.bouncycastle.crypto.signers.PSSSigner;
import org.bouncycastle.jce.provider.BouncyCastleProvider;

public class PssTornStateRepro {
    static final byte[] MSG = "torn".getBytes();
    static KeyPair kp;

    public static void main(String[] args) throws Exception {
        Security.addProvider(new BouncyCastleProvider());
        KeyPairGenerator kpg = KeyPairGenerator.getInstance("RSA", "BC");
        kpg.initialize(2048);
        kp = kpg.generateKeyPair();
        // a PSSParameterSpec the JDK class accepts but BC rejects: trailerField 2
        PSSParameterSpec rejected = new PSSParameterSpec("SHA-256", "MGF1", MGF1ParameterSpec.SHA256, 20, 2);

        // 1. exception type: Signature.setParameter declares InvalidAlgorithmParameterException
        Signature s0 = Signature.getInstance("SHA256withRSAandMGF1", "BC");
        s0.initSign(kp.getPrivate());
        try { s0.setParameter(rejected); System.out.println("1. setParameter accepted (unexpected)"); }
        catch (Throwable t) { System.out.println("1. setParameter(trailerField=2) -> " + t.getClass().getName() + ": " + t.getMessage()); }

        // 2. torn state after init: the live signer still uses the default salt 32, but getParameters() now reports 20
        s0.update(MSG);
        byte[] sig = s0.sign();
        int reported = s0.getParameters().getParameterSpec(PSSParameterSpec.class).getSaltLength();
        System.out.println("2. after the rejection: getParameters() reports saltLength=" + reported
            + "; the signature verifies with salt 20: " + verify(sig, 20) + ", with salt 32: " + verify(sig, 32));

        // 3. rejected before init: the next initSign builds the signer from the half-applied fields
        Signature s1 = Signature.getInstance("SHA256withRSAandMGF1", "BC");
        try { s1.setParameter(rejected); } catch (Throwable t) { /* the same IllegalArgumentException */ }
        s1.initSign(kp.getPrivate());
        s1.update(MSG);
        byte[] sig2 = s1.sign();
        System.out.println("3. rejected before init, then initSign and sign: verifies with salt 20: " + verify(sig2, 20)
            + ", with salt 32 (the default): " + verify(sig2, 32));

        // 4. reference behaviour from the JDK provider
        try { Signature j = Signature.getInstance("RSASSA-PSS", "SunRsaSign"); j.setParameter(rejected); System.out.println("4. SunRsaSign accepted (unexpected)"); }
        catch (Throwable t) { System.out.println("4. SunRsaSign RSASSA-PSS setParameter(trailerField=2) -> " + t.getClass().getName() + ": " + t.getMessage()); }

        // 5. the unchecked type is reachable from data: a certificate whose RSASSA-PSS-params declare trailerField 2.
        //    BC's own AlgorithmParameters("PSS") decodes that value as-is and X509SignatureUtil passes it to setParameter.
        for (int trailer : new int[]{1, 2}) {
            X509Certificate c = (X509Certificate) CertificateFactory.getInstance("X.509", "BC").generateCertificate(new ByteArrayInputStream(certDeclaringTrailer(trailer)));
            try { c.verify(kp.getPublic(), "BC"); System.out.println("5. certificate declaring trailerField=" + trailer + ": X509Certificate.verify(pub, \"BC\") -> verified"); }
            catch (Throwable t) { System.out.println("5. certificate declaring trailerField=" + trailer + ": X509Certificate.verify(pub, \"BC\") -> " + t.getClass().getName() + ": " + t.getMessage()); }
        }

        // 6. on the RSASSA-PSS service (default SHA-1 / MGF1-SHA-1 / salt 20) a rejected SHA-256 spec before init leaves a mixed-digest signature
        Signature s2 = Signature.getInstance("RSASSA-PSS", "BC");
        try { s2.setParameter(new PSSParameterSpec("SHA-256", "MGF1", MGF1ParameterSpec.SHA256, 32, 2)); } catch (Throwable t) { /* rejected */ }
        s2.initSign(kp.getPrivate());
        s2.update(MSG);
        byte[] sig3 = s2.sign();
        PSSParameterSpec rep = s2.getParameters().getParameterSpec(PSSParameterSpec.class);
        System.out.println("6. RSASSA-PSS, rejected SHA-256 spec before init: getParameters() reports " + rep.getDigestAlgorithm() + "/"
            + ((MGF1ParameterSpec) rep.getMGFParameters()).getDigestAlgorithm() + "/salt " + rep.getSaltLength() + "/trailerField " + rep.getTrailerField()
            + "; signature verifies as SHA-1 content + SHA-256 MGF: " + lightweightVerify(sig3, new SHA1Digest(), new SHA256Digest(), 32)
            + ", as SHA-256 content + SHA-256 MGF: " + lightweightVerify(sig3, new SHA256Digest(), new SHA256Digest(), 32));
    }

    static boolean verify(byte[] sig, int salt) throws Exception {
        Signature v = Signature.getInstance("SHA256withRSAandMGF1", "BC");
        v.setParameter(new PSSParameterSpec("SHA-256", "MGF1", MGF1ParameterSpec.SHA256, salt, 1));
        v.initVerify(kp.getPublic());
        v.update(MSG);
        return v.verify(sig);
    }

    static boolean lightweightVerify(byte[] sig, Digest content, Digest mgf, int salt) {
        RSAPublicKey pub = (RSAPublicKey) kp.getPublic();
        PSSSigner v = new PSSSigner(new RSAEngine(), content, mgf, salt);
        v.init(false, new RSAKeyParameters(false, pub.getModulus(), pub.getPublicExponent()));
        v.update(MSG, 0, MSG.length);
        return v.verifySignature(sig);
    }

    // a self-signed certificate signed with SHA-256/MGF1-SHA-256/salt 20/trailer 1, whose signature AlgorithmIdentifier declares the given trailerField
    static byte[] certDeclaringTrailer(int trailer) throws Exception {
        AlgorithmIdentifier sha256 = new AlgorithmIdentifier(NISTObjectIdentifiers.id_sha256, DERNull.INSTANCE);
        AlgorithmIdentifier mgf1 = new AlgorithmIdentifier(PKCSObjectIdentifiers.id_mgf1, sha256);
        AlgorithmIdentifier sigAlg = new AlgorithmIdentifier(PKCSObjectIdentifiers.id_RSASSA_PSS, new RSASSAPSSparams(sha256, mgf1, new ASN1Integer(20), new ASN1Integer(trailer)));
        V3TBSCertificateGenerator g = new V3TBSCertificateGenerator();
        g.setSerialNumber(new ASN1Integer(1));
        g.setIssuer(new X500Name("CN=t"));
        g.setSubject(new X500Name("CN=t"));
        g.setStartDate(new Time(new Date(System.currentTimeMillis() - 1000)));
        g.setEndDate(new Time(new Date(System.currentTimeMillis() + 86400000L)));
        g.setSignature(sigAlg);
        g.setSubjectPublicKeyInfo(SubjectPublicKeyInfo.getInstance(kp.getPublic().getEncoded()));
        TBSCertificate tbs = g.generateTBSCertificate();
        Signature s = Signature.getInstance("SHA256withRSAandMGF1", "BC");
        s.setParameter(new PSSParameterSpec("SHA-256", "MGF1", MGF1ParameterSpec.SHA256, 20, 1));
        s.initSign(kp.getPrivate());
        s.update(tbs.getEncoded("DER"));
        ASN1EncodableVector v = new ASN1EncodableVector();
        v.add(tbs);
        v.add(sigAlg);
        v.add(new DERBitString(s.sign()));
        return new DERSequence(v).getEncoded("DER");
    }
}

Actual behaviour

1. setParameter(trailerField=2) -> java.lang.IllegalArgumentException: unknown trailer field
2. after the rejection: getParameters() reports saltLength=20; the signature verifies with salt 20: false, with salt 32: true
3. rejected before init, then initSign and sign: verifies with salt 20: true, with salt 32 (the default): false
4. SunRsaSign RSASSA-PSS setParameter(trailerField=2) -> java.security.InvalidAlgorithmParameterException: Only supports TrailerFieldBC(1)
5. certificate declaring trailerField=1: X509Certificate.verify(pub, "BC") -> verified
5. certificate declaring trailerField=2: X509Certificate.verify(pub, "BC") -> java.lang.IllegalArgumentException: unknown trailer field
6. RSASSA-PSS, rejected SHA-256 spec before init: getParameters() reports SHA-256/SHA-256/salt 32/trailerField 2; signature verifies as SHA-1 content + SHA-256 MGF: true, as SHA-256 content + SHA-256 MGF: false

Line 2 shows the signer and getParameters() disagreeing about the salt that was used. Line 3 shows the rejected parameters being applied after the caller was told the call failed. Line 5 shows the unchecked exception escaping X509Certificate.verify, whose contract is CertificateException and SignatureException, for a certificate that BC's own CertificateFactory parsed without complaint (the SUN CertificateFactory rejects the same encoding at decode with IOException: Unsupported trailerField value 2). Line 6 shows a signature whose content digest is SHA-1 while everything the caller can inspect says SHA-256, and whose reported trailerField of 2 describes no signature at all.

Expected behaviour

setParameter should throw InvalidAlgorithmParameterException, as Signature.setParameter declares and as SunRsaSign does for the same spec (line 4), and a rejected spec should leave the engine's parameters unchanged. Through the certificate path the caller should see a checked exception, which is what an InvalidAlgorithmParameterException from setParameter already becomes there.

Root cause

PSSSignatureSpi.engineSetParameter (PSSSignatureSpi.java, line numbers as of 402aed6) checks the digest and the mask generation function first, throwing InvalidAlgorithmParameterException when they do not match (the last two of those checks at lines 258 and 263), but then assigns engineParams, paramSpec, mgfDigest and saltLength (lines 266-269) before calling getTrailer(paramSpec.getTrailerField()) at line 270. getTrailer (lines 50-59) accepts only 1 and otherwise throws IllegalArgumentException("unknown trailer field") at line 58. The throw leaves the four fields already updated, and the two steps that would have completed the change never run: setupContentDigest() at line 272, and the PSSSigner rebuild guarded by if (key != null) at lines 274-285. So after the exception getParameters(), which is rebuilt from paramSpec whenever engineParams is null, reports the rejected spec while an initialised signer keeps its old parameters (line 2); a later initSign builds a new signer from the half-applied saltLength and mgfDigest (line 3); and on a service whose default content digest differs from the rejected spec's, setupContentDigest() having been skipped leaves the old content digest paired with the new MGF digest (line 6). PSSParameterSpec itself permits any non-negative trailerField, so getTrailer is the only check on the setParameter path (the constructor makes the same call for the default spec). The method is shared by every service backed by PSSSignatureSpi; it is shown here with SHA256withRSAandMGF1 and RSASSA-PSS.

The certificate path reaches the same method. X509SignatureUtil.setSignatureParameters decodes the certificate's RSASSA-PSS-params through AlgorithmParameters("PSS"), which passes trailerField through unchanged, and hands the resulting spec to setParameter, catching only GeneralSecurityException, so the IllegalArgumentException propagates out of X509Certificate.verify (line 5).

Impact

Contract and correctness. The half-applied state needs the application's own setParameter call, but the unchecked exception type does not: a certificate whose signature parameters declare trailerField 2 reaches the same IllegalArgumentException through X509Certificate.verify (line 5), out of a method whose contract is CertificateException and SignatureException. No key material is exposed and every signature produced is a well-formed PSS signature under some parameter set, but that set is not the one the caller can see: after a rejected call getParameters() describes a signature that was not made (line 2), a rejected call before initialisation is silently applied (line 3), and on the RSASSA-PSS service, whose default digest is SHA-1, the result is a signature with a SHA-1 content digest and a SHA-256 mask generation function, reported as SHA-256 throughout and carrying a trailerField of 2 that no verifier, BC included, can act on from the reported parameters (line 6). A catch block written for the declared InvalidAlgorithmParameterException does not see the unchecked exception at all. #2396 and #2412 addressed this shape in other signature SPIs, each of which has its own engineSetParameter; PSSSignatureSpi is separate.

Suggested direction

Validate the trailer field before committing anything, and report it through the contract: check newParamSpec.getTrailerField() != 1 up front alongside the existing digest and MGF checks and throw InvalidAlgorithmParameterException from there; only then reset engineParams and assign paramSpec, mgfDigest, saltLength and trailer. Checking in engineSetParameter rather than changing getTrailer keeps the constructor, which shares getTrailer and cannot throw a checked exception, unchanged. With the check ahead of the assignments a rejected spec leaves the engine exactly as it was, and because InvalidAlgorithmParameterException is a GeneralSecurityException, the certificate path then reports it as SignatureException with no further change.

The program above is complete and self-contained; it needs only the Bouncy Castle provider jar (bcprov) on the classpath.