Support RawPublicKey (non-X509) certificates, e.g. for P2P

[infinity0]

I am looking into using rustls for a p2p network, which authenticates peers (specifically their public keys) via custom means outside of the public X509 root certs.

Currently in rustls it is possible for a client to omit server certificate verification via ClientConfig.dangerous(), however the server via ServerConfig.cert_resolver still has to provide a CertifiedKey which hard-codes a X509 certificate.

TLS 1.3 mandates support for different types of certificates, specifically including RawPublicKey. This makes p2p usage more convenient as we don't have to faff around with creating a self-signed X509 certificate. Please consider supporting this in rustls. Of course the default behaviour for a client would be to fail to verify these "certificates".

(TLS 1.2 has optional support for the same functionality, but implemented differently as an extension, RFC 7250. The use-case I mention here is only about new protocols, so I don't consider this a necessary part of this issue.)

[burdges]

A priori, one wants back certs however, which using a certificates inside the handshake provides.

Story 1. We expand rustls by support for sr25519 and ecdsa secp256k1 (already support ed25519). We then ask that controller keys certify nodes TLS keys, which nodes supply upon opening connections thereby proving their intent to use that certificate.

Story 2. We expand rustls by RawPublicKey. We then ask that controller keys certify nodes TLS keys on-chain like all other session keys now, although afaik our long-term transport key still never get handled this way even now.

Story 3. We start like story 2 but go further by asking that nodes TLS keys certify their controller key on-chain ala paritytech/substrate#7398

Is there any difference? In story 2, Alice could register Bob's session key under her own controller key, perhaps creating strange effects, like Alice avoiding slashing while offline or whatever. Afaik, there are only negligible differences between stories 1 and 3 so long as the chain acts as a source of truth, but these are transport keys so they get used before the chain gets established, so maybe some mild shenanigans exist.

At first blush, 1 sounds easier and slightly safer, although any safety might be outweighed by sending the certificate with every connections, plus the confusion of another session key that requires certification by a controller. I'm not open minded about RawPublicKey but it makes the system more fragile.

[infinity0]

I'm not sure how far sr25519 is to making it into any official RFC - that's the main benefit of RawPublicKey, it's already in TLS 1.3

Besides, all these other alternative mechanisms can be implemented outside of TLS - as long as you know the public key of the peer, that suffices to enable you to implement whatever you like beyond that, and you can avoid X509 totally.

[burdges]

Authentication mistakes remain a concern, which TLS addresses somewhat. We're probably slightly better off specifying an authentication that works more inside TLS, so not RawPublicKey plus later messages.

There is nothing wrong with custom X509 extensions. We never accept regular root certificates, so who cares?

We could replace X509 with some scheme that better fits our trust root, like providing a Merkle proof from the chain's state root along with a controller signature, but then supply the certificate inside the channel exactly where an X509 certificate goes, so again not RawPublicKey plus later messages.

We cannot avoid some chicken vs egg here since we do not know the chain's state roots when say syncing, but doing authentication greedily makes authentication errors more informative.

[burdges]

Apologies for the Polkadot specific derail, but it's broadly relevant that peer-to-peer projects want to reinvent the wheel, and they do pay costs for doing so.

[infinity0]

This issue is about getting something simple that is already part of the TLS 1.3 specs, into rustls for general use, outside of the context of any specific p2p protocol.

[..] then supply the certificate inside the channel exactly where an X509 certificate goes [..]

This can be done as a TLS extension, as described by the same section of the RFC I quoted in the OP that mentions RawPublicKey. It would be more complex for rustls to implement and provide an API for this than RawPublicKey. For type-safety, the type of the extended certificate will have to be a parameter to the ServerConfig.

edit: To be clear it would be great if rustls did this too, and I think "Authentication mistakes remain a concern" is legitimate, although I'd personally think it's relatively minor and would be satisfied with just RawPublicKey.

---

There is nothing wrong with custom X509 extensions.

Unrelated, but a common motivation for avoiding X509 is its over-complexity for what its common purpose is. When designing a cross language protocol, avoiding X509 in favour of something simpler, makes it easier to implement the protocol from scratch and to understand it fully. I don't know if it's theoretically possible to send a valid 1GB certificate during the handshake phase, and I don't want to have to know this, I'd rather just avoid X509. We don't need a self-driving car to crack a nut.

[est31]

@infinity0 you can use rcgen in the meantime. It makes generation of self signed certificates really easy. The simplest form just takes in a list of domain names, but you can customize it to your will: https://docs.rs/rcgen/0.8.5/rcgen/fn.generate_simple_self_signed.html

[jeffparsons]

Would someone with experience working on Rustls be able to comment on what they imagine support for this would look like? E.g. at least which bits of the code base would need to take into account the possibility of encountering a RawPublicKey?

EDIT: From skimming some of the relevant source, it's not super-obvious to me at what point the distinction would be made between actual certificates and "not-actually-a-certificates" like RawPublicKey. E.g. I can see an argument for not bothering typical consumers of the API with having to think about the existence of the latter.

[djc]

I think we might want to change Certificate to become an enum with X509 and RawPublicKey variants to more closely match the RFC. That would obviously be an API-breaking change so it might be a while before that gets released.

[burdges]

We could likely close this, but maybe certificates should be handled via some closure-like construction, but maybe ResolvesClientCert already suffices?

[djc]

It was brought to our attention today that the Solana community would like to see support for this. We discussed with the rustls maintainers privately to hash out how we might end up merging support for this; herewith a summary of that discussion.

• In principle, we're open to having some support for raw public keys in rustls. Based on our current thinking, we do think it might needed to be guarded by a danger() method call, like some other features that are easy to misuse.
• That's because it is quite different from RFC 5280 certificates in that there is no intrinsic binding between a certificate and a public key, and expiration or revocation have to be handled out of band.
• We assume support for clients will see them supplied with a limited set of server public keys, but servers will need to do some out of band verification of the client identity, and this might need support for asynchronous verification.

As such, if someone were to take this on we'd suggest that they work on a design for the feature first, resulting in an RFC. After that is reviewed by the maintainers the author could move forward to implementation.

(@ctz and @cpu feel free to add/correct me as necessary.)

[ctz]

I have been thinking a bit about how we could support this with the absolute minimum impact on other users. Here's some ideas, starting with three observations:

First observation: a client that has a set of raw public keys (RPK) for a server will ~never accept a X509 certificate chain instead (it has pretty specific a-priori information about the server and how it will be authenticated), and the converse is also true. So, as a client, we know 100% ahead of time if RPKs are required for a particular connection. The same argument is also true of servers requiring client auth with RPKs. At protocol level, this means the client_certificate_type/server_certificate_type extensions should only by offered with a single RawPublicKey item (never in combination with X509).

Second observation: RFC7520 (and its follow-on support in RFC8446) works by modalizing the representation of certificates for a whole connection, if negotiated by the client_certificate_type/server_certificate_type extensions. The wire format of "certificates" does not change. Things are a lot easier if we accept this leaking up into the API.

Third observation: RFC8446 incorporated RFC7520 unchanged, but I think that was a design error. The client_certificate_type should have been negotiated in the other direction, starting with the Server -> Client CertificateRequest message offering the certificate types the server is willing to accept, and concluding with the client confirming the its certificate type in the Certificate message. That underlines the first observation: a client needs to be willing commit to RPK before it has any information about what key types, signature algorithms, etc are supported by the server, or the server's identity, or indeed whether a server will even ask for client authentication!

That means -- at minimum -- we need:

1. a way for clients to opt-in to and require RPK support from the server, for the servers's identity.
2. a way for clients to opt-in to and require RPK support from the server, for the client's identity.
3. a way for servers to opt-in to and require RPK support from the client, for the client's identity.
4. a way for servers to opt-in to and require RPK support from the client, for the servers's identity.

I think (1) and (3) are best achieved by adding something like fn requires_raw_public_keys(&self) -> RawPublicKeySupport { RawPublicKeySupport::No } to the ServerCertVerifier/ClientCertVerifier trait. Implementing that function and returning Yes means:

• the client would offer server_certificate_type([RawPublicKey]) in its ClientHello, and require that the server agrees.
• the server would confirm client_certificate_type(RawPublicKey) in its ServerHello (TLS1.2) or EncryptedExtensions (TLS1.3), or otherwise reject the handshake with unsupported_certificate
• that the remainder of the trait implementation interprets "certificate" data as an SPKI, not an X509 certificate.
• after successful validation, existing public API functions like peer_certificates() also yield SPKIs

I think (2) and (4) similarly are best achieved by adding something like fn only_raw_public_keys(&self) -> RawPublicKeySupport { RawPublicKeySupport::No } to the ResolvesClientCert/ResolvesServerCert, which means:

• the client would offer client_certificate_type([RawPublicKey]) in its ClientHello, and require that the server agrees.
• the server would confirm server_certificate_type(RawPublicKey) in its ServerHello (TLS1.2) or EncryptedExtensions (TLS1.3), or otherwise reject the handshake with unsupported_certificate
• that the eventually-resolved CertifiedKey from these traits comes with a "certificate" in SPKI format.

We would also want to expose the two certificate type extensions in the ClientHello type, for use in the Acceptor API. This would allow a server to maintain both X509 and raw public key identities for itself.

I think that just about covers changes to the core of the library.

In the implementation work for this, I would also expect to see:

1. in the rustls-webpki crate, a new public API for allowing signature validation given an SPKI
2. exemplar ServerCertVerifier/ClientCertVerifier implementations, which opt into to RPK and check the peer's identity is on a pre-configured list of SPKIs. This would rely on (1).
3. exemplar ResolvesClientCert/ResolvesServerCert implementations supporting a single RPK (like AlwaysResolvesChain for example)
4. automated basic positive interop tests against OpenSSL 3.2
5. targetted negative tests for clients/servers rejecting the peer for missing RPK support (this should be able to use the existing transfer_altered mechanism)
6. support for SPKIs in rustls-pemfile
7. support for RPKs in example code (added to tlsclient-mio & tlsserver-mio, or separately). This would rely on (6).
8. update documentation to explain "RPK mode", and signpost this from affected API functions (like peer_certificates())

@djc & @cpu -- how do you feel about this, as a minimal option? And, have I missed anything?

[djc]

This sounds like a good path forward, thanks for writing it up!

(We can discuss the value of having an enum RawPublicKeySupport { Yes, No } but that is a minor detail.)

[ripatel-fd]

@ctz Thanks for sharing this.

a client that has a set of raw public keys (RPK) for a server will ~never accept a X509 certificate chain instead

This makes perfect sense to me, and seems indeed true in standard applications of RPK like DANE/TLSA or TOFU.

I think I have an exception to this assumption though: It is possible to implement a pseudo-RPK mechanism via pinning of the X.509 SubjectPublicKeyInfo. This is the workaround Solana and libp2p are currently doing (due to lack of support for RPK in various languages).

That X.509-based mechanism would obviously be obsoleted by RPK and these applications would undergo a transition period to RPK.
Clients might want to support X.509 as a fallback during the transition period because an upgraded client might encounter a server that hasn't upgraded yet.

But I understand the assumption "RPK xor X.509" might be important for sake of API cleanliness, and that rustls probably shouldn't encourage such hacks.

[lamafab]

@est31: @infinity0 you can use rcgen in the meantime. It makes generation of self signed certificates really easy. The simplest form just takes in a list of domain names, but you can customize it to your will: https://docs.rs/rcgen/0.8.5/rcgen/fn.generate_simple_self_signed.html

Asking in the context of quinn: how do you recommend doing authentication here when you only have the peer's public key? I assume you'd configure quinn to disable all client/server authentication and then move that part to the application level, respectively having the client sign his self-signed certificate with the long-term static key and sending the signature to the server who can verify it by calling quinn::Connection::peer_identity() and vice-versa, for example?

EDIT: I guess it's sufficient to just set the long-term static key in CertificateParams.key_pair which the other peer can verify by calling quinn::Connection::peer_identity() and check this against a white-list?

EDIT-2:

... which the other peer can verify by calling quinn::Connection::peer_identity() and check this against a white-list?

Or rather the rustls::RootCertStore rustls::client::ServerCertVerifier (for ClientConfig) and rustls::server::ClientCertVerifier (for ServerConfig), respectively.

[lamafab]

Alright, working with all those different crates and converting between types is a little verbose but I came up with a proof of concept that works well.

Implement Custom CertVerifier

We implement a custom CertVerifier that extracts the public key from the cert and checks it against a whitelist, for both client and server config. Logic and requirements are application specific.

Note that this does not validate timestamps or expiry dates.

Code

// NOTE: `AccountId` is application specific and simply wraps
// the DER encoded public key.

#[derive(Default)]
struct CertVerifier {
    white_list: RwLock<HashSet<AccountId>>,
}

impl CertVerifier {
    fn allow_peer(&self, id: AccountId) {
        let mut l = self.white_list.write().unwrap();
        l.insert(id);
    }
    fn verify_cert(&self, cert: &rustls::Certificate) -> Result<(), rustls::Error> {
        let Ok((_, peer_x509)) = x509_parser::parse_x509_certificate(&cert.0) else {
            return Err(rustls::Error::General(
                "failed to parse x509 certificate".to_string(),
            ));
        };

        // Retrieve the mandated subject alt name.
        // NOTE: We simply use the `DNSName` variant since that's what `rcgen`
        // uses by default when generating cert params.
        let Ok(Some(Some(x509_parser::extensions::GeneralName::DNSName(subject_alt)))) = peer_x509
            .subject_alternative_name()
            .map(|ext| ext.map(|alt| alt.value.general_names.first()))
        else {
            return Err(rustls::Error::General(
                "subject alternative name must be present".to_string(),
            ));
        };

        // Subject alt name must match our app name.
        if *subject_alt != SUBJECT_ALT_NAME {
            return Err(rustls::Error::General(
                "invalid subject alternative name".to_string(),
            ));
        }

        // Verify signature of the certificate.
        if peer_x509
            .verify_signature(Some(peer_x509.public_key()))
            .is_err()
        {
            return Err(rustls::Error::InvalidCertificate(
                rustls::CertificateError::BadSignature,
            ));
        }

        // Pubkey must be an elliptic curve point.
        let Ok(x509_parser::public_key::PublicKey::EC(ec_point)) = peer_x509.public_key().parsed()
        else {
            return Err(rustls::Error::General(
                "pubkey must be elliptic curve point".to_string(),
            ));
        };

        // Derive PKCS_ECDSA_P256_SHA256 pubkey from encoded point.
        let Ok(encoded_point) =
            p256::elliptic_curve::sec1::EncodedPoint::<p256::NistP256>::from_bytes(ec_point.data())
        else {
            return Err(rustls::Error::General(
                "pubkey must be PKCS_ECDSA_P256_SHA256".to_string(),
            ));
        };

        // `AccountId` is application specific.
        let pubkey = p256::ecdsa::VerifyingKey::from_encoded_point(&encoded_point).unwrap();
        let key_der = p256::pkcs8::EncodePublicKey::to_public_key_der(&pubkey)
            .unwrap()
            .to_vec();
        let account_id = AccountId::from_der(key_der).unwrap();

        // Check whether the peer is whitelisted.
        let l = self.white_list.read().unwrap();
        if !l.contains(&account_id) {
            return Err(rustls::Error::General("permission denied".to_string()));
        }

        Ok(())
    }
}

Implement Traits for CertVerifier

We simply wrap those mandated traits by quinn around the CertVerifier.

Code

impl rustls::server::ClientCertVerifier for CertVerifier {
    fn client_auth_root_subjects(&self) -> &[rustls::DistinguishedName] {
        &[]
    }
    fn verify_client_cert(
        &self,
        end_entity: &rustls::Certificate,
        _intermediates: &[rustls::Certificate],
        _now: std::time::SystemTime,
    ) -> Result<rustls::server::ClientCertVerified, rustls::Error> {
        self.verify_cert(&end_entity)?;
        Ok(rustls::server::ClientCertVerified::assertion())
    }
}

impl rustls::client::ServerCertVerifier for CertVerifier {
    fn verify_server_cert(
        &self,
        end_entity: &rustls::Certificate,
        _intermediates: &[rustls::Certificate],
        server_name: &rustls::ServerName,
        _scts: &mut dyn Iterator<Item = &[u8]>,
        _ocsp_response: &[u8],
        _now: std::time::SystemTime,
    ) -> Result<rustls::client::ServerCertVerified, rustls::Error> {
        debug_assert_eq!(
            server_name,
            &rustls::ServerName::try_from(SUBJECT_ALT_NAME)
                .expect("subject name must be a valid DNS name")
        );

        self.verify_cert(&end_entity)?;
        Ok(rustls::client::ServerCertVerified::assertion())
    }
}

Setup QUIC endpoints with custom CertVerifier

The CertVerifier is used for both the client and server config.

Code

fn setup_quic_endpoint(key_der: &[u8], bind: SocketAddr) -> (Endpoint, Arc<CertVerifier>) {
    let keypair =
        rcgen::KeyPair::from_der_and_sign_algo(key_der, &rcgen::PKCS_ECDSA_P256_SHA256).unwrap();

    // Generate self-signed certificate with the provided keypair.
    let subject_alts = vec![SUBJECT_ALT_NAME.to_string()];
    let mut params = rcgen::CertificateParams::new(subject_alts);
    params.key_pair = Some(keypair); // Set keypair
    let cert = rcgen::Certificate::from_params(params).unwrap();

    // Custom cert verifier for all incoming and outgoing connections.
    let cert_verifier = Arc::new(CertVerifier::default());

    // Prepare our certificate and private key.
    let cert_der = rustls::Certificate(cert.serialize_der().unwrap());
    let priv_key = rustls::PrivateKey(cert.serialize_private_key_der());

    // Setup the server config.
    let server_config = rustls::ServerConfig::builder()
        .with_safe_defaults()
        // How we verify incoming connections (as responder).
        .with_client_cert_verifier(cert_verifier.clone())
        // How we verify ourselves to others (as responder).
        .with_single_cert(vec![cert_der.clone()], priv_key.clone())
        .unwrap();

    let client_config = rustls::ClientConfig::builder()
        .with_safe_defaults()
        // How we verify incoming connections (as initiator).
        .with_custom_certificate_verifier(cert_verifier.clone())
        // How we verify outselves to others (as initator).
        .with_client_auth_cert(vec![cert_der], priv_key)
        .unwrap();

    let server_config = Arc::new(server_config);
    let client_config = Arc::new(client_config);

    // Create the two-way endpoint with client and server configurations.
    let quic_server_config = quinn::ServerConfig::with_crypto(server_config);
    let mut endpoint = Endpoint::server(quic_server_config, bind).unwrap();
    endpoint.set_default_client_config(quinn::ClientConfig::new(client_config));

    (endpoint, cert_verifier)
}

Execute and Test Implementation

Unless the peer is explicitly whitelisted, there will be a an error during connection establishment.

Code

#[tokio::test]
async fn quic_endpoints_with_custom_verifier() {
    // Start endpoint for Alice.
    let alice_kp = p256::ecdsa::SigningKey::random(&mut rand::thread_rng());
    let alice_sec = p256::pkcs8::EncodePrivateKey::to_pkcs8_der(&alice_kp).unwrap();
    let alice_pub =
        p256::pkcs8::EncodePublicKey::to_public_key_der(alice_kp.verifying_key()).unwrap();

    let (alice, alice_verifier) =
        setup_quic_endpoint(alice_sec.as_bytes(), "127.0.0.1:8888".parse().unwrap());

    // Start endpoint for Bob.
    let bob_kp = p256::ecdsa::SigningKey::random(&mut rand::thread_rng());
    let bob_sec = p256::pkcs8::EncodePrivateKey::to_pkcs8_der(&bob_kp).unwrap();
    let bob_pub =
        p256::pkcs8::EncodePublicKey::to_public_key_der(bob_kp.verifying_key()).unwrap();

    let (bob, bob_verifier) =
        setup_quic_endpoint(bob_sec.as_bytes(), "127.0.0.1:9999".parse().unwrap());

    // Alice whitelists Bob
    let bob_id = AccountId::from_der(bob_pub.to_vec()).unwrap();
    alice_verifier.allow_peer(bob_id);

    // Alice starts connections.
    let _alice_conn = alice
        .connect("127.0.0.1:9999".parse().unwrap(), SUBJECT_ALT_NAME)
        .unwrap()
        .await
        .unwrap();

    // THIS FAILS: Bob did not whitelist Alice.
    assert!(bob.accept().await.unwrap().await.is_err());

    // Bob whitelists Alice.
    let alice_id = AccountId::from_der(alice_pub.to_vec()).unwrap();
    bob_verifier.allow_peer(alice_id);

    // Alice starts connections.
    let _alice_conn = alice
        .connect("127.0.0.1:9999".parse().unwrap(), SUBJECT_ALT_NAME)
        .unwrap()
        .await
        .unwrap();

    // Bob accepts connection from Alice.
    let _bob_conn = bob.accept().await.unwrap().await.unwrap();
}