pkcs8: DER file decryption failure migrating from 0.7.6 to latest

[mkatychev]

Hello, after upgrading my pkcs8 version from 0.7.6 to 0.10.2, I get an error about DER metadata when
storing a pkcs5 wrapped pkcs8 document:

Error { kind: Noncanonical { tag: Tag(0xa1: CONTEXT-SPECIFIC [1] (constructed)) }, position: None }', 
examples/decrypt_key.rs:4:71

I have created a repo with the issue reproduced:
https://github.com/mkatychev/pkcs_8_breaking_example

cargo run --example can be used to regen the der file and reproduce successful decryption witk pkcs8 0.7 and the error that is shown with 0.10:

$ cargo run --example=new_key

$ cargo run -q --example=decrypt_key_07
Public Key: [e1, 82, 21, 31, 80, c0, fb, 3e, 3e, 8f, 60, 6a, dd, 63, 7a, 33, 84, 0, 6, 15, 7c, ec, ae, a6, da, cb, d2, 55, 50, 20, 3c, f2]


$ cargo run -q --example=decrypt_key
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: Noncanonical { tag: Tag(0xa1: CONTEXT-SPECIFIC [1] (constructed)) }, position: None }', examples/decrypt_key.rs:4:71
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Running openssl asn1parse produces an identical schema for both:

$ openssl asn1parse -inform der -in pk.der
    0:d=0  hl=3 l= 195 cons: SEQUENCE
    3:d=1  hl=2 l=  95 cons: SEQUENCE
    5:d=2  hl=2 l=   9 prim: OBJECT            :PBES2
   16:d=2  hl=2 l=  82 cons: SEQUENCE
   18:d=3  hl=2 l=  49 cons: SEQUENCE
   20:d=4  hl=2 l=   9 prim: OBJECT            :PBKDF2
   31:d=4  hl=2 l=  36 cons: SEQUENCE
   33:d=5  hl=2 l=  16 prim: OCTET STRING      [HEX DUMP]:F8AEA4A27DA067ECA79EDF81ECF461BA
   51:d=5  hl=2 l=   2 prim: INTEGER           :0800
   55:d=5  hl=2 l=  12 cons: SEQUENCE
   57:d=6  hl=2 l=   8 prim: OBJECT            :hmacWithSHA256
   67:d=6  hl=2 l=   0 prim: NULL
   69:d=3  hl=2 l=  29 cons: SEQUENCE
   71:d=4  hl=2 l=   9 prim: OBJECT            :aes-256-cbc
   82:d=4  hl=2 l=  16 prim: OCTET STRING      [HEX DUMP]:E469BE26EEFA0BCF9E91916BA48E104E
  100:d=1  hl=2 l=  96 prim: OCTET STRING      [HEX DUMP]:6457A2315D170708F6CA8C885BCBAEF6067B6A3204DC96986F941F5B6F1E3C108CB0E84BFA988310211E18C3E82E28C3E57193AEFF665C4BC7A2FA346DF2C3D3404A7346F0B35FC8E8FFD0B96F7701C59D04CEA6C4AF5045F7B4C70E93F2518B

This is the decryption pattern used in the old 0.7 without SecretDocument:

pub fn decrypt_07(
        encrypted: pkcs8_07::EncryptedPrivateKeyDocument,
        password: &[u8],
    ) -> Result<Self, Error> {
        let pk_doc = encrypted.decrypt(password).map_err(|e| e.to_string())?;
        let pk_info = pk_doc.private_key_info();
        let sk =
            ed25519_zebra::SigningKey::try_from(pk_info.private_key).map_err(|e| e.to_string())?;
        Ok(Self::new(sk))
    }

And this is the new one that produces the CONTEXT-SPECIFIC tag error:

pub fn decrypt(
        encrypted: pkcs8::EncryptedPrivateKeyInfo,
        password: &[u8],
    ) -> Result<Self, Error> {
        let secret = encrypted.decrypt(password)?;
        let pk_info: pkcs8::PrivateKeyInfo = secret.decode_msg()?;
        let sk =
            ed25519_zebra::SigningKey::try_from(pk_info.private_key).map_err(|e| e.to_string())?;
        Ok(Self::new(sk))
    }

---

Inlining the decrypt_key example produces this panic suggesting secret.decode_msg() is the source of the incompatibility:

$ cargo run -q --example=decrypt_key_inlined
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: Noncanonical { tag: Tag(0xa1: CONTEXT-SPECIFIC [1] (constructed)) }, position: None }', examples/decrypt_key_inlined.rs:11:62
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

fn main() {
    let path = "./pk.der";
    let password: &[u8] = b"password";
    let doc = std::fs::read(path).unwrap();
    let info = pkcs8::EncryptedPrivateKeyInfo::try_from(doc.as_ref()).unwrap();
    let encrypted = info;
    let password = password;
    let secret = encrypted.decrypt(password).unwrap();
    let pk_info: pkcs8::PrivateKeyInfo = secret.decode_msg().unwrap(); // panic is here
    let sk = ed25519_zebra::SigningKey::try_from(pk_info.private_key)
        .map_err(|e| e.to_string())
        .unwrap();
    let keypair = Keypair::new(sk);

    println!("Public Key: {:x?}", keypair.vk.as_ref());
}

Any advice on how to allow the new SecretDocument would be greatly appreciated

[tarcieri]

Where did you get the original key in pk.der?

I'm unable to decrypt it with OpenSSL, for example:

$ openssl version
OpenSSL 3.0.9 30 May 2023 (Library: OpenSSL 3.0.9 30 May 2023)
$ openssl pkcs8 -inform der -in pk.der -outform pem -out -
Enter Password: [password]
Error decrypting key
4067030240000000:error:068000A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1188:
4067030240000000:error:0688010A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:613:Field=attributes, Type=PKCS8_PRIV_KEY_INFO
4067030240000000:error:11800065:PKCS12 routines:PKCS12_item_decrypt_d2i_ex:decode error:../crypto/pkcs12/p12_decr.c:153:

[mkatychev]

The original pk.der can be created by running the cargo run --example=new_key example

[tarcieri]

I'm not entirely sure what's happening, but if you can't get OpenSSL to decrypt the key, it's in some way malformed

[mkatychev]

@tarcieri this is the code block responsible for encoding/encrypting for pk.der:
https://github.com/mkatychev/pkcs_8_breaking_example/blob/master/src/lib.rs#L41-L80
Is there an example/standard way in the formats repo that can produce something that openssl can read?

[tarcieri]

All of the test vectors in the pkcs8 crate were created using OpenSSL and the pkcs8 crate is capable of reproducing them byte-for-byte.

The ed25519 crate's pkcs8 feature implements and tests Ed25519 key serialization against OpenSSL-generated test vectors.

To really figure out what's happening I'd need to see the decryption of pk.der which shouldn't be too hard to produce. I would suspect it's something related to ed25519-zebra.

If you can create a minimal reproduction that uses the pkcs8 or ed25519 crates exclusively, I can also better advise.

[gshuflin]

All of the test vectors in the pkcs8 crate were created using OpenSSL and the pkcs8 crate is capable of reproducing them byte-for-byte.

The ed25519 crate's pkcs8 feature implements and tests Ed25519 key serialization against OpenSSL-generated test vectors.

To really figure out what's happening I'd need to see the decryption of pk.der which shouldn't be too hard to produce. I would suspect it's something related to ed25519-zebra.

If you can create a minimal reproduction that uses the pkcs8 or ed25519 crates exclusively, I can also better advise.

I'm a colleague of @mkatychev debugging this same issue. I have a repo here: https://github.com/knox-networks/pkcs8-error-example that reproduces the failure to decrypt an ed25519 secret key generated with openssl, without using ed25519-zebra.

The README has run instructions, and references the code in examples/decrypt_key_inlined.rs. When I run it locally I get the following output:

➤ just generate-and-decrypt-keys
just generate-keys
nix run nixpkgs#openssl -- genpkey -algorithm ed25519 -outform der -out private_key.der
nix run nixpkgs#openssl -- pkcs8 --inform der --in private_key.der -topk8 -outform der -out pkcs8-version.der
Enter Encryption Password:
Verifying - Enter Encryption Password:
rm private_key.der
cargo run --example=decrypt_key_inlined
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
     Running `target/debug/examples/decrypt_key_inlined`
Secret Key len: 34, key: [4, 32, 179, 92, 97, 16, 91, 229, 164, 28, 40, 10, 40, 134, 151, 185, 22, 102, 191, 74, 208, 23, 142, 112, 249, 175, 13, 175, 108, 163, 66, 172, 40, 68]

Note that the &[u8] of the secret key generated is of length 34, not 32, so no ed25519 implementation should be able to successfully parse it.

[tarcieri]

That's actually the expected PKCS#8 encoding for an Ed25519 key per RFC8410 Section 7

Here's a decoded version of this example from the RFC:

-----BEGIN PRIVATE KEY-----
MC4CAQAwBQYDK2VwBCIEINTuctv5E1hK1bbY8fdp+K06/nwoy/HU++CXqI9EdVhC
-----END PRIVATE KEY-----

https://lapo.it/asn1js/#MC4CAQAwBQYDK2VwBCIEINTuctv5E1hK1bbY8fdp-K06_nwoy_HU--CXqI9EdVhC

PrivateKeyInfo SEQUENCE (3 elem)
  version Version INTEGER 0
  privateKeyAlgorithm AlgorithmIdentifier SEQUENCE (1 elem)
    algorithm OBJECT IDENTIFIER 1.3.101.112 curveEd25519 (EdDSA 25519 signature algorithm)
  privateKey PrivateKey OCTET STRING (34 byte) 0420D4EE72DBF913584AD5B6D8F1F769F8AD3AFE7C28CBF1D4FBE097A88F44755842
    OCTET STRING (32 byte) D4EE72DBF913584AD5B6D8F1F769F8AD3AFE7C28CBF1D4FBE097A88F44755842

As you can see, the encoding is (perhaps somewhat bizarrely) a nested OCTET STRING, always 34-bytes long, and always containing the leading bytes 04 20.

The pkcs8 crate is agnostic to how the privateKey field is encoded and just exposes it as a &[u8], however it needs to be subsequently processed by an algorithm-specific decoder. What you're seeing is exactly what is in the key generated by OpenSSL, which conforms to the aforementioned RFC.

For that, you can use the PKCS#8 support in the ed25519 crate, rather than using the pkcs8 directly:

https://docs.rs/ed25519/latest/ed25519/pkcs8/struct.KeypairBytes.html

This handles decoding/encoding that nested OCTET STRING and giving you the [u8; 32] you're expecting.

Or failing that, just strip the [0x04, 0x20] prefix when decoding, and add it when encoding.

[gshuflin]

It looks like if I generate a secret key with the above-mentioned openssl genpkey ... and then openssl pkcs8 commands, I can read back the bytes of that key, strip the [0x04, 0x20] prefix, and successfully decode them as an ed25519 secret key. So that is good.

I'm still having trouble using openssl to generate and read in a valid public key. E.g. if I have a pkcs8 key.der secret key, I'd like to be able to use openssl tools to generate a representation of the bytes of the public key that I can treat as a valid ed25519 pubkey, with for instance ed25519_zebra::VerificationKeyBytes::try_from(bytes). I've tried the following without success:

nix run nixpkgs#openssl -- pkcs8 -in key.der --inform der --out temp.pem
nix run nixpkgs#openssl -- ec -in temp.pem -pubout

this yields, when the password is input, output like:

read EC key
writing EC key
-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEAt9M7XBE/Pdk1laYgyZbsrUy8hjGv2zHvNE8YMlJQs8o=
-----END PUBLIC KEY-----

which appears to be a base64-encoded public key. However when I try to decode the bytes of this key and use it with our ed25519 rust types, it fails. I wonder if I'm using the wrong set of openssl commands to generate this.

[tarcieri]

Take a look at the example here:

https://github.com/RustCrypto/signatures/blob/fb866b4/ed25519/tests/pkcs8.rs#L50-L60

You need to use this type:

https://docs.rs/ed25519/latest/ed25519/pkcs8/struct.PublicKeyBytes.html

use ed25519::pkcs8::{DecodePublicKey, PublicKeyBytes};

const PUBLIC_KEY_PEM: &str = "-----BEGIN PUBLIC KEY-----\n\
MCowBQYDK2VwAyEAt9M7XBE/Pdk1laYgyZbsrUy8hjGv2zHvNE8YMlJQs8o=\n\
-----END PUBLIC KEY-----";

fn main() {
    let pk = PublicKeyBytes::from_public_key_pem(PUBLIC_KEY_PEM).unwrap();
    dbg!(pk);
}

[src/main.rs:9] pk = PublicKeyBytes(B7D33B5C113F3DD93595A620C996ECAD4CBC8631AFDB31EF344F18325250B3CA)

[tarcieri]

Also all of this seems to be working as intended, so closing this issue