Vault/OpenBao mTLS client identity fails to parse in v0.6.1 despite valid PEM cert/key accepted by reqwest::Identity::from_pem

[PureOxygen-Dev]

Summary
Using Rustguac v0.6.1 with the new [vault] mTLS settings:

[vault]
addr = "https://:8200"
role_id = ""
secret_id = ""
ca_cert = "/opt/rustguac/certs/openbao/ca.pem"
client_cert = "/opt/rustguac/certs/openbao/client.pem"
client_key = "/opt/rustguac/certs/openbao/client-key.pem"
Rustguac fails at startup with:

vault auth error: failed to parse client identity: builder error
We validated that the cert/key material is structurally correct, matches, is injected into the container unchanged, and is accepted by a standalone reproducer calling the same reqwest::Identity::from_pem(...) API.

Environment
Rustguac version claimed by source/docs: 0.6.1
Deployed image: sol1/rustguac:latest
Runtime: Docker Swarm
Vault backend: OpenBao
Auth method: AppRole
mTLS certs delivered to container as Swarm secrets via a loader script
OIDC works normally
Vault/OpenBao address book init fails only when building the mTLS client identity
Error
Rustguac logs:

ERROR rustguac: VAULT CONNECTION FAILED
ERROR rustguac: Address: https://:8200
ERROR rustguac: Error: vault auth error: failed to parse client identity: builder error
ERROR rustguac: Address book and drive features are UNAVAILABLE
ERROR rustguac: Sessions (SSH/RDP/VNC) will still work normally
What we validated

1. Generated cert/key material is valid
   The client artifacts generated from OpenBao are:

client certificate PEM
issuing CA PEM
unencrypted PKCS#8 private key PEM
The key header is:

-----BEGIN PRIVATE KEY-----
The full-chain cert file contains two certs:

leaf cert
issuing CA cert
2. OpenSSL accepts the material
We verified:

cert parses
key parses
cert/key public key digests match
3. reqwest::Identity::from_pem(...) accepts the same files
We built a standalone reproducer that does exactly:

let data = fs::read(&path)?;
reqwest::Identity::from_pem(&data)?;
Both of these combined PEM layouts succeeded:

cert chain + private key
private key + cert chain
4. Files inside the container are byte-for-byte identical to local files
We logged SHA256 of the staged files inside the container and compared them with the local generated files.

Container-side hashes matched local hashes exactly for:

CA cert
client fullchain cert
client key
So:

OpenBao KV content is correct
Swarm secret injection is correct
loader staging is correct
file corruption in transit is not the issue
Relevant source observation
From src/vault.rs in v0.6.1, Rustguac appears to do:

let cert_pem = std::fs::read(cert_path)?;
let key_pem = std::fs::read(key_path)?;
let mut identity_pem = cert_pem;
identity_pem.extend_from_slice(&key_pem);
let identity = reqwest::Identity::from_pem(&identity_pem)?;
That is consistent with how we built the reproducer. The reproducer succeeds; Rustguac still fails with builder error.

Expected behavior
Rustguac should accept valid PEM client identity material when:

client_cert points to a PEM chain file
client_key points to an unencrypted PKCS#8 PEM key
ca_cert points to a valid PEM CA file
Actual behavior
Rustguac fails before any Vault/OpenBao auth request with:

failed to parse client identity: builder error
Reproduction outline
Configure [vault] with ca_cert, client_cert, and client_key
Use valid PEM client identity material from OpenBao PKI
Start Rustguac
Observe immediate failure while building Vault TLS client identity

[davekempe]

Root cause identified: reqwest's Identity::from_pem() with the rustls backend is stricter than native-tls (OpenSSL) for PKCS#8 keys — which is why your standalone reproducer worked but rustguac didn't (your reproducer likely used default reqwest features which pull in native-tls/OpenSSL).

We've rewritten the mTLS client to build the rustls ClientConfig directly, bypassing Identity::from_pem() entirely. The new code:

• Parses certs and private key individually with rustls-pemfile (with detailed error messages for each step)
• Handles PKCS#8 (BEGIN PRIVATE KEY), RSA PKCS#1 (BEGIN RSA PRIVATE KEY), and EC (BEGIN EC PRIVATE KEY) keys
• Supports fullchain cert files (leaf + issuing CA) as OpenBao delivers them
• Feeds the parsed certs/key directly into rustls::ClientConfig::builder().with_client_auth_cert() — no round-trip through reqwest's PEM parser

Fix is in the upcoming v0.6.2 release. We also have test coverage for the exact scenario (PKCS#8 EC key + fullchain cert from a CA, matching the OpenBao PKI output).