Issuer Compromise Recovery

Documentation for the image tag 1.0.0-rc1. See Versions for the documentation of a release.

Issuer compromise recovery

What to do when the private key of a Qertum CA is, or may be, in someone else's hands. Written for the person who holds the admin role and, for the offline cases, access to the offline root.

Case A is rehearsed by an automated test against the real API and the real offline root tool (IssuerCompromiseRehearsalTests); case B's trust-bundle behaviour is pinned by the same test class; case C is covered by ForceReplaceTests. What the rehearsals do not cover is listed under What was not rehearsed. Read Known gaps before you rely on this in an incident.

Which case is it?

Case	What leaked	Typical setup	Trust stores change?
A. Issuing intermediate	the key of an online intermediate; the offline root is intact	offline root + online intermediate (the recommended setup)	no, the root is unchanged
B. Offline root	the key of the offline root	the same setup, root media or passphrase lost	yes, every relying party must drop the old root
C. Single online root	the one CA key on the server	small-customer single container, no offline root	yes, every relying party

If you are unsure which key leaked, treat it as the higher case. A key that sat on the server (anything Online in qca issuers list) is exposed if the server was breached. An Offline root is exposed only if its media or passphrase were.

Before an incident (ten minutes, once)

• Know where the offline root directory is, who knows its passphrase, and that the machine that runs qca root can reach the files you will carry (request file in, certificate and CRL out).
• Keep an off-host audit anchor: a webhook subscribed to AuditCheckpoint pointing at another host (single-container README, "Tamper-evident audit log"). After an incident it is how you show the audit log was not rewritten.
• List the templates pinned to an issuer (Web: Templates, or GET /v1/certificate-templates); a template pinned to an issuer that can no longer sign refuses to issue (409), it never falls back to another issuer.
• List where intermediates and roots are installed outside Qertum (servers that present the chain, trust stores, MDM profiles). That list is the long part of any recovery.

Case A: the issuing intermediate is compromised

Goal: stop it issuing, make relying parties distrust it, move every service to a new intermediate. The root, and so every trust store, stays as it is.

1. Describe the old intermediate. qca issuers list, then qca issuers show <old-id>. Note its serial number (hex), name constraints and path length: the replacement must be at least as strict.
2. Create the replacement. qca issuers create-intermediate --name "Issuing CA 2" --cn "Example Issuing CA 2" --permit-dns example.com (repeat the constraints from step 1). It is PendingCertificate and prints a request. qca issuers request <new-id> --file issuing-ca-2.request.json saves the request file.
3. Sign it on the offline root. On the offline machine: qca root sign --dir ./offline-root --request issuing-ca-2.request.json (asks for the passphrase) writes the certificate and chain files.
4. Install it. qca issuers install <new-id> --cert issuing-ca-2.pem --chain issuing-ca-2.chain.pem.
5. Make it the default. qca issuers make-default <new-id> --yes. From now on templates that are not pinned are signed by the replacement.
6. Re-pin templates that were pinned to the old intermediate: Web, Templates, issuer select, or PUT /v1/certificate-templates/{id}/issuer (settings.edit). Do this before step 8, or those templates' renewals are refused.
7. Revoke the old intermediate at the root. On the offline machine: qca root crl --dir ./offline-root --revoke <old-serial-hex> --reason keyCompromise writes root.crl. Upload it: qca issuers upload-crl <root-issuer-id> root.crl. Qertum then marks the old intermediate Retired and not default, publishes the root CRL at /crl/<root-issuer-id>, and shows the issuer revoked item on the dashboard. Relying parties that check revocation now reject chains through the old intermediate.
8. Replace the certificates it issued. qca bulk renew --issuer <old-id> previews the set, then qca bulk execute <operation-id> --confirm-count N --wait. Certificates whose key the client holds (ACME and EST) are not renewed this way: those clients re-enroll on their next order, which now goes to the replacement. CA-capable certificates are renewed by ceremony, not in bulk.
9. Revoke the originals (belt and braces). qca bulk revoke --issuer <old-id> --reason "issuer key compromised" after the replacements are deployed. It is optional once step 7 is done, but it puts every serial on a CRL and in the audit trail. A revoked certificate still counts as unexpired for archiving (see gap 1).
10. Distribute the new chain to every service that presents an intermediate: GET /v1/issuers/<new-id>/chain (anonymous), or qca issuers trust-bundle for the roots. The root did not change, so no trust store changes.
11. Keep the evidence. qca audit checkpoint --output json > incident-anchor.json now, and qca audit verify on a copy of the audit logs later. Export the audit log entries around the incident (qca audit list).
12. Archive when everything it signed has expired (or been replaced and expired): qca issuers archive <old-id> --yes deletes its key.

Done when: qca issuers list shows the replacement Active and default and the old one Retired; a new certificate chains to the same root; GET /crl/<root-issuer-id> lists the old intermediate's serial.

Case B: the offline root is compromised

Nothing above the root can revoke it, so the only recovery is a new root and getting every relying party to trust it instead.

1. Make a new root somewhere safe: qca root init --dir ./offline-root-2 (new passphrase, new media).
2. Create, sign and install a new intermediate under it exactly as in case A steps 2-4 but signing with the new root directory. Installing it registers the new root.
3. Make the new intermediate the default (qca issuers make-default) and retire the old intermediate (qca issuers retire <old-id> --yes). Re-pin templates (case A step 6).
4. Replace every certificate: qca bulk renew --issuer <old-id>, then qca bulk execute. Revoke the originals the same way as in case A.
5. Re-trust. This is the long step and Qertum cannot do it: install the new root in every trust store and remove the old one, on every device, server and client that trusted it. Until the old root is removed from a store, anyone with its key can mint certificates that store accepts.
6. Do not distribute trust from Qertum's own bundle until the old issuers are archived. GET /v1/issuers/trust-bundle lists the root of every Active and Retired issuer, so the compromised root stays in it while the old intermediate is Retired (see gap 2). Build your trust distribution from the new root alone.

Case C: single online root (no offline root)

The CA key lived on the server, so the whole CA is replaced.

1. Replace the CA: Web, Onboarding or Settings, generate or import a new root with force (POST /v1/ca/generate?force=true or POST /v1/ca/import?force=true, admin only, confirm must be exactly REPLACE <current CA name>). Generating needs the new RootCA request approved (independent approval applies); importing replaces at once. The old issuer becomes Destroyed: its key and files are deleted and its root drops out of the trust bundle.
2. Replace every certificate (renew in bulk, or re-issue from the templates) and re-trust the new root everywhere, as in case B step 5.
3. Check GET /v1/issuers/trust-bundle holds only the new root.

Expected recovery time

Part	Time	Basis
Qertum steps of case A (create, sign offline, install, make default, sign and upload root CRL, verify)	about 0.4 s of machine time	measured by the rehearsal on a development machine: 91 + 12 + 52 + 21 + 175 ms
Reach the offline root, authorize, enter the passphrase, carry files across	15-60 min	estimate, not measured
Re-pinning templates, running bulk renew and revoke	minutes of operator time, plus the queue consumer's processing time	estimate, the bulk steps are not rehearsed
Deploying new certificates and the new chain to every service	hours to days, depending on how many and how automated	estimate; usually the longest part of case A
Case B/C trust redistribution to every relying party	days to weeks	estimate; outside Qertum

Treat the estimates as placeholders: run the rehearsal below, then a hands-on drill with your own setup, and replace them with your numbers.

Rehearsing

REHEARSAL_TIMINGS_FILE=timings.txt dotnet test Qertum.sln --filter "FullyQualifiedName~IssuerCompromiseRehearsalTests"

It plays case A against real services and the real offline root tool, asserts the end state (replacement default, old intermediate Retired, root CRL lists it, new certificates chain to the same root, trust bundle unchanged, archive refused while certificates are unexpired) and writes the per-step machine times to the file. The second test pins case B's trust-bundle behaviour. Rerun it after any change to issuers, the CRL upload or the trust bundle.

For a hands-on drill, do the case A steps above against a non-production stack with a throwaway offline root, time each step including the people, and record the date and the result in your own runbook.

What was not rehearsed

• Bulk renew and bulk revoke (steps 8-9): the test host has no queue consumer, so they are described from the command help, not exercised here. The bulk machinery has its own tests (BulkOperationLifecycleTests).
• ACME and EST clients following the change of default issuer, and real relying parties (browsers, OS stores, servers) acting on the root CRL.
• OCSP for certificates of the retired issuer (covered separately by IssuerOcspTests).
• Anything involving real people, offline media or networks.

Known gaps

1. The compromised key stays on the host until its last certificate expires. A Retired issuer cannot be archived while any certificate it signed is unexpired (revoked ones count), and the only early deletion is for a PendingCertificate issuer or a force-replaced default. If the server itself was breached, rotate the data volume's secrets and treat the host as untrusted; Qertum has no "destroy this retired issuer's key now".
2. A compromised root keeps being served in the trust bundle while its issuers are Retired, as in case B step 6. Only an Archived or Destroyed issuer drops out.
3. A template pinned to the compromised intermediate refuses to issue until you re-pin it (case A step 6). That is the safe behaviour, but it is easy to forget during an incident.

Gaps 1 and 2 are known limits of the current release.