[URGENT] Libvirt Certificates Expired

[fnavidan]

Hi Guys

I'm running CS 4.20.2.0 in a single node where management and agent is running on the same machine. Libvirt certificates have been expired and I am not sure how to properly regenerate them manually as I cannot provision host keys neither using UK nor using CMK. Following certs have expiresd:

/etc/pki/libvirt/servercert.pem
/etc/pki/libvirt/clientcert.pem
/etc/cloudstack/agent/cloud.crt

As libvirt and agent are both failed, management is unable to provision host keys.

I have also tried following cmk command:

cmk provision certificate hostid=97cd673c-a651-4580-8a2c-f302265906b6 reconnect=true forced=true

Also note that no explicit ca plugin is specified in management or agent properties

Please guide me through proper way of host keys and agent cert recovery so that trust chain won't break.

[fnavidan]

I was able to recover the KVM host successfully using following steps.

1. Change default certificate validity period to 10 years:
   Global Settings -> ca.framework.cert.validity.period: Set to 3650 (~10 years)

2. Generate CSR:
   On KVM host run:
   /usr/share/cloudstack-common/scripts/util/keystore-setup
   /etc/cloudstack/agent/agent.properties
   /etc/cloudstack/agent/cloud.jks
   <KEYSTORE_PASSWORD>
   3650
   /etc/cloudstack/agent/cloud.csr

3. Issue certificate using cmk

   • On management server run:
     cmk issue certificate
     domain=
     csr="$(cat /etc/cloudstack/agent/cloud.csr)"
     duration=3650

   • Save "certificates" value returned in json format into /etc/cloudstack/agent/cloud.crt on KVM host.

   • Replace literal "\n" in json value with new line using:
     printf "%b" "$(cat /etc/cloudstack/agent/cloud.crt)" > /etc/cloudstack/agent/cloud.crt

4. Import certificate to keystore.
   On KVM host run:
   /usr/share/cloudstack-common/scripts/util/keystore-cert-import
   /etc/cloudstack/agent/agent.properties
   "<KEYSTORE_PASSWORD>"
   /etc/cloudstack/agent/cloud.jks
   ssh
   /etc/cloudstack/agent/cloud.crt
   "$(cat /etc/cloudstack/agent/cloud.crt | sed ':a;N;$!ba;s/\n/^/g;s/ //g')"
   /etc/cloudstack/agent/cloud.ca.crt
   "$(cat /etc/cloudstack/agent/cloud.ca.crt | sed ':a;N;$!ba;s/\n/^/g;s/ //g')"
   /etc/cloudstack/agent/cloud.key
   ""

5. Restart libvirtd and cloudstack-agent
   systemctl restart libvirtd
   systemctl restart cloudstack-agent