Nightly test failure uninstalling PKI

[flo-renaud]

The IPA tests detected a regression during uninstallation when the pki packages from the copr repository @pki/master are used.
See PR #2434 with the test test_backup_and_restore_TestBackupReinstallRestoreWithKRA (Report, logs).

The test scenario is the following:

• install ipa server with KRA
• create a vault and archive data
• backup
• uninstall ipa server
• restore ipa server

The simplest reproducer is:

• install ipa server
• uninstall ipa server

The ipa-server-install --uninstall command succeeds but prints an error related to the failure of pkidestroy. The PKI server is not uninstalled and any tentative IPA reinstallation would fail because the instance has been left in place.

Logs from pki-ca-destroy are available here and show that the command pkidestroy -i pki-tomcat -s CA is unable to unregister CA subsystem:

2023-02-15 15:42:43 WARNING: Unable to unregister CA subsystem from security domain: POST /ca/agent/ca/updateDomainXML HTTP/1.0

This failure is not surprising as IPA uninstaller is stopping the services before it calls pkidestroy. In the past, this didn't cause any issue but I suspect that commit b6b6d0b has changed the behavior.
IMO it should be possible to call pkidestroy even if the pki service is stopped.

Companion issue reported against IPA: https://pagure.io/freeipa/issue/9330

[edewata]

So this is an existing problem, but pkidestroy had been hiding it, and now the problem is exposed, which is one of the intentions of commit b6b6d0b.

Is there a reason for IPA to stop PKI server before removing it? The way it's designed now is that PKI subsystems are registered to a security domain which is also running on the same PKI server. So in order to remove the CA properly, PKI server needs to be running such that pkidestroy can deregister the CA from the security domain.

My suggestion for now is to fix IPA to keep PKI server running and let pkidestroy complete the uninstallation which will eventually stop PKI server as well.

[edewata]

Since the behavior of pkidestroy is correct (i.e. failing when the security domain is down), I'm going to close this ticket.

If we want to be able to use pkidestroy while the server is down (i.e. without the security domain), I think IPA will need to remove its dependency on security domain first, then we can update PKI to support installation/uninstallation without security domain.

[rcritten]

Does this mean that if the CA (or other services I assume) cannot be started then they cannot be uninstalled?

[edewata]

Does pkidestroy --force work? I have not tried it for this particular scenario.

Currently the security domain service is running in CA subystem. If the CA subsystem is down the security domain service will be down too, so it won't be possible to notify the security domain service about the removal of another subsystem.

It might be possible to forcibly remove a subsystem with the above command, but you might end up with outdated information in the security domain database. If IPA is relying on this information for something, it might have a problem later, so you'll need to clean up the outdated information manually once you manage to bring the security domain service (i.e. the CA subsystem) back up, or clean it up directly from the security domain database (we can provide CLIs for this, but it can only run locally on CA's machine).

There might be other options, but they probably require major redesign.

[rcritten]

We'll have to see. We did something similar for the KRA a few years ago, trying to ensure that the services are running at shutdown but not treating them as a blocker. We don't pass --force though. I think we can work this out in IPA.