Allow acquire _CrossProcessLock on owner kill

[stanislavlevin]

For now the lock can be acquired only if the previous owner has
released filelock or on timeout (1h).
There are cases when the owner dies before release the lock (e.g. on upgrade).
This leads to lock-waiting for expire date by all the other processes
being synced.

An additional check for owner status was added. If a previous owner
is dead then an awaiting process becomes a new lock owner.

Fixes: https://pagure.io/freeipa/issue/7924

[tiran]

@frasertweedale please take a look

@stanislavlevin I think this is logic is not correct. There are places where a process acquires a lock and then exists, e.g. stop_pkicad. I don't fully understand the locking logic. Fraser is your man.

[flo-renaud]

Hi @stanislavlevin
as the name implies, the lock is not tied to a given process. For instance, during the RA certificate renewal, certmonger launches a pre-save command that takes the lock (renew_ra_cert_pre), then saves the certificate and finally launches a post-save command that releases the lock (renew_ra_cert).
In this case, the lock does not store the PID of the renew_ra_cert_pre process but rather its name ('renew_ra_cert') in the owner attribute.
So it is completely legit to have a 'dead' owner and the proposed fix is not suitable.

Which error did you actually see on upgrade?

[stanislavlevin]

mmm, I see.
There is no error, certmonger just stuck on 'SUBMITTING' and it's not obvious why (from admin/user point of view) and what to do.

[stanislavlevin]

@flo-renaud , ok, let's close this PR and move the discussion to the opened issue (https://pagure.io/freeipa/issue/7924)