Controller rotation reconciliation: single-writer baseline and multi-process boundary

[yshyn-iohk]

Context

PR #432 addresses the partial-failure hazards tracked by #240, including controller rotation/recovery when a transaction may have finalized but the client loses the receipt or finality connection.

The current implementation persists a replacement controller secret before submission, retains it after an ambiguous outcome, and uses a process-local fail-fast reservation to prevent overlapping mutation of the same DID private state.

This discussion asks maintainers and integrators to confirm the intended deployment and reconciliation model before we add distributed coordination complexity.

Proposed near-term architecture

Adopt a single-writer-per-DID invariant for the current API baseline:

• Only one application process updates a given Midnight DID at a time.
• Within that process, overlapping rotation, recovery, promotion, discard, join, or deployment work fails immediately rather than queueing.
• The reservation has no elapsed-time expiry. It is released only when the owning operation settles or the process exits; releasing it while provider or transaction work may still execute would permit stale mutation.
• A process restart clears the in-memory reservation, but it does not clear durable pending controller state. The application must reconcile before another mutation.

Under this invariant, an external coordinator is not required for the normal supported deployment. An integrator that intentionally permits multiple processes to update the same DID must provide a distributed lock/fencing design; that is a separate architecture with separate operational complexity and should not be implied by the basic API.

Ambiguous rotation/recovery outcome

A successful method return or transaction receipt is useful evidence, but the authoritative success criterion should be the latest trusted finalized smart-contract state.

When submission/finality fails because connectivity is lost:

1. Retain the pending replacement secret. Do not delete it, generate a replacement, or retry blindly.
2. Wait for ledger/indexer connectivity and finalized-state observation to recover.
3. Read the DID contract version, activation state, and controllerPublicKey.
4. Verify that the retained secret derives the observed controller public key.
5. If the candidate key is the finalized current controller key, treat current controller custody as successful and promote/recover the retained private state.
6. If the original version/key is still observed, keep the outcome pending until non-finalization is authoritative; a first restored read alone must not cause candidate deletion.
7. If a different mutation consumed the expected version or another controller key is current, report a conflict/superseded outcome and require explicit recovery handling.

The important invariant is: never discard the only secret that may correspond to the finalized on-ledger controller key.

Explicit boundary

The current private-state provider exposes no cross-process compare-and-set operation. Therefore this proposal does not claim transparent safety for multiple independent writer processes.

If multi-process DID writers become a requirement, the design should explicitly add a distributed lock or provider-enforced fencing/CAS mechanism, including ownership, lease renewal, stale-owner rejection, recovery, and operational monitoring. That should be designed and reviewed separately rather than hidden inside the single-process API.

Questions for maintainers and integrators

1. Is one writer process per DID an acceptable normative assumption for the 0.6 API baseline?
2. What finalized-state and freshness guarantees can the ledger/indexer provider expose to reconciliation code?
3. Should the API replace caller-supplied rotationFinalized: true/false assertions with a reconcilePendingControllerOperation helper that reads and classifies ledger state?
4. Should the supported restart procedure be: restore connectivity, read finalized ledger state, reconcile pending private state, then permit new mutations?
5. Is multi-process mutation explicitly a non-goal until a distributed locking/fencing design is proposed?

Related: #432, #240.

[yshyn-iohk]

The proposed baseline is approved for the current API:

• one application writer process per DID;
• fail-fast, no-expiry ownership for the protected controller/private-state lifecycle within that process;
• retain the pending candidate after ambiguous submission/finality failure;
• after connectivity returns, use provider-specific authoritative finalized evidence before asserting promotion or discard;
• intentional multi-process writers are outside the baseline and must provide a distributed lock or equivalent fencing mechanism.

PR #432 now records this decision in the API README, 0.6 migration guide, method specification, changelog, and tracked retrospective at exact head 50074d8b6f2b391b3a2b539a6630f7e77108f054. The docs also state that rotationFinalized is an unchecked caller assertion and that the current ledger helper does not add a finality/freshness guarantee to the configured public data provider.

Exact-head review and CI have been requested for the updated PR.