A failed status update leaves the SolrBackup permanently stuck in progress

[autokctrl-team]

Environment

• solr-operator built from ed5c5c7d28a4c1189d19f581259e05385c0d4b20
• Solr 9.7.0
• Kubernetes v1.35.0 (kind v0.31.0, 3 nodes)
• A SolrBackup against a healthy single-node SolrCloud

What happened

A SolrBackup submitted a backup to Solr, but the operator failed to save InProgress=true to the apiserver. The backup then remained stuck.

This happens if the operator restarts after Solr accepts the backup but before the status update is saved. After the restart, the operator reads InProgress=false and submits the same backup again. Solr rejects the duplicate async ID, so the operator never polls or cleans up the original request.

The backup only recovered after I manually called DELETESTATUS. The next submission then succeeded and the backup completed.

Where the source code is wrong

reconcileSolrCollectionBackup decides what to do from the saved InProgress value. If the value is false, it submits a backup without first checking Solr:

// controllers/solrbackup_controller.go:285-300
if collectionBackupStatus.Finished {
    return true, nil
} else if !collectionBackupStatus.InProgress {
    started, err = util.StartBackupForCollection(...)   // no pre-check
    if err != nil {
        return true, err
    }
    collectionBackupStatus.InProgress = started          // in-memory only
    ...
} else if collectionBackupStatus.InProgress {
    // REQUESTSTATUS poll, and DELETESTATUS cleanup on finish
}

InProgress is only persisted at the end of Reconcile:

// controllers/solrbackup_controller.go:178-181
if !reflect.DeepEqual(unmodifiedBackupResource.Status, backup.Status) {
    err = r.Status().Patch(ctx, backup, client.MergeFrom(unmodifiedBackupResource))
}

If this patch does not complete after Solr accepts the backup, etcd still contains InProgress=false. Every later reconcile submits the same async ID. Solr rejects it, and the function returns before setting InProgress=true. This repeats indefinitely.

Solr keeps completed async records until DELETESTATUS is called. However, the operator only calls DELETESTATUS when InProgress=true, so it cannot clean up the record.

The cluster-operation code avoids this problem by checking Solr before submitting:

// controllers/util/solr_update_util.go:564-570
// First check to see if the Async Replace request has started
if asyncState, message, asyncErr := solr_api.CheckAsyncRequest(ctx, solrCloud, requestId); asyncErr != nil {
    ...
} else if asyncState == "notfound" {
    // Submit new Replace Node request

This code can detect an existing request even if an operator status update was lost. The backup code does not perform this check.

Expected behavior

A failed status update should not leave the backup stuck after Solr has accepted it.

Before submitting, the backup code should call CheckAsyncRequest with the async ID:

• notfound → submit the backup
• running / completed / failed → set InProgress=true and use the existing polling path