Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to determine if MedusaBackupJob and MedusaRestoreJob are completed successfully? #1302

Open
c3-clement opened this issue Apr 29, 2024 · 0 comments
Labels
question Further information is requested

Comments

@c3-clement
Copy link
Contributor

I'm opening this issue in the context of #1301

According to https://docs.k8ssandra.io/tasks/backup-restore/#checking-backup-completion, MedusaBackupJob and MedusaRestoreJob are completed successfully when the field status.finishTime is set.
However, this does not seem to be accurate.

In k8ssandra-operator 1.14 , I managed to obtain this MedusaRestoreJob:

apiVersion: v1
items:
- apiVersion: medusa.k8ssandra.io/v1alpha1
  kind: MedusaRestoreJob
  metadata:
    creationTimestamp: "2024-04-26T17:10:41Z"
    generateName: k8scass-cs-001-k8scass-001-k9sbq-
    generation: 1
    labels:
      app: c3aiops
      ops.c3.ai/parent-resource: eksdoucy-c3cassandrarestore-k8scass-cs-001-6pr92
      role: c3aiops-C3CassandraRestore
    name: k8scass-cs-001-k8scass-001-k9sbq-rh4hx
    namespace: eksdoucy
    ownerReferences:
    - apiVersion: cassandra.datastax.com/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: CassandraDatacenter
      name: k8scass-001
      uid: 2ad0341f-cb57-425e-9a3f-533a31a83152
    resourceVersion: "5691659"
    uid: 596f85ce-01da-464a-a666-482f330090ae
  spec:
    backup: k8scass-cs-001-k8scass-001-k9sbq
    cassandraDatacenter: k8scass-001
  status:
    finishTime: "2024-04-26T17:10:56Z"
    message: target backup has not completed successfully
    restoreKey: 6dca1918-1fdf-47dd-86b1-3f29f0b14c9b
    restoreMapping: {}
    startTime: "2024-04-26T17:10:56Z"
kind: List
metadata:
  resourceVersion: ""

The field finishTime is set ""2024-04-26T17:10:56Z", but the restoration failed with the error "target backup has not completed successfully" (see status.message field)
How to determine when the restoration succeeded or failed?

Regarding MedusaBackupJob, I noticed that medusarestorejob_controller is doing multiple checks to validate a MedusaBackup : https://github.com/k8ssandra/k8ssandra-operator/blob/main/controllers/medusa/medusarestorejob_controller.go#L287
Therefore, it's possible to have MedusaBackupJob with finishTime set with an invalid associated MedusaBackup.

We are building automations to orchestrate backups and restorations. We also export backup status to Prometheus.

Could you clarify the correct way to identify when MedusaBackupJob / MedusaRestoreJob are:

  • in progress
  • failed
  • succeeded

Thanks!

@c3-clement c3-clement added the question Further information is requested label Apr 29, 2024
@c3-clement c3-clement changed the title Clarify the correct way to determine if MedusaBackupJob and MedusaRestoreJob are completed successfully How to determine if MedusaBackupJob and MedusaRestoreJob are completed successfully? Apr 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
Status: No status
Development

No branches or pull requests

1 participant