-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Restore fails to complete with error on AzureFile volumes #3027
Comments
This error is in the backup-logs.txt: lookup minio-server.minio.svc: no such host There's some issue with resolving the Minio service. Can you verify that that is the correct service name and that it is resolvable from the Velero deployment? |
I confirmed the address minio-server.minio.svc is reachable from a different pod. I see the same error in the backup logs, but the backup is at-least partially successful and I see the files in my blob-container, so I assumed the error is momentary and cleared up later. Also the actual error in the restore is |
OK, so that appears to be a Kubernetes DNS resolution issue. Please see if you can get a successful restore. Closing for now, if the problem continues please re-open and we can help you troubleshoot it a bit. |
I commented this is not a DNS issue. The actual error in the restore logs is this |
Hi @sivaramsk. I'd like to check that the snapshot that is being restored exists in your storage location, that will help us in determining why this error is being triggered. Can you check the following path in your storage location |
One thing I did not mention in the original description. After the backup is completed, I cleared the resources running in the cluster, like delete the deployments, statefulsets, pv, pvcs, etc, and then did the restore in the same kubernetes cluster. Would the restic snapshot continue to work even after the source data volume is deleted? |
Hi @sivaramsk. Thanks for providing that information and extra context! Removing those resources should not be an issue as the restic backup should be in your configured backup storage location. |
@zubron I am not sure where you got the snapshot ID
Where the snapshot ID causing the restore to fail is |
Unfortunately I don't have the setup as of now, but will try to reproduce it again and update. |
@ashish-amarnath You're right, I messed up there, sorry 😞 There were a number of restores that failed in this particular setup and I don't know how I managed to copy the details for a pod volume restore that succeeded 🤦♀️ Looking through the list of PodVolumeRestores again though, I noticed something suspicious for all the snapshots that couldn't be found ( For example: Snapshot ID This is the case for all the failed restores showing the Snapshot ID It seems like this is definitely a bug in Velero where it is attempting to use the wrong Snapshot ID for certain PodVolumeRestores. |
I took another look at this and found where it is failing. We call In the case where there pods of the same name, but have differently named volumes, the restic restore will fail because it will attempt to mount all volumes in the restic init container that match that a given pod name, even if it is for a pod in a different namespace: https://gist.github.com/zubron/dce63c10b5aea026f988b8a14d2934c5 |
What steps did you take and what happened:
What did you expect to happen:
Restore to become successful.
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)
https://gist.github.com/sivaramsk/62ce61275335de296989378482f5aeb4
Anything else you would like to add:
Storageclass was patched with nouser_xattr before the backup.
Environment:
Velero version (use
velero version
):Client:
Version: v1.5.1
Git commit: -
Server:
Version: v1.5.1
Velero features (use
velero client config get features
): features:Kubernetes version (use
kubectl version
):Client Version: 1.18.8
Server Version: 1.17.11
Kubernetes installer & version: AKS, created with terraform aks modules
Cloud provider or hardware configuration: AKS
Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.
The text was updated successfully, but these errors were encountered: