Timeout Restic

[jepangLee]

What steps did you take and what happened

i installed velero with eks and s3, annotate

backup.velero.io/backup-volumes: mysql-data

to pod of mysql stateful_set and backed up

but, only podVolumes can't backup

What did you expect to happen

backup stateful_set, pvc and pv's data

The output of the following commands will help us better understand what's going on

(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
  • deployment log
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
  • backup describe
• velero backup logs <backupname>
  • backup log

Anything else you would like to add

[Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero install command

  velero install \                          
      --provider aws \
      --plugins velero/velero-plugin-for-aws:v1.1.0 \
      --bucket $bucket \
      --backup-location-config region=$region \
      --use-volume-snapshots=false \
      --secret-file ./credentials-velero \
      --use-restic

• deployment/velero change point

  spec:
    template:
      spec:
        containers:
          - args:
           ...
            - --restic-timeout=1m
           ...
  
  # To produce results quickly
  # using default value are the same result

• resources

• Velero version (use velero version):

  Client:
          Version: v1.4.2
          Git commit: -
  Server:
          Version: v1.5.1
  
  

• Velero features (use velero client config get features):

  features: <NOT SET>

• Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-4c6976", GitCommit:"4c6976793196d70bc5cd29d56ce5440c9473648e", GitTreeState:"clean", BuildDate:"2020-07-17T18:46:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

  Cloud provider: S3
  k8s cluster: EKS

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[jepangLee]

and i have a question

--use-restic option creates only the Restic Daemonset, other resources don't change
how do Velero and Restic communicate?

[carlisia]

Hey @jepangLee, I see you are using the Velero client version 1.4.2 and the server version 1.5.1. Those two must have the same version, else you might run into problems. Please fix this issue and perform your backup again. If you continue to see any problem, please gather new logs for us.

[ashish-amarnath]

@jepangLee Wrt

--use-restic option creates only the Restic Daemonset, other resources don't change
how do Velero and Restic communicate?

The restic daemon set and velero communicate with each other as a result of the creation of custom resources of types podvolumebackup and podvolumerestores.

The Velero server, running as a pod in the cluster, on detecting that certain pod volumes need to be backed up using restic creates podvolumebackup objects. The creation of these objects triggers the reconciliation loop that is run by the restic daemonset pod that is running on the same node on which the pod whose volume we want to be backed up. The corresponding daemonset pod then runs restic commands to copy the pod volume's filesystem as a restic snapshot.

[ashish-amarnath]

@jepangLee To troubleshoot this issue further, can you please provide the following information:

1. Insepct the podvolumebackup (PVB) status to identify the failed or timed out PVB using the below command

   kubectl -n velero get podvolumebackup -ojson | jq .items[].status
   

2. If you have the pod whose pod volume failed to backup, identify the restic daemonset pod that was responsible for backing up this volume as shown below

   kubectl -n velero get pods -l name=restic -owide | grep `kubectl -n <YOUR_NAMESPACE> get po <POD_NAME> -owide | awk '{ print $7}' | tail -n1` | awk '{print $1}'
   

   and obtain logs from that pod using

   kubectl -n velero logs <RESTIC_POD_NAME_FROM_PREVIOUS_STEP>
   

[carlisia]

This issue has been inactive for a while so I'm closing it.