Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

timeout on pre hook error #2599

Closed
Seb0042 opened this issue Jun 2, 2020 · 2 comments · Fixed by #2610
Closed

timeout on pre hook error #2599

Seb0042 opened this issue Jun 2, 2020 · 2 comments · Fixed by #2610
Assignees
Labels
Enhancement/User End-User Enhancement to Velero

Comments

@Seb0042
Copy link

Seb0042 commented Jun 2, 2020

What steps did you take and what happened:
I added annotations on the pod to be saved. As it has a prehook command that can exceeds 30 seconds I add a number above 30 for timeout.

metadata:
  annotations:
    backup.velero.io/backup-volumes: backup-test
    pre.hook.backup.velero.io/command: '["/bin/bash","/extras/velero_backup.bash"]'
    pre.hook.backup.velero.io/container: test
    pre.hook.backup.velero.io/timeout: "300"

When I launch the velero backup create I have an error about timeout on the prehook command.

time="2020-06-02T16:47:41Z" level=error msg="Error executing hook" backup=production-system/test error="timed out after 30s" error.file="/github.com/vmware-tanzu/velero/pkg/pod
exec/pod_command_executor.go:168" error.function="github.com/vmware-tanzu/velero/pkg/podexec.(*defaultPodCommandExecutor).ExecutePodCommand" hookPhase=pre hookSource=annotation
 hookType=exec logSource="pkg/backup/item_hook_handler.go:100" name=mypod namespace=myns resource=pods
time="2020-06-02T16:47:41Z" level=error msg="Error backing up item" backup=production-system/test error="timed out after 30s" error.file="/github.com/vmware-tanzu/velero/pkg/po
dexec/pod_command_executor.go:168" error.function="github.com/vmware-tanzu/velero/pkg/podexec.(*defaultPodCommandExecutor).ExecutePodCommand" logSource="pkg/backup/backup.go:45
4" name=mypod

What did you expect to happen:
I expect that timeout of 300 seconds would be applied.
NOTE: when I change the timeout, I have to put "" around the number. I if don't do that I have an error because (int64 instead of a string).

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

  • velero backup describe <backupname> :
Name:         test
Namespace:    velero
Labels:       velero.io/storage-location=minio
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.18.2
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=18

Phase:  PartiallyFailed (run `velero backup logs test` for more information)

Errors:    2
Warnings:  0

Namespaces:
  Included:  production-app
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  minio

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-06-02 16:47:10 +0000 UTC
Completed:  2020-06-02 16:47:42 +0000 UTC

Expiration:  2020-07-02 16:47:10 +0000 UTC

Total items to be backed up:  271
Items backed up:              271

Velero-Native Snapshots: <none included>
  • velero backup logs <backupname>: can't give you all the logs, too much to anonymize. I gave the two lines with error above.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

  • Velero version (use velero version):
    Client:
    Version: v1.4.0
    Git commit: 5963650
    Server:
    Version: v1.4.0

  • Velero features (use velero client config get features): N/A

  • Kubernetes version (use kubectl version): v1.18.2

  • Kubernetes installer & version: kubeadm v1.18.2

  • Cloud provider or hardware configuration: Vmware virtual machines

  • OS (e.g. from /etc/os-release): CentOS 7

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

  • 👍 for "I would like to see this bug fixed as soon as possible"
  • 👎 for "There are more important bugs to focus on right now"
@nrb
Copy link
Contributor

nrb commented Jun 4, 2020

Hi @Seb0042.

pre.hook.backup.velero.io/timeout: "300" requires a unit at the end; it doesn't automatically apply one. So the appropriate value would be "300s".

I looked at the code really quickly, and it appears there's a TODO for us to explicitly log this. I'll open a PR to do that.

@nrb nrb self-assigned this Jun 4, 2020
@nrb nrb added the Enhancement/User End-User Enhancement to Velero label Jun 4, 2020
@Seb0042
Copy link
Author

Seb0042 commented Jun 5, 2020

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement/User End-User Enhancement to Velero
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants