Helm installed Velero uses restic even when restic is not enabled

[gsc-k8s-config-management]

I'm attempting to get Velero working in our clusters using the helm chart.

Here is my values.yaml, with some values interpolated by our CD process:

##
## Configuration settings that directly affect the Velero deployment YAML.
##

# Details of the container image to use in the Velero deployment & daemonset (if
# enabling restic). Required.
image:
  repository: velero/velero
  tag: v1.3.1
  pullPolicy: IfNotPresent

# Annotations to add to the Velero deployment's pod template. Optional.
#
# If using kube2iam or kiam, use the following annotation with your AWS_ACCOUNT_ID
# and VELERO_ROLE_NAME filled in:
#  iam.amazonaws.com/role: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<VELERO_ROLE_NAME>
podAnnotations: {}

# Resource requests/limits to specify for the Velero deployment. Optional.
resources: {}

# Init containers to add to the Velero deployment's pod spec. At least one plugin provider image is required.
initContainers:
   - name: velero-plugin-for-gcp
     image: velero/velero-plugin-for-gcp:v1.0.1
     imagePullPolicy: IfNotPresent
     volumeMounts:
       - mountPath: /target
         name: plugins

# SecurityContext to use for the Velero deployment. Optional.
# Set fsGroup for `AWS IAM Roles for Service Accounts`
# see more informations at: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
securityContext: {}
  # fsGroup: 1337

# Tolerations to use for the Velero deployment. Optional.
tolerations: []

# Node selector to use for the Velero deployment. Optional.
nodeSelector: {}

# Extra volumes for the Velero deployment. Optional.
extraVolumes: []

# Extra volumeMounts for the Velero deployment. Optional.
extraVolumeMounts: []

# Settings for Velero's prometheus metrics. Enabled by default.
metrics:
  enabled: true
  scrapeInterval: 30s

  serviceMonitor:
    enabled: true
    additionalLabels:
      release: collection

##
## End of deployment-related settings.
##


##
## Parameters for the `default` BackupStorageLocation and VolumeSnapshotLocation,
## and additional server settings.
##
configuration:
  # Cloud provider being used (e.g. aws, azure, gcp).
  provider: gcp

  # Parameters for the `default` BackupStorageLocation. See
  # https://velero.io/docs/v1.0.0/api-types/backupstoragelocation/
  backupStorageLocation:
    # Cloud provider where backups should be stored. Usually should
    # match `configuration.provider`. Required.
    name: gcp
    # Bucket to store backups in. Required.
    bucket: ${serviceVariable.bucket}
    # Prefix within bucket under which to store backups. Optional.
    prefix: ${serviceVariable.cluster}


  # Parameters for the `default` VolumeSnapshotLocation. See
  # https://velero.io/docs/v1.0.0/api-types/volumesnapshotlocation/
  volumeSnapshotLocation:
    # Cloud provider where volume snapshots are being taken. Usually
    # should match `configuration.provider`. Required.,
    name: gcp
    # Additional provider-specific configuration. See link above
    # for details of required/optional fields for your provider.
    config:
      # The GCP location where snapshots should be stored. See the GCP documentation
      # (https://cloud.google.com/storage/docs/locations#available_locations) for the
      # full list. If not specified, snapshots are stored in the default location
      # (https://cloud.google.com/compute/docs/disks/create-snapshots#default_location).
      #
      # Optional.
      snapshotLocation: "${serviceVariable.snapshot_zone}"
      # The project ID where existing snapshots should be retrieved from during restores, if 
      # different than the project that your IAM account is in. This field has no effect on 
      # where new snapshots are created; it is only useful for restoring existing snapshots 
      # from a different project.
      # 
      # Optional (defaults to the project that the GCP IAM account is in).
      project: "${serviceVariable.project}"

  # These are server-level settings passed as CLI flags to the `velero server` command. Velero
  # uses default values if they're not passed in, so they only need to be explicitly specified
  # here if using a non-default value. The `velero server` default values are shown in the
  # comments below.
  # --------------------
  # `velero server` default: 1m
  backupSyncPeriod: 1m
  # `velero server` default: namespaces,persistentvolumes,persistentvolumeclaims,secrets,configmaps,serviceaccounts,limitranges,pods
  restoreResourcePriorities: namespaces,persistentvolumes,persistentvolumeclaims,secrets,configmaps,serviceaccounts,limitranges,pods
  # `velero server` default: false
  restoreOnlyMode: false

  # additional key/value pairs to be used as environment variables such as "AWS_CLUSTER_NAME: 'yourcluster.domain.tld'"
  extraEnvVars: {}

  # Set log-level for Velero pod. Default: info. Other options: debug, warning, error, fatal, panic.
  logLevel: info

  # Set log-format for Velero pod. Default: text. Other option: json.
  logFormat: json

##
## End of backup/snapshot location settings.
##


##
## Settings for additional Velero resources.
##

rbac:
  # Whether to create the Velero role and role binding to give all permissions to the namespace to Velero.
  create: true
  # Whether to create the cluster role binding to give administrator permissions to Velero
  clusterAdministrator: true

# Information about the Kubernetes service account Velero uses.
serviceAccount:
  server:
    create: true
    #name:
    #annotations:

# Info about the secret to be used by the Velero deployment, which
# should contain credentials for the cloud provider IAM account you've
# set up for Velero.
credentials:
  # Whether a secret should be used as the source of IAM account
  # credentials. Set to false if, for example, using kube2iam or
  # kiam to provide IAM credentials for the Velero pod.
  useSecret: true

  # Name of a pre-existing secret (if any) in the Velero namespace
  # that should be used to get IAM account credentials. Optional.
  existingSecret: velero

  # Data to be stored in the Velero secret, if `useSecret` is
  # true and `existingSecret` is empty. This should be the contents
  # of your IAM credentials file.
  # secretContents:

# Whether to create backupstoragelocation crd, if false => do not create a default backup location
backupsEnabled: true
# Whether to create volumesnapshotlocation crd, if false => disable snapshot feature
snapshotsEnabled: true

# Whether to deploy the restic daemonset.
deployRestic: false

# Backup schedules to create.
# Eg:
# schedules:
#   mybackup:
#     schedule: "0 0 * * *"
#     template:
#       ttl: "240h"
#       includedNamespaces:
#        - foo
schedules:
  fullcluster:
    schedule: "${serviceVariable.backup_schedule}"
    template:
      ttl: "${serviceVariable.backup_ttl}"

# Velero ConfigMaps.
# Eg:
# configMaps:
#   restic-restore-action-config:
#     labels:
#       velero.io/plugin-config: ""
#       velero.io/restic: RestoreItemAction
#     data:
#       image: gcr.io/heptio-images/velero-restic-restore-help
configMaps: {}

##
## End of additional Velero resource settings.
##

Although I have nothing in here indicating restic should be used, the only way that Velero is attempting to backup persistent volumes is using restic. This fails since the daemonset is not deployed, so backups hang. I want Velero to use gcp snapshots, but cannot find a way to disable restic, which seems opposite of all documentation implying I need to explicitly enable it. Hopefully I am missing something obvious, thanks for any assistance!

[thedevelopnik]

I accidentally opened this as one of our service accounts, but I swear it's a real person behind it :).

[nrb]

@thedevelopnik What information are you seeing that makes you think Velero is using restic? Could you provide that? I'm guessing it's that a velero backup never completes?

If that's the case, could you post the output of velero backup describe of an affected backup please?

Also, did you label any pods with the backup.velero.io/backup-volumes label?

[thedevelopnik]

Thanks for the quick response @nrb !

Yes, I did include that annotation based on a guide I found, is that not necessary for non-restic backups?

Here's a sampling of logs from output of k logs -n velero <velero-pod> | grep restic

{"command":"/velero","kind":"RestoreItemAction","level":"info","logSource":"pkg/plugin/clientmgmt/registry.go:100","msg":"registering plugin","name":"velero.io/restic","time":"2020-03-10T21:38:54Z"}
{"level":"warning","logSource":"pkg/cmd/server/server.go:478","msg":"Velero restic daemonset not found; restic backups/restores will not work until it's created","time":"2020-03-10T21:39:01Z"}
{"controller":"restic-repository","level":"info","logSource":"pkg/controller/generic_controller.go:76","msg":"Starting controller","time":"2020-03-10T21:39:01Z"}
{"controller":"restic-repository","level":"info","logSource":"pkg/controller/generic_controller.go:79","msg":"Waiting for caches to sync","time":"2020-03-10T21:39:01Z"}
{"controller":"restic-repository","level":"info","logSource":"pkg/controller/generic_controller.go:83","msg":"Caches are synced","time":"2020-03-10T21:39:01Z"}
{"backup":"velero/velero-fullcluster-20200310213901","group":"v1","level":"info","logSource":"pkg/backup/item_backupper.go:418","msg":"Skipping snapshot of persistent volume because volume is being backed up with restic.","name":"pvc-9030563d-317a-11ea-abad-42010a9600a7","namespace":"","persistentVolume":"pvc-9030563d-317a-11ea-abad-42010a9600a7","resource":{"Group":"","Resource":"persistentvolumes"},"time":"2020-03-10T21:39:37Z"}
{"controller":"restic-repository","level":"info","logSource":"pkg/controller/restic_repository_controller.go:156","msg":"Initializing restic repository","name":"twistlock-default-fw7mh","namespace":"velero","time":"2020-03-10T21:39:37Z"}

It calls out that it's skipping that snapshot to use restic. That is on the pod where I added the annotation.

Here is output from a describe of the last backup:

Name:         velero-fullcluster-20200311000001
Namespace:    velero
Labels:       app.kubernetes.io/instance=velero
              app.kubernetes.io/managed-by=Tiller
              app.kubernetes.io/name=velero
              helm.sh/chart=velero-2.9.1
              velero.io/schedule-name=velero-fullcluster
              velero.io/storage-location=default
Annotations:  <none>

Phase:  PartiallyFailed (run `velero backup logs velero-fullcluster-20200311000001` for more information)

Errors:    25
Warnings:  0

Namespaces:
  Included:  *
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  auto

TTL:  72h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2020-03-10 18:00:01 -0600 MDT
Completed:  2020-03-10 19:00:46 -0600 MDT

Expiration:  2020-03-13 18:00:01 -0600 MDT

Persistent Volumes:  0 of 24 snapshots completed successfully (specify --details for more information)

Restic Backups (specify --details for more information):
  New:  1

And here is a sample from the logs on why one of the other 24 volumes failed:

{"backup":"velero/velero-fullcluster-20200311000001","error":"error taking snapshot of volume: rpc error: code = Unknown desc = googleapi: Error 400: Invalid resource usage: 'Invalid value for storage location: us-east4-c'., invalidResourceUsage","group":"v1","level":"error","logSource":"pkg/backup/resource_backupper.go:287","msg":"Error backing up item","name":"redis-data-webapp-redis-qa-slave-1","namespace":"","resource":"persistentvolumeclaims","time":"2020-03-11T01:00:32Z"}

So from this and your comment it looks like we don't need the backup-volumes annotation? And my guess on the storage issue is that it should be us-east4, as a region, not the specific zone?

This is clearly not a helm chart issue then, so happy to close this issue and take it somewhere else to not junk things up here.

[thedevelopnik]

@nrb that was it, didn't need the annotation and needed to set the region broader, thanks for the assist! This issue can be closed.

[nrb]

@thedevelopnik Sorry for the delayed response - that's correct, you only need that label for restic backups. Persistent volume snapshots will be taken without having to annotate or label anything.