Local persistent volume raises warnings during backup (No volume ID returned by volume snapshotter for persistent volume))

[M-Gregoire]

What steps did you take and what happened:

I have a persistent volume defined:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: <size>
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /path/to/folder
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
              - <node name>

and the following associated PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  storageClassName: local-storage
  volumeName: my-pv
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: <size>

I then create my backup with the usual command:
velero backup create my-backup

Once done, looking at the logs, I see this:

time="<time>" level=warning msg="No volume ID returned by volume snapshotter for persistent volume" backup=velero/my-backup logSource="pkg/backup/item_backupper.go:533" name=my-pv namespace= persistentVolume=my-pv resource=persistentvolumes volumeSnapshotLocation=default

Looking at kubectl -n velero get backup my-backup -oyaml I do have a volumeSnapshotLocation properly defined:

<snip>
spec:
  volumeSnapshotLocations:
  - default
<snip>

And kubectl -n velero get volumesnapshotlocations -oyaml does give back a default volumeSnapshotLocation.

This seems to occur only with local persistent storage, not with pvc in a Ceph blockpool.

What did you expect to happen:

I expected this to work without warning as It used to previously.

Environment:

• Velero version (use velero version): v1.11
• Velero AWS plugin to backup to S3 minio: v1.7.0

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[allenxu404]

The warning "No volume ID returned by volume snapshotter for persistent volume" suggests Velero fails to get volume snapshotter to deal with this pv.

VolumeSnapshotter is instantiated based on the provider of volume storage. Seem local persistent volume is not a supported volume type for snapshots. Need to futher investigate it.

[M-Gregoire]

Thank you for taking the time to answer back.

According to the documentation file system backup:

NOTE: hostPath volumes are not supported, but the local volume type is supported.

This setup was also working perfectly fine a few versions ago, so It looks like a regression.

[M-Gregoire]

I'm using the AWS plugin to backup my data to an s3 bucket (Provider for my volumeSnapshotLocation):

However, looking at the source code of it: https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/109ba05302ff24dc61ec1b33121a0a6fff1b3b86/velero-plugin-for-aws/volume_snapshotter.go#L281

Seems like only a CSI or an AWS EBS type of PV would be supported, and not local-storage. is this intended?

Otherwise, happy to create a PR to add support for the local type as It's supposed to be supported according to the doc.

Thanks

[sseago]

The volume snapshotter doesn't interact with s3 at all. The volume snapshotter is designed to create native snapshots of ebs volumes. The AWS plugin is still needed if you're storing your backups in an s3 bucket, but that uses the object store plugin of velero-plugin-for-aws. If you're not using ebs volumes, you don't need to create a VolumeSnapshotLocation. If your data is not in ebs or on a csi volume, then you need to use the filesystem backup type which stores volume data in the bucket using restic or kopia. Pass the --default-volumes-to-fs-backup flag to velero backup create to use restic/kopia for backup instead of attempting to create a native snapshot.

[M-Gregoire]

Thanks you for your answer. I have both CSI and local-storage volumes so I reckon I still need a VolumeSnapshotLocation.

I always thought those warnings were the reasons some of my volumes were not backup-ed but investigating a bit more closely, I'm not quite sure that's the case.

Looking at Podvolumebackups, there are a few failed ones with following status:

Status: 
    Start Timestamp: ...
    Completion Timestamp:  ...        
    Message:               get a podvolumebackup with status "InProgress" during the server starting, mark it as "Failed"
    Phase:                 Failed      

What surprises me is that I deleted Velero and all associated resources, making sure there were no podvolumebackup resources anymore. Reinstalling everything, and running a backup again leads to the same error.

Full error from the logs:

time="..." level=error msg="Error backing up item" backup=velero/<backup_name> error="pod volume backup failed: get a podvolumebackup with status \"InProgress\" during the server starting, mark it as \"Failed\"" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/podvolume/backupper.go:250" error.function="github.com/vmware-tanzu/velero/pkg/podvolume.(*backupper).BackupPodVolumes" logSource="pkg/backup/backup.go:435" name=<pod>

I'm using Velero v1.11

Couldn't find any similar issue. Any idea?

[sseago]

The PVB errors seem to suggest that you had a file system backup in progress when the node agent pod crashed/restarted. If a restart happens during a backup or restore, the operation will not continue where it left off -- it must be restarted. Since the velero backup picked up that message, the velero pod isn't restarting, but something caused your node agent pod to restart in the middle of a pod volume backup. You probably need to look into why this is restarting.

VolumeSnapshotLocation isn't relevant to this error, since that's only used for VolumeSnapshotter plugins that back up PVs via native snapshots. They aren't used by the CSI plugin if you're using that plugin for csi volume backups, and they're not used for restic or kopia PodVolumeBackups.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands.

[M-Gregoire]

Thank you for your extensive answer! I think this issue can be closed.