Velero installation on Openshift 4.6 environment hosted on vSphere

[sabarivijay88]

Hi Team,

We need to to install Velero tool on Openshift 4.6 environment hosted on vSphere environment.

We are not able to configure the backup storage location on vSphere to test this functionality.

Kindly guide us the steps to configure the Velero tool on Openshift environment hosted on vSphere.

Thanks in advance.

[blackpiglet]

Hi, @sabarivijay88

I think we need more information to give useful advice. When creating an issue, please follow the provided template format. That will help us better understand the problem and environment.
For example, what procedures you followed that finally lead you the failure, better to give related log too, and which version of Velero did you use.

These are some useful links to help you install velero and trouble shooting:

• https://github.com/vmware-tanzu/velero-plugin-for-vsphere/blob/main/docs/vanilla.md
• https://velero.io/docs/v1.7/debugging-install/

[sabarivijay88]

Hi Team,

We have installed Openshift v4.7 on vSphere environment,

[root@spapnltispjmp01 ~]# oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.7.24 True False 92d Cluster version is 4.7.24

[root@spapnltispjmp01 ~]# oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.7.24 True False False 2d20h
baremetal 4.7.24 True False False 171d
cloud-credential 4.7.24 True False False 265d
cluster-autoscaler 4.7.24 True False False 265d
config-operator 4.7.24 True False False 265d
console 4.7.24 True False False 55d
csi-snapshot-controller 4.7.24 True False False 56d
dns 4.7.24 True False False 57d
etcd 4.7.24 True False False 65d
image-registry 4.7.24 True False False 33d
ingress 4.7.24 True False False 33d
insights 4.7.24 True False False 265d
kube-apiserver 4.7.24 True False False 265d
kube-controller-manager 4.7.24 True False False 265d
kube-scheduler 4.7.24 True False False 265d
kube-storage-version-migrator 4.7.24 True False False 4d12h
machine-api 4.7.24 True False False 265d
machine-approver 4.7.24 True False False 265d
machine-config 4.7.24 True False False 3d13h
marketplace 4.7.24 True False False 40d
monitoring 4.7.24 True False False 3d13h
network 4.7.24 True False False 265d
node-tuning 4.7.24 True False False 92d
openshift-apiserver 4.7.24 True False False 2d20h
openshift-controller-manager 4.7.24 True False False 57d
openshift-samples 4.7.24 True False False 92d
operator-lifecycle-manager 4.7.24 True False False 265d
operator-lifecycle-manager-catalog 4.7.24 True False False 265d
operator-lifecycle-manager-packageserver 4.7.24 True False False 3d13h
service-ca 4.7.24 True False False 265d
storage 4.7.24 True False False 55d

We have installed Velero using CLI to perform backup & restore on our cluster,

[root@spapnltispjmp01 ~]# oc get projects |grep velero
velero Active

[root@spapnltispjmp01 ~]# velero version
Client:
Version: v1.6.0
Git commit: 5bd70fd
Server:
Version: v1.6.0

[root@spapnltispjmp01 ~]# oc get pods -n velero
NAME READY STATUS RESTARTS AGE
velero-84ffdd8bfd-jnjlh 1/1 Running 0 21h

Tested by creating sample backup:

[root@spapnltispjmp01 ~]# velero get backups
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
testbkp Failed 0 0 2021-11-22 12:56:20 +0100 CET 29d default

[root@spapnltispjmp01 ~]# velero describe backup testbkp
Name: testbkp
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/source-cluster-k8s-gitversion=v1.20.0+558d959
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=20

Phase: Failed (run velero backup logs testbkp for more information)

Errors: 0
Warnings: 0

Namespaces:
Included: svijay
Excluded:

Resources:
Included: *
Excluded:
Cluster-scoped: auto

Label selector:

Storage Location: default

Velero-Native Snapshot PVs: auto

TTL: 720h0m0s

Hooks:

Backup Format Version: 1.1.0

Started: 2021-11-22 12:56:20 +0100 CET
Completed: <n/a>

Expiration: 2021-12-22 12:56:20 +0100 CET

Velero-Native Snapshots:

Please find the Velero pod logs below,

time="2021-11-23T05:14:44Z" level=error msg="Current backup storage locations available/unavailable/unknown: 0/0/1)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:166"
time="2021-11-23T05:15:10Z" level=error msg="Error getting backup store for this location" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'Velero_OnpremtestBackups' could not be found."" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/velero-plugin-for-microsoft-azure/object_store.go:205" error.function=main.getStorageAccountKey logSource="pkg/controller/backup_sync_controller.go:175"
time="2021-11-23T05:15:41Z" level=error msg="Error getting backup store for this location" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'Velero_OnpremtestBackups' could not be found."" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/velero-plugin-for-microsoft-azure/object_store.go:205" error.function=main.getStorageAccountKey logSource="pkg/controller/backup_sync_controller.go:175"
time="2021-11-23T05:16:11Z" level=error msg="Error getting backup store for this location" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'Velero_OnpremtestBackups' could not be found."" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/velero-plugin-for-microsoft-azure/object_store.go:205" error.function=main.getStorageAccountKey logSource="pkg/controller/backup_sync_controller.go:175"
time="2021-11-23T05:16:42Z" level=error msg="Error getting backup store for this location" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'Velero_OnpremtestBackups' could not be found."" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/velero-plugin-for-microsoft-azure/object_store.go:205" error.function=main.getStorageAccountKey logSource="pkg/controller/backup_sync_controller.go:175"
time="2021-11-23T05:17:12Z" level=error msg="Error getting backup store for this location" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'Velero_OnpremtestBackups' could not be found."" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/velero-plugin-for-microsoft-azure/object_store.go:205" error.function=main.getStorageAccountKey logSource="pkg/controller/backup_sync_controller.go:175"
time="2021-11-23T05:17:43Z" level=error msg="Error getting backup store for this location" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'Velero_OnpremtestBackups' could not be found."" error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/velero-plugin-for-microsoft-azure/object_store.go:205" error.function=main.getStorageAccountKey logSource="pkg/controller/backup_sync_controller.go:175"

We are not sure about the procedure to config backup & storage location for Onprem(vSphere) environment.

Kindly guide me to complete the setup.

Thanks in advance

[blackpiglet]

In the log, it seems using Azure plugin to retrieve BackupStorageLocation information.

Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'Velero_OnpremtestBackups' could not be found.

But, in the above description, the OpenShift is installed on vSphere environment.
I think there is some issue with the velero installment command. Maybe using wrong plugin.
Could you give the detail velero installation command line?

[sabarivijay88]

Hi,

I tried to configure azure resource group as backup location for Onprem (vSphere) platform and I am not seeing any documentation about backup storage location for On-Prem env.

Could you please help me with sample configuration yaml and documentation for velero installation on vSphere which utilize local disk as backup storage location instead of azure resource group or AWS S3 bucket.

Kindly guide me to complete the velero installation on On-Prem environment.

[blackpiglet]

I see.
This should be a velero.io document issue. There is supposed to be a on-premises link in the site, but the link is not shown. Please check it here: velero on-premises
If you are using on-Premise, I think MinIO and Restic can be used to backup cluster.

[sabarivijay88]

Thanks for your update,

I read the documentation which you referred on your update.

Found this below things from the documentation,

===
Currently, only AWS plugin is supported and compatible with vSphere plugin. Please refer to velero-plugin-for-aws for more details about using AWS S3 as the object store for backups. S3-compatible object stores, e.g, MinIO, are also supported via AWS plugin. Please refer to install with MinIO.

So we need AWS S3 bucket to complete the On-Prem setup and I found this on Velero Plugin for vSphere in Vanilla Kubernetes Cluster documentation.

Do we have any documentation for Openshift On-Prem cluster and please confirm whether any other options instead of AWS S3 bucket like local disk for backup.

[blackpiglet]

To my knowledge, there is no document specific to OpenShift on Velero.
If the environment prefers to use non-public-cloud storage location, MinIO is a good choice.
These are the documents will help you setup the MinIO:

• MinIO with Velero
• MinIO setup example

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[blackpiglet]

No response for a long time.
Close for now. If there is any update in future, feel free to reopen.