Volume Snapshot | Secondary storage mount issue| #6199
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
mountNfsSecondaryStorage script
CLOUDSTACK VERSION
4.16.1
CONFIGURATION
Cloudstack 4.16.1, XCP-ng 8.2.1
OS / ENVIRONMENT
Centos7.9, XCP-ng 8.2.1
SUMMARY
Unable to take VM volume snapshot or Create new VM|| Secondary storage mount issue
STEPS TO REPRODUCE
Storage>Volume>Take volume snapshot.
Sometimes create VM's not working.
I am getting the error as attached screenshot and while checking the logs we are getting below error
P.S: I have verified that nfs server is working fine and secondary storage is mounted in xen server. Can someone point out the reason on this.
ACS managment logs
=================================
2022-04-04 16:16:46,934 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-28:ctx-361e0e2c job-101 ctx
-38aba9ea) (logid:26068ccc) Failed to create snapshot due to an internal error creating snapshot for volume 9
04a2153-67ce-4565-a12a-4b59faca0e73
java.lang.RuntimeException: Unexpected exception
Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Could not mount secondary storage 10.31.9.226:/storage/cloudstack/secondary/ on host /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:301)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171)
at com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1278)
... 45 more
2022-04-04 16:16:46,937 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-28:ctx-361e0e2c job-101) (logid:26068ccc) Complete async job-101, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 904a2153-67ce-4565-a12a-4b59faca0e73"}
=================================
/var/log/cloud/cloud.log
=================================
2022-04-04 15:48:51 DEBUG [root] Successfully mounted 10.31.9.226:/storage/cloudstack/secondary/ to /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3
2022-04-04 15:48:51 DEBUG [root] #### CLOUD exit mountNfsSecondaryStorage ####
2022-04-04 15:48:52 DEBUG [root] #### CLOUD enter makeDirectory ####
2022-04-04 15:48:52 DEBUG [root] #### CLOUD exit makeDirectory ####
2022-04-04 15:48:52 DEBUG [root] #### CLOUD enter mountNfsSecondaryStorage ####
2022-04-04 15:48:52 DEBUG [root] Unexpected error while trying to mount 10.31.9.226:/storage/cloudstack/secondary/ to /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3
2022-04-04 16:16:45 DEBUG [root] #### CLOUD enter mountNfsSecondaryStorage ####
2022-04-04 16:16:45 DEBUG [root] Unexpected error while trying to mount 10.31.9.226:/storage/cloudstack/secondary/ to /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3
=================================
xensource.log
=================================
Apr 4 16:16:45 xenserver-acs-1 xapi: [error||547180 ||backtrace] Async.host.call_plugin R:cb5d0a7171d5 failed with exception Server_error(XENAPI_PLUGIN_FAILURE, [ mountNfsSecondaryStorage; SROSError; Error reporting error, unknown key Unexpected error while trying to mount 10.31.9.226:/storage/cloudstack/secondary/ to /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3 ])
Apr 4 16:16:45 xenserver-acs-1 xapi: [error||547180 ||backtrace] Raised Server_error(XENAPI_PLUGIN_FAILURE, [ mountNfsSecondaryStorage; SROSError; Error reporting error, unknown key Unexpected error while trying to mount 10.31.9.226:/storage/cloudstack/secondary/ to /var/cloud_mount/92ac41e2-5210-36ff-8d7b-97407b9fcfd3 ])
Apr 4 16:16:45 xenserver-acs-1 xapi: [error||547180 ||backtrace] 1/1 xapi Raised at file (Thread 547180 has no backtrace table. Was with_backtraces called?, line 0
Apr 4 16:16:45 xenserver-acs-1 xapi: [error||547180 ||backtrace]
=================================
EXPECTED RESULTS
ACTUAL RESULTS
~~
~
Volume Snapshot Failed, VM Creation also getting failed sometimes