CLOUDSTACK-9200: Fixed failed to delete snapshot if snapshot is stuck in Allocated state without any job associated with it #1282

anshul1886 · 2015-12-23T09:47:39Z

https://issues.apache.org/jira/browse/CLOUDSTACK-9200

This issue is hard to reproduce but if occurs then it may lead to account resources cleanup failures.

pdube · 2015-12-23T14:33:12Z

How do you know if there are no jobs associated with it?

anshul1886 · 2015-12-24T04:39:52Z

@pdube Snapshot can be in Allocated state for a split second only in normal scenarios. If it is in Allocated state for more than that time, then it simply means we are either restarting or have to restart the management server. Now if that's the case then it should have been cleaned up by storage cleanup thread during management server start if it has job associated with it. But that is not happening so it is falling in this scenario.

Slair1 · 2016-04-26T16:35:00Z

Thanks! I had the same issue, retention settings for snapshots were failing because it couldn't delete two snapshots that were stuck in "allocated' state.

koushik-das · 2016-04-28T08:24:30Z

@anshul1886 The fix is only for XS snapshots. What about other HVs?

anshul1886 · 2016-04-28T10:05:20Z

@koushik-das All hypervisor uses XenServerSnapshotStrategy so it should be applicable on all hypervisors.

swill · 2016-04-28T15:15:09Z

@anshul1886 I see in some other places in that file when snapshotDao.remove(snapshotId); is called there is a return true right after it. Can you help me understand why we would or would not do that in this case? Thanks...

rohityadavcloud · 2016-05-02T12:08:18Z

@anshul1886 rebase against latest master, thanks

Do you think a cleanup action of some sort be performed, as this will only delete the snapshot in database

bvbharatk · 2016-05-27T20:19:53Z

ACS CI BVT Run

Sumarry:
Build Number 67
Hypervisor xenserver
NetworkType Advanced
Passed=68
Failed=4
Skipped=3

Link to logs Folder (search by build_no): https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0

Failed tests:

test_vpc_vpn.py
- ContextSuite context=TestRVPCSite2SiteVpn>:setup Failed
- ContextSuite context=TestVpcRemoteAccessVpn>:setup Failing since 2 runs
- ContextSuite context=TestVpcSite2SiteVpn>:setup Failing since 2 runs
test_vm_life_cycle.py
- test_10_attachAndDetach_iso Failed

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_static_role_account_acls
test_deploy_vgpu_enabled_vm

Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_routers.py
test_reset_vm_on_reboot.py
test_snapshots.py
test_deploy_vms_with_varied_deploymentplanners.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_volumes.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_disk_offerings.py

Slair1 · 2017-02-09T16:45:35Z

@anshul1886 any idea why this hasn't been merged yet? are there concerns/issues?

karuturi · 2017-02-09T16:51:12Z

@Slair1 Its missing required reviews and tests.(We have a huge backlog. any help in code review/test run is welcome). please refer to https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up

… in Allocated state without any job associated with it

anshul1886 · 2017-03-09T09:11:04Z

@swill Added the return true statement. @rhtyd Snapshot in Allocated state means nothing is done for that snapshot so only deletion from DB is needed.

cloudmonger · 2017-03-09T16:23:24Z

ACS CI BVT Run

Sumarry:
Build Number 446
Hypervisor xenserver
NetworkType Advanced
Passed=105
Failed=0
Skipped=7

Link to logs Folder (search by build_no): https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0

Failed tests:

Skipped tests:
test_01_test_vm_volume_snapshot
test_vm_nic_adapter_vmxnet3
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm

Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_deploy_vm_iso.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_volumes.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_routers_network_ops.py
test_disk_offerings.py

karuturi · 2017-04-06T06:38:39Z

@swill @rhtyd can you review?

rohityadavcloud · 2017-04-06T07:17:41Z

...e/storage/snapshot/src/org/apache/cloudstack/storage/snapshot/XenserverSnapshotStrategy.java

@@ -245,6 +245,12 @@ public boolean deleteSnapshot(Long snapshotId) {
            return true;
        }

+        if(snapshotVO.getState() == Snapshot.State.Allocated) {


@anshul1886 the code to check and remove snapshot when they are stuck in Allocated state is already in line 223 above, in the same method. Why this redundancy, is this a synchronization issue?

See https://github.com/apache/cloudstack/pull/1282/files#diff-2b08c10a0eea4bef15bece6ff38ed1f9R223

@rhtyd , you're right, this change has since already been merged by PR #977.

@rhtyd Missed that change. Closing PR now.

rohityadavcloud · 2017-04-06T07:18:48Z

@anshul1886 I've left a comment, the code to check and remove snapshot on allocated state is already in that method?

CLOUDSTACK-9200: Fixed failed to delete snapshot if snapshot is stuck…

5ad183a

… in Allocated state without any job associated with it

anshul1886 force-pushed the CLOUDSTACK-9200 branch from 1fb9b64 to 5ad183a Compare March 9, 2017 09:07

rohityadavcloud reviewed Apr 6, 2017

View reviewed changes

anshul1886 closed this Apr 14, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLOUDSTACK-9200: Fixed failed to delete snapshot if snapshot is stuck in Allocated state without any job associated with it #1282

CLOUDSTACK-9200: Fixed failed to delete snapshot if snapshot is stuck in Allocated state without any job associated with it #1282

anshul1886 commented Dec 23, 2015

pdube commented Dec 23, 2015

anshul1886 commented Dec 24, 2015

Slair1 commented Apr 26, 2016

koushik-das commented Apr 28, 2016

anshul1886 commented Apr 28, 2016

swill commented Apr 28, 2016

rohityadavcloud commented May 2, 2016

bvbharatk commented May 27, 2016

Slair1 commented Feb 9, 2017

karuturi commented Feb 9, 2017

anshul1886 commented Mar 9, 2017

cloudmonger commented Mar 9, 2017

karuturi commented Apr 6, 2017

rohityadavcloud Apr 6, 2017

rohityadavcloud Apr 6, 2017

Slair1 Apr 13, 2017

anshul1886 Apr 14, 2017

rohityadavcloud commented Apr 6, 2017

CLOUDSTACK-9200: Fixed failed to delete snapshot if snapshot is stuck in Allocated state without any job associated with it #1282

CLOUDSTACK-9200: Fixed failed to delete snapshot if snapshot is stuck in Allocated state without any job associated with it #1282

Conversation

anshul1886 commented Dec 23, 2015

pdube commented Dec 23, 2015

anshul1886 commented Dec 24, 2015

Slair1 commented Apr 26, 2016

koushik-das commented Apr 28, 2016

anshul1886 commented Apr 28, 2016

swill commented Apr 28, 2016

rohityadavcloud commented May 2, 2016

bvbharatk commented May 27, 2016

ACS CI BVT Run

Slair1 commented Feb 9, 2017

karuturi commented Feb 9, 2017

anshul1886 commented Mar 9, 2017

cloudmonger commented Mar 9, 2017

ACS CI BVT Run

karuturi commented Apr 6, 2017

rohityadavcloud Apr 6, 2017

Choose a reason for hiding this comment

rohityadavcloud Apr 6, 2017

Choose a reason for hiding this comment

Slair1 Apr 13, 2017

Choose a reason for hiding this comment

anshul1886 Apr 14, 2017

Choose a reason for hiding this comment

rohityadavcloud commented Apr 6, 2017