Skip to content

Fix infrastructure leak on exception while attaching/detaching volumes in VMware #10860

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Aug 4, 2025
Merged

Fix infrastructure leak on exception while attaching/detaching volumes in VMware #10860

merged 2 commits into from
Aug 4, 2025

Conversation

erikbocks
Copy link
Contributor

Description

In VMware environments, when a VM resides on a host in the Disconnected state, and an attach/detach volume operation is initiated, an exception containing infrastructure data is thrown. This PR addresses the issue by handling the AgentUnavailableException separately. The exception will still appear in the application logs, allowing operators to troubleshoot effectively.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI
  • test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

I made the following tests, in my local lab:

  1. Created a new VM and attached a volume to it.
  2. Shutdown my VMware host.
  3. Tried to attach a new volume, and the exception containing the infrastructure data was thrown.
  4. Tried to detach the previously attached volume, and the same exception was thrown.
  5. Built and installed CloudStack's packages with my fix.
  6. Repeated the same processes, and validated that the new error message contained no infrastructure data.

DaanHoogland
DaanHoogland approved these changes May 13, 2025
View reviewed changes
Copy link
Contributor

@DaanHoogland DaanHoogland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm

@codecov Codecov
Copy link

codecov bot commented May 13, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.57%. Comparing base (b57994e) to head (9443d2e).
⚠️ Report is 134 commits behind head on main.

Files with missing lines Patch % Lines
...n/java/com/cloud/storage/VolumeApiServiceImpl.java 0.00% 6 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               main   #10860      +/-   ##
============================================
- Coverage     16.57%   16.57%   -0.01%     
+ Complexity    14059    14056       -3     
============================================
  Files          5772     5772              
  Lines        512938   512944       +6     
  Branches      62304    62305       +1     
============================================
- Hits          85026    85020       -6     
- Misses       418431   418442      +11     
- Partials       9481     9482       +1
Flag Coverage Δ
uitests 3.89% <ø> (ø)
unittests 17.47% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sureshanaparti sureshanaparti added this to the 4.21.0 milestone Jun 5, 2025
@erikbocks
Copy link
Contributor Author

Thank you for the review @sureshanaparti.

@DaanHoogland
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13675

@DaanHoogland
Copy link
Contributor

@blueorangutan test

@blueorangutan
Copy link

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian test result (tid-13500)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 89073 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10860-t13500-kvm-ol8.zip
Smoke tests completed. 130 look OK, 11 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File
test_nic_secondaryip_add_remove Error 1518.41 test_multipleips_per_nic.py
ContextSuite context=TestNestedVirtualization>:setup Error 0.00 test_nested_virtualization.py
ContextSuite context=TestNetworkACL>:setup Error 0.00 test_network_acl.py
ContextSuite context=TestIpv6Network>:setup Error 0.00 test_network_ipv6.py
test_delete_account Error 1517.39 test_network.py
test_delete_network_while_vm_on_it Error 1.26 test_network.py
test_deploy_vm_l2network Error 1.20 test_network.py
test_l2network_restart Error 2.35 test_network.py
ContextSuite context=TestPortForwarding>:setup Error 3.59 test_network.py
ContextSuite context=TestPublicIP>:setup Error 12.44 test_network.py
test_reboot_router Failure 0.09 test_network.py
test_releaseIP Error 6.53 test_network.py
test_releaseIP_using_IP Error 6.02 test_network.py
ContextSuite context=TestRouterRules>:setup Error 6.11 test_network.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup Error 1521.96 test_network.py
ContextSuite context=TestPrivateGwACL>:setup Error 0.00 test_privategw_acl.py
ContextSuite context=TestAdapterTypeForNic>:setup Error 0.00 test_nic_adapter_type.py
ContextSuite context=TestNonStrictAffinityGroups>:setup Error 0.00 test_nonstrict_affinity_group.py
ContextSuite context=TestIsolatedNetworksPasswdServer>:setup Error 0.00 test_password_server.py
ContextSuite context=TestPortForwardingRules>:setup Error 0.00 test_portforwardingrules.py
ContextSuite context=TestProjectSuspendActivate>:setup Error 1529.70 test_projects.py

@DaanHoogland
Copy link
Contributor

@blueorangutan test ol8 vmware-80u3

@blueorangutan
Copy link

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-80u3) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian Build Failed (tid-13517)

@DaanHoogland
Copy link
Contributor

@blueorangutan test ol8 vmware-70u3

@blueorangutan
Copy link

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

@blueorangutan
Copy link

[SF] Trillian Build Failed (tid-13521)

@github-actions GitHub Actions
Copy link

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@erik-bock-silva erik-bock-silva force-pushed the fix-infra-leak-on-logs branch from b4495d9 to 9443d2e Compare July 16, 2025 14:10
@rosi-shapeblue rosi-shapeblue self-assigned this Aug 1, 2025
rosi-shapeblue
rosi-shapeblue approved these changes Aug 4, 2025
View reviewed changes
Copy link
Collaborator

@rosi-shapeblue rosi-shapeblue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

The fix successfully:

  • Catches AgentUnavailableException during volume detach operations
  • Prevents infrastructure data leakage in error messages
  • Provides clean user-facing errors with appropriate guidance
  • Maintains server-side logging for administrator troubleshooting
  • Doesn't break normal operations (attach worked fine)

API response on detach volume with disconnected host

(localcloud) 🐱 > detach volume id=b861fb24-ea21-4a5b-b460-1cb4447352fc
{
  "account": "admin",
  "accountid": "0337a84a-6ec3-11f0-b236-1e003c00027b",
  "cmd": "org.apache.cloudstack.api.command.admin.volume.DetachVolumeCmdByAdmin",
  "completed": "2025-08-04T07:51:57+0000",
  "created": "2025-08-04T07:51:55+0000",
  "domainid": "bdb72b93-6ec2-11f0-b236-1e003c00027b",
  "domainpath": "ROOT",
  "jobid": "d033150b-9e59-4a06-97ac-084b07c84d55",
  "jobinstanceid": "b861fb24-ea21-4a5b-b460-1cb4447352fc",
  "jobinstancetype": "Volume",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "Failed to detach volume test-data-volume from VM VM-c36f5f65-b56e-4967-9cd1-0102bc393d66. Please contact your system administrator."
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "03386e3e-6ec3-11f0-b236-1e003c00027b"
}
🙈 Error: async API failed for job d033150b-9e59-4a06-97ac-084b07c84d55

Logs on detach volume with disconnected host

[root@ref-trl-9139-v-Mol8-rositsa-kyuchukova-mgmt1 ~]# tail -f /var/log/cloudstack/management/management-server.log | grep -E "AgentUnavailable|detach|b861fb24"
2025-08-04 07:51:55,653 DEBUG [c.c.a.ApiServlet] (qtp253011924-20:[ctx-affcd668]) (logid:ba325ae6) ===START===  10.0.33.232 -- GET  apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=detachVolume&id=b861fb24-ea21-4a5b-b460-1cb4447352fc&response=json&signature=uiS0dfCi7VDfoyWtp0UCjp2OY%2B0%3D
2025-08-04 07:51:55,686 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp253011924-20:[ctx-affcd668, ctx-61a5b442, ctx-1887c75d]) (logid:ba325ae6) submit async job-52, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.volume.DetachVolumeCmdByAdmin","cmdInfo":"{\"apiKey\":\"LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q\",\"signature\":\"uiS0dfCi7VDfoyWtp0UCjp2OY+0\\u003d\",\"response\":\"json\",\"ctxUserId\":\"2\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"154\",\"id\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"ctxDetails\":\"{\\\"interface com.cloud.storage.Volume\\\":\\\"b861fb24-ea21-4a5b-b460-1cb4447352fc\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"cmdEventType\":\"VOLUME.DETACH\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":52,"initMsid":32986355466875,"instanceId":4,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"d033150b-9e59-4a06-97ac-084b07c84d55"}
2025-08-04 07:51:55,689 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-36:[ctx-7141649a, job-52]) (logid:d033150b) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.volume.DetachVolumeCmdByAdmin","cmdInfo":"{\"apiKey\":\"LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q\",\"signature\":\"uiS0dfCi7VDfoyWtp0UCjp2OY+0\\u003d\",\"response\":\"json\",\"ctxUserId\":\"2\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"154\",\"id\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"ctxDetails\":\"{\\\"interface com.cloud.storage.Volume\\\":\\\"b861fb24-ea21-4a5b-b460-1cb4447352fc\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"cmdEventType\":\"VOLUME.DETACH\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":52,"initMsid":32986355466875,"instanceId":4,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"d033150b-9e59-4a06-97ac-084b07c84d55"}
2025-08-04 07:51:55,691 DEBUG [c.c.a.ApiServlet] (qtp253011924-20:[ctx-affcd668, ctx-61a5b442, ctx-1887c75d]) (logid:ba325ae6) ===END===  10.0.33.232 -- GET  apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=detachVolume&id=b861fb24-ea21-4a5b-b460-1cb4447352fc&response=json&signature=uiS0dfCi7VDfoyWtp0UCjp2OY%2B0%3D
2025-08-04 07:51:55,708 INFO  [c.c.s.VolumeApiServiceImpl] (API-Job-Executor-36:[ctx-7141649a, job-52, ctx-4fe31cde]) (logid:d033150b) Trying to attach volume {"id":4,"name":"test-data-volume","uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc"} to VM instance {"id":3,"uuid":"c36f5f65-b56e-4967-9cd1-0102bc393d66"}, update async job-52 progress status
2025-08-04 07:51:57,101 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-10:[ctx-d6e5e97d, job-52/job-53, ctx-009225ff]) (logid:d033150b) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume test-data-volume from VM VM-c36f5f65-b56e-4967-9cd1-0102bc393d66. Please contact your system administrator.
2025-08-04 07:51:57,101 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-10:[ctx-d6e5e97d, job-52/job-53, ctx-009225ff]) (logid:d033150b) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume test-data-volume from VM VM-c36f5f65-b56e-4967-9cd1-0102bc393d66. Please contact your system administrator.
2025-08-04 07:51:57,101 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-10:[ctx-d6e5e97d, job-52/job-53]) (logid:d033150b) Unable to complete AsyncJob {"accountId":2,"cmd":"com.cloud.vm.VmWorkDetachVolume","cmdInfo":"rO0ABXNyAB9jb20uY2xvdWQudm0uVm1Xb3JrRGV0YWNoVm9sdW1lhvReA-s81AMCAAFMAAh2b2x1bWVJZHQAEExqYXZhL2xhbmcvTG9uZzt4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1ldAASTGphdmEvbGFuZy9TdHJpbmc7eHAAAAAAAAAAAgAAAAAAAAACAAAAAAAAAAN0ABRWb2x1bWVBcGlTZXJ2aWNlSW1wbHNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAABA","cmdVersion":0,"completeMsid":null,"created":"Mon Aug 04 07:51:55 UTC 2025","id":53,"initMsid":32986355466875,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"0e14bf02-1fbe-4f8c-a166-eed6ce89e356"}, job origin: 52 com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume test-data-volume from VM VM-c36f5f65-b56e-4967-9cd1-0102bc393d66. Please contact your system administrator.
2025-08-04 07:51:57,130 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-36:[ctx-7141649a, job-52, ctx-4fe31cde]) (logid:d033150b) Skipping delivering event Event {"description":"{\"details\":\"detaching volume: b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"event\":\"VOLUME.DETACH\",\"status\":\"Completed\"}","eventId":null,"eventType":"VOLUME.DETACH","eventUuid":null,"resourceType":"Volume","resourceUUID":null} to any webhook as account ID is missing
2025-08-04 07:51:57,138 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-36:[ctx-7141649a, job-52]) (logid:d033150b) Unexpected exception while executing org.apache.cloudstack.api.command.admin.volume.DetachVolumeCmdByAdmin com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume test-data-volume from VM VM-c36f5f65-b56e-4967-9cd1-0102bc393d66. Please contact your system administrator.
2025-08-04 07:51:57,138 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-36:[ctx-7141649a, job-52]) (logid:d033150b) Complete async job-52, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to detach volume test-data-volume from VM VM-c36f5f65-b56e-4967-9cd1-0102bc393d66. Please contact your system administrator."}

Logs on attach volume with disconnected host

2025-08-04 07:52:32,968 DEBUG [c.c.a.ApiServlet] (qtp253011924-22:[ctx-2d8a3210]) (logid:348c805d) ===START===  10.0.33.232 -- GET  apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=attachVolume&id=b861fb24-ea21-4a5b-b460-1cb4447352fc&response=json&virtualmachineid=c36f5f65-b56e-4967-9cd1-0102bc393d66&signature=a3YDwnoVMB056rG3xqE14H4%2BokA%3D
2025-08-04 07:52:32,998 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp253011924-22:[ctx-2d8a3210, ctx-ce697025, ctx-b6015f28]) (logid:348c805d) submit async job-54, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.volume.AttachVolumeCmdByAdmin","cmdInfo":"{\"apiKey\":\"LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q\",\"virtualmachineid\":\"c36f5f65-b56e-4967-9cd1-0102bc393d66\",\"signature\":\"a3YDwnoVMB056rG3xqE14H4+okA\\u003d\",\"response\":\"json\",\"ctxUserId\":\"2\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"157\",\"id\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"ctxDetails\":\"{\\\"interface com.cloud.storage.Volume\\\":\\\"b861fb24-ea21-4a5b-b460-1cb4447352fc\\\",\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"c36f5f65-b56e-4967-9cd1-0102bc393d66\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"cmdEventType\":\"VOLUME.ATTACH\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":54,"initMsid":32986355466875,"instanceId":4,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"b21006ee-ceee-4ca0-a289-ebe89fd11b5f"}
2025-08-04 07:52:32,998 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-37:[ctx-afc67bd9, job-54]) (logid:b21006ee) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.volume.AttachVolumeCmdByAdmin","cmdInfo":"{\"apiKey\":\"LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q\",\"virtualmachineid\":\"c36f5f65-b56e-4967-9cd1-0102bc393d66\",\"signature\":\"a3YDwnoVMB056rG3xqE14H4+okA\\u003d\",\"response\":\"json\",\"ctxUserId\":\"2\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"157\",\"id\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"ctxDetails\":\"{\\\"interface com.cloud.storage.Volume\\\":\\\"b861fb24-ea21-4a5b-b460-1cb4447352fc\\\",\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"c36f5f65-b56e-4967-9cd1-0102bc393d66\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"cmdEventType\":\"VOLUME.ATTACH\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":54,"initMsid":32986355466875,"instanceId":4,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"b21006ee-ceee-4ca0-a289-ebe89fd11b5f"}
2025-08-04 07:52:32,999 DEBUG [c.c.a.ApiServlet] (qtp253011924-22:[ctx-2d8a3210, ctx-ce697025, ctx-b6015f28]) (logid:348c805d) ===END===  10.0.33.232 -- GET  apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=attachVolume&id=b861fb24-ea21-4a5b-b460-1cb4447352fc&response=json&virtualmachineid=c36f5f65-b56e-4967-9cd1-0102bc393d66&signature=a3YDwnoVMB056rG3xqE14H4%2BokA%3D
2025-08-04 07:52:33,015 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-37:[ctx-afc67bd9, job-54, ctx-71f67bbc]) (logid:b21006ee) Skipping delivering event Event {"description":"{\"details\":\"Volume Id: b861fb24-ea21-4a5b-b460-1cb4447352fc VmId: c36f5f65-b56e-4967-9cd1-0102bc393d66\",\"event\":\"VOLUME.ATTACH\",\"status\":\"Completed\"}","eventId":null,"eventType":"VOLUME.ATTACH","eventUuid":null,"resourceType":"Volume","resourceUUID":null} to any webhook as account ID is missing

Logs on detach volume with host up

2025-08-04 07:57:26,806 DEBUG [c.c.a.ApiServlet] (qtp253011924-77365:[ctx-36cb9a83]) (logid:fb21b6c4) ===START===  10.0.33.232 -- GET  apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=detachVolume&id=b861fb24-ea21-4a5b-b460-1cb4447352fc&response=json&signature=uiS0dfCi7VDfoyWtp0UCjp2OY%2B0%3D
2025-08-04 07:57:26,840 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp253011924-77365:[ctx-36cb9a83, ctx-3f6ffc07, ctx-7299748e]) (logid:fb21b6c4) submit async job-58, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.volume.DetachVolumeCmdByAdmin","cmdInfo":"{\"apiKey\":\"LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q\",\"signature\":\"uiS0dfCi7VDfoyWtp0UCjp2OY+0\\u003d\",\"response\":\"json\",\"ctxUserId\":\"2\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"163\",\"id\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"ctxDetails\":\"{\\\"interface com.cloud.storage.Volume\\\":\\\"b861fb24-ea21-4a5b-b460-1cb4447352fc\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"cmdEventType\":\"VOLUME.DETACH\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":58,"initMsid":32986355466875,"instanceId":4,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"9fdf554d-210a-4ce2-90e9-280148c103b6"}
2025-08-04 07:57:26,841 DEBUG [c.c.a.ApiServlet] (qtp253011924-77365:[ctx-36cb9a83, ctx-3f6ffc07, ctx-7299748e]) (logid:fb21b6c4) ===END===  10.0.33.232 -- GET  apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=detachVolume&id=b861fb24-ea21-4a5b-b460-1cb4447352fc&response=json&signature=uiS0dfCi7VDfoyWtp0UCjp2OY%2B0%3D
2025-08-04 07:57:26,845 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-41:[ctx-7b119732, job-58]) (logid:9fdf554d) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.volume.DetachVolumeCmdByAdmin","cmdInfo":"{\"apiKey\":\"LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q\",\"signature\":\"uiS0dfCi7VDfoyWtp0UCjp2OY+0\\u003d\",\"response\":\"json\",\"ctxUserId\":\"2\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"163\",\"id\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"ctxDetails\":\"{\\\"interface com.cloud.storage.Volume\\\":\\\"b861fb24-ea21-4a5b-b460-1cb4447352fc\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"cmdEventType\":\"VOLUME.DETACH\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":58,"initMsid":32986355466875,"instanceId":4,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"9fdf554d-210a-4ce2-90e9-280148c103b6"}
2025-08-04 07:57:26,859 INFO  [c.c.s.VolumeApiServiceImpl] (API-Job-Executor-41:[ctx-7b119732, job-58, ctx-0b4db09d]) (logid:9fdf554d) Trying to attach volume {"id":4,"name":"test-data-volume","uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc"} to VM instance {"id":3,"uuid":"c36f5f65-b56e-4967-9cd1-0102bc393d66"}, update async job-58 progress status
2025-08-04 07:57:27,317 DEBUG [c.c.a.t.Request] (Work-Job-Executor-11:[ctx-ae07cfc2, job-58/job-59, ctx-cb9af017]) (logid:9fdf554d) Seq 1-7125257560453545991: Sending  { Cmd , MgmtId: 32986355466875, via: 1(10.0.32.154), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DettachCommand":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4f324b33-7d7d-3a06-9332-d0ebc71774d1","name":"ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1/?ROLE=Primary&STOREUUID=4f324b33-7d7d-3a06-9332-d0ebc71774d1","isManaged":"false"}},"name":"test-data-volume","size":"(5.00 GB) 5368709120","path":"1f8244f6911b4a9d8aeb472483ccc8b2","volumeId":"4","vmName":"i-2-3-VM","accountId":"2","chainInfo":"{"diskDeviceBusName":"scsi0:0","diskChain":["[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk"]}","format":"OVA","provisioningType":"THIN","poolId":"1","id":"4","deviceId":"1","hypervisorType":"VMware","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"false"}},"diskSeq":"1","path":"1f8244f6911b4a9d8aeb472483ccc8b2","type":"DATADISK","details":{}},"vmName":"i-2-3-VM","_managed":"false","_storageHost":"10.0.32.4","_storagePort":"2049","forced":"false","waitDetachDevice":"10000","wait":"0","bypassHostMaintenance":"false"}}] }
2025-08-04 07:57:27,318 DEBUG [c.c.a.t.Request] (Work-Job-Executor-11:[ctx-ae07cfc2, job-58/job-59, ctx-cb9af017]) (logid:9fdf554d) Seq 1-7125257560453545991: Executing:  { Cmd , MgmtId: 32986355466875, via: 1(10.0.32.154), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DettachCommand":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4f324b33-7d7d-3a06-9332-d0ebc71774d1","name":"ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1/?ROLE=Primary&STOREUUID=4f324b33-7d7d-3a06-9332-d0ebc71774d1","isManaged":"false"}},"name":"test-data-volume","size":"(5.00 GB) 5368709120","path":"1f8244f6911b4a9d8aeb472483ccc8b2","volumeId":"4","vmName":"i-2-3-VM","accountId":"2","chainInfo":"{"diskDeviceBusName":"scsi0:0","diskChain":["[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk"]}","format":"OVA","provisioningType":"THIN","poolId":"1","id":"4","deviceId":"1","hypervisorType":"VMware","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"false"}},"diskSeq":"1","path":"1f8244f6911b4a9d8aeb472483ccc8b2","type":"DATADISK","details":{}},"vmName":"i-2-3-VM","_managed":"false","_storageHost":"10.0.32.4","_storagePort":"2049","forced":"false","waitDetachDevice":"10000","wait":"0","bypassHostMaintenance":"false"}}] }
2025-08-04 07:57:27,318 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-136:[ctx-4af91280]) (logid:9fdf554d) Executing resource command DettachCommand: [{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4f324b33-7d7d-3a06-9332-d0ebc71774d1","name":"ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","id":1,"poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","port":2049,"url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1/?ROLE=Primary&STOREUUID=4f324b33-7d7d-3a06-9332-d0ebc71774d1","isManaged":false}},"name":"test-data-volume","size":5368709120,"path":"1f8244f6911b4a9d8aeb472483ccc8b2","volumeId":4,"vmName":"i-2-3-VM","accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk\"]}","format":"OVA","provisioningType":"THIN","poolId":1,"id":4,"deviceId":1,"hypervisorType":"VMware","directDownload":false,"deployAsIs":false,"checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":false}},"diskSeq":1,"path":"1f8244f6911b4a9d8aeb472483ccc8b2","type":"DATADISK","details":{}},"vmName":"i-2-3-VM","_managed":false,"_storageHost":"10.0.32.4","_storagePort":2049,"forced":false,"waitDetachDevice":10000,"wait":0,"bypassHostMaintenance":false}].
2025-08-04 07:57:27,319 DEBUG [c.c.s.r.VmwareStorageSubsystemCommandHandler] (DirectAgent-136:[ctx-4af91280, 10.0.32.154, job-58/job-59, cmd: DettachCommand]) (logid:9fdf554d) Executing command DettachCommand: [{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4f324b33-7d7d-3a06-9332-d0ebc71774d1","name":"ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","id":1,"poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","port":2049,"url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1/?ROLE=Primary&STOREUUID=4f324b33-7d7d-3a06-9332-d0ebc71774d1","isManaged":false}},"name":"test-data-volume","size":5368709120,"path":"1f8244f6911b4a9d8aeb472483ccc8b2","volumeId":4,"vmName":"i-2-3-VM","accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk\"]}","format":"OVA","provisioningType":"THIN","poolId":1,"id":4,"deviceId":1,"hypervisorType":"VMware","directDownload":false,"deployAsIs":false,"checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":false}},"diskSeq":1,"path":"1f8244f6911b4a9d8aeb472483ccc8b2","type":"DATADISK","details":{}},"vmName":"i-2-3-VM","_managed":false,"_storageHost":"10.0.32.4","_storagePort":2049,"forced":false,"waitDetachDevice":10000,"wait":0,"bypassHostMaintenance":false}].
2025-08-04 07:57:31,195 DEBUG [c.c.a.t.Request] (DirectAgent-136:[ctx-4af91280]) (logid:9fdf554d) Seq 1-7125257560453545991: Processing:  { Ans: , MgmtId: 32986355466875, via: 1(10.0.32.154), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.AttachAnswer":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4f324b33-7d7d-3a06-9332-d0ebc71774d1","name":"ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1/?ROLE=Primary&STOREUUID=4f324b33-7d7d-3a06-9332-d0ebc71774d1","isManaged":"false"}},"name":"test-data-volume","size":"(5.00 GB) 5368709120","path":"1f8244f6911b4a9d8aeb472483ccc8b2","volumeId":"4","vmName":"i-2-3-VM","accountId":"2","chainInfo":"{"diskDeviceBusName":"scsi0:0","diskChain":["[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk"]}","format":"OVA","provisioningType":"THIN","poolId":"1","id":"4","deviceId":"1","hypervisorType":"VMware","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"false"}},"diskSeq":"1","path":"[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk","type":"DATADISK","details":{}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-08-04 07:57:31,285 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-41:[ctx-7b119732, job-58, ctx-0b4db09d]) (logid:9fdf554d) Skipping delivering event Event {"description":"{\"details\":\"detaching volume: b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"event\":\"VOLUME.DETACH\",\"status\":\"Completed\"}","eventId":null,"eventType":"VOLUME.DETACH","eventUuid":null,"resourceType":"Volume","resourceUUID":null} to any webhook as account ID is missing
2025-08-04 07:57:31,299 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-41:[ctx-7b119732, job-58, ctx-0b4db09d]) (logid:9fdf554d) Complete async job-58, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.VolumeResponse/volume/{"id":"b861fb24-ea21-4a5b-b460-1cb4447352fc","name":"test-data-volume","zoneid":"dca8a8fc-c751-4be2-b2b6-2807260294c6","zonename":"ref-trl-9139-v-Mol8-rositsa-kyuchukova","type":"DATADISK","provisioningtype":"thin","size":"(5.00 GB) 5368709120","created":"2025-08-04T07:32:59+0000","state":"Ready","account":"admin","domainid":"bdb72b93-6ec2-11f0-b236-1e003c00027b","domain":"ROOT","domainpath":"/","storagetype":"shared","hypervisor":"VMware","diskofferingid":"7ad37a9c-f0dc-4a26-822d-8348609f6d24","diskofferingname":"Small","diskofferingdisplaytext":"Small Disk, 5 GB","storage":"ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","destroyed":"false","isextractable":"true","displayvolume":"true","path":"1f8244f6911b4a9d8aeb472483ccc8b2","storageid":"4f324b33-7d7d-3a06-9332-d0ebc71774d1","chaininfo":"{"diskDeviceBusName":"scsi0:0","diskChain":["[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk"]}","quiescevm":"false","supportsstoragesnapshot":"false","deleteprotection":"false","physicalsize":"0","virtualsize":"5368709120","utilization":"0.0%","clusterid":"873dbf23-6a29-41a4-9771-0c96761a0693","clustername":"p1-c1","podid":"ded01417-72b6-4e6a-8c2f-03fdec041dbb","podname":"Pod1","externaluuid":"6000C295-0feb-3e21-e1b0-6c2f68eea7fe","tags":[],"hasannotations":"false","jobid":"9fdf554d-210a-4ce2-90e9-280148c103b6","jobstatus":"0"}

Logs on attach volume with host up

2025-08-04 07:58:49,001 DEBUG [c.c.a.ApiServlet] (qtp253011924-20:[ctx-219c12ba]) (logid:4cb913ca) ===START===  10.0.33.232 -- GET  apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=attachVolume&id=b861fb24-ea21-4a5b-b460-1cb4447352fc&response=json&virtualmachineid=c36f5f65-b56e-4967-9cd1-0102bc393d66&signature=a3YDwnoVMB056rG3xqE14H4%2BokA%3D
2025-08-04 07:58:49,032 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp253011924-20:[ctx-219c12ba, ctx-a077c010, ctx-75ea1b8c]) (logid:4cb913ca) submit async job-60, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.volume.AttachVolumeCmdByAdmin","cmdInfo":"{\"apiKey\":\"LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q\",\"virtualmachineid\":\"c36f5f65-b56e-4967-9cd1-0102bc393d66\",\"signature\":\"a3YDwnoVMB056rG3xqE14H4+okA\\u003d\",\"response\":\"json\",\"ctxUserId\":\"2\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"166\",\"id\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"ctxDetails\":\"{\\\"interface com.cloud.storage.Volume\\\":\\\"b861fb24-ea21-4a5b-b460-1cb4447352fc\\\",\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"c36f5f65-b56e-4967-9cd1-0102bc393d66\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"cmdEventType\":\"VOLUME.ATTACH\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":60,"initMsid":32986355466875,"instanceId":4,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"73720bf2-cfbf-49f1-b40b-63a497c31829"}
2025-08-04 07:58:49,036 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-42:[ctx-9a06b0e0, job-60]) (logid:73720bf2) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.volume.AttachVolumeCmdByAdmin","cmdInfo":"{\"apiKey\":\"LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q\",\"virtualmachineid\":\"c36f5f65-b56e-4967-9cd1-0102bc393d66\",\"signature\":\"a3YDwnoVMB056rG3xqE14H4+okA\\u003d\",\"response\":\"json\",\"ctxUserId\":\"2\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"166\",\"id\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"ctxDetails\":\"{\\\"interface com.cloud.storage.Volume\\\":\\\"b861fb24-ea21-4a5b-b460-1cb4447352fc\\\",\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"c36f5f65-b56e-4967-9cd1-0102bc393d66\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"cmdEventType\":\"VOLUME.ATTACH\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":60,"initMsid":32986355466875,"instanceId":4,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"73720bf2-cfbf-49f1-b40b-63a497c31829"}
2025-08-04 07:58:49,037 DEBUG [c.c.a.ApiServlet] (qtp253011924-20:[ctx-219c12ba, ctx-a077c010, ctx-75ea1b8c]) (logid:4cb913ca) ===END===  10.0.33.232 -- GET  apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=attachVolume&id=b861fb24-ea21-4a5b-b460-1cb4447352fc&response=json&virtualmachineid=c36f5f65-b56e-4967-9cd1-0102bc393d66&signature=a3YDwnoVMB056rG3xqE14H4%2BokA%3D
2025-08-04 07:58:49,063 INFO  [c.c.s.VolumeApiServiceImpl] (API-Job-Executor-42:[ctx-9a06b0e0, job-60, ctx-524b0fd3]) (logid:73720bf2) Trying to attach volume [VolumeObject {"dataStore":"StoragePool {\"id\":1,\"name\":\"ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1\",\"poolType\":\"NetworkFilesystem\",\"uuid\":\"4f324b33-7d7d-3a06-9332-d0ebc71774d1\"}","volumeVO":"Volume {\"id\":4,\"instanceId\":null,\"name\":\"test-data-volume\",\"uuid\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"volumeType\":\"DATADISK\"}"}] to VM instance [VM instance {"id":3,"instanceName":"i-2-3-VM","state":"Running","type":"User","uuid":"c36f5f65-b56e-4967-9cd1-0102bc393d66"}], update async job-60 [AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.volume.AttachVolumeCmdByAdmin","cmdInfo":"{\"apiKey\":\"LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q\",\"virtualmachineid\":\"c36f5f65-b56e-4967-9cd1-0102bc393d66\",\"signature\":\"a3YDwnoVMB056rG3xqE14H4+okA\\u003d\",\"response\":\"json\",\"ctxUserId\":\"2\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"166\",\"id\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"ctxDetails\":\"{\\\"interface com.cloud.storage.Volume\\\":\\\"b861fb24-ea21-4a5b-b460-1cb4447352fc\\\",\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"c36f5f65-b56e-4967-9cd1-0102bc393d66\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"b861fb24-ea21-4a5b-b460-1cb4447352fc\",\"cmdEventType\":\"VOLUME.ATTACH\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":60,"initMsid":32986355466875,"instanceId":4,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"73720bf2-cfbf-49f1-b40b-63a497c31829"}] progress status
2025-08-04 07:58:51,115 DEBUG [c.c.s.VolumeApiServiceImpl] (Work-Job-Executor-12:[ctx-0f8e7d8f, job-60/job-61, ctx-b021f9a9]) (logid:73720bf2) Attach volume Volume {"id":4,"instanceId":null,"name":"test-data-volume","uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc","volumeType":"DATADISK"} on VM VM instance {"id":3,"instanceName":"i-2-3-VM","state":"Running","type":"User","uuid":"c36f5f65-b56e-4967-9cd1-0102bc393d66"} has controller info: {dataDiskController=osdefault, rootDiskController=ide}
2025-08-04 07:58:51,118 DEBUG [c.c.a.t.Request] (Work-Job-Executor-12:[ctx-0f8e7d8f, job-60/job-61, ctx-b021f9a9]) (logid:73720bf2) Seq 1-7125257560453545999: Sending  { Cmd , MgmtId: 32986355466875, via: 1(10.0.32.154), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.AttachCommand":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4f324b33-7d7d-3a06-9332-d0ebc71774d1","name":"ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1/?ROLE=Primary&STOREUUID=4f324b33-7d7d-3a06-9332-d0ebc71774d1","isManaged":"false"}},"name":"test-data-volume","size":"(5.00 GB) 5368709120","path":"1f8244f6911b4a9d8aeb472483ccc8b2","volumeId":"4","accountId":"2","chainInfo":"{"diskDeviceBusName":"scsi0:0","diskChain":["[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk"]}","format":"OVA","provisioningType":"THIN","poolId":"1","id":"4","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","hypervisorType":"VMware","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"false"}},"diskSeq":"1","path":"1f8244f6911b4a9d8aeb472483ccc8b2","type":"DATADISK","details":{"storageHost":"10.0.32.4","managed":"false","storagePort":"2049","storage.pool.disk.wait":"60","volumeSize":"(5.00 GB) 5368709120"}},"vmName":"i-2-3-VM","inSeq":"false","forced":"false","controllerInfo":{"dataDiskController":"osdefault","rootDiskController":"ide"},"wait":"0","bypassHostMaintenance":"false"}}] }
2025-08-04 07:58:51,121 DEBUG [c.c.a.t.Request] (Work-Job-Executor-12:[ctx-0f8e7d8f, job-60/job-61, ctx-b021f9a9]) (logid:73720bf2) Seq 1-7125257560453545999: Executing:  { Cmd , MgmtId: 32986355466875, via: 1(10.0.32.154), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.AttachCommand":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4f324b33-7d7d-3a06-9332-d0ebc71774d1","name":"ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1/?ROLE=Primary&STOREUUID=4f324b33-7d7d-3a06-9332-d0ebc71774d1","isManaged":"false"}},"name":"test-data-volume","size":"(5.00 GB) 5368709120","path":"1f8244f6911b4a9d8aeb472483ccc8b2","volumeId":"4","accountId":"2","chainInfo":"{"diskDeviceBusName":"scsi0:0","diskChain":["[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk"]}","format":"OVA","provisioningType":"THIN","poolId":"1","id":"4","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","hypervisorType":"VMware","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"false"}},"diskSeq":"1","path":"1f8244f6911b4a9d8aeb472483ccc8b2","type":"DATADISK","details":{"storageHost":"10.0.32.4","managed":"false","storagePort":"2049","storage.pool.disk.wait":"60","volumeSize":"(5.00 GB) 5368709120"}},"vmName":"i-2-3-VM","inSeq":"false","forced":"false","controllerInfo":{"dataDiskController":"osdefault","rootDiskController":"ide"},"wait":"0","bypassHostMaintenance":"false"}}] }
2025-08-04 07:58:51,121 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-276:[ctx-ab9bf866]) (logid:73720bf2) Executing resource command AttachCommand: [{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4f324b33-7d7d-3a06-9332-d0ebc71774d1","name":"ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","id":1,"poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","port":2049,"url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1/?ROLE=Primary&STOREUUID=4f324b33-7d7d-3a06-9332-d0ebc71774d1","isManaged":false}},"name":"test-data-volume","size":5368709120,"path":"1f8244f6911b4a9d8aeb472483ccc8b2","volumeId":4,"accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk\"]}","format":"OVA","provisioningType":"THIN","poolId":1,"id":4,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDownload":false,"deployAsIs":false,"checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":false}},"diskSeq":1,"path":"1f8244f6911b4a9d8aeb472483ccc8b2","type":"DATADISK","details":{"storageHost":"10.0.32.4","managed":"false","storagePort":"2049","storage.pool.disk.wait":"60","volumeSize":"5368709120"}},"vmName":"i-2-3-VM","inSeq":false,"forced":false,"controllerInfo":{"dataDiskController":"osdefault","rootDiskController":"ide"},"wait":0,"bypassHostMaintenance":false}].
2025-08-04 07:58:51,122 DEBUG [c.c.s.r.VmwareStorageSubsystemCommandHandler] (DirectAgent-276:[ctx-ab9bf866, 10.0.32.154, job-60/job-61, cmd: AttachCommand]) (logid:73720bf2) Executing command AttachCommand: [{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4f324b33-7d7d-3a06-9332-d0ebc71774d1","name":"ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","id":1,"poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","port":2049,"url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1/?ROLE=Primary&STOREUUID=4f324b33-7d7d-3a06-9332-d0ebc71774d1","isManaged":false}},"name":"test-data-volume","size":5368709120,"path":"1f8244f6911b4a9d8aeb472483ccc8b2","volumeId":4,"accountId":2,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk\"]}","format":"OVA","provisioningType":"THIN","poolId":1,"id":4,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDownload":false,"deployAsIs":false,"checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":false}},"diskSeq":1,"path":"1f8244f6911b4a9d8aeb472483ccc8b2","type":"DATADISK","details":{"storageHost":"10.0.32.4","managed":"false","storagePort":"2049","storage.pool.disk.wait":"60","volumeSize":"5368709120"}},"vmName":"i-2-3-VM","inSeq":false,"forced":false,"controllerInfo":{"dataDiskController":"osdefault","rootDiskController":"ide"},"wait":0,"bypassHostMaintenance":false}].
2025-08-04 07:58:53,866 DEBUG [c.c.a.t.Request] (DirectAgent-276:[ctx-ab9bf866]) (logid:73720bf2) Seq 1-7125257560453545999: Processing:  { Ans: , MgmtId: 32986355466875, via: 1(10.0.32.154), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.AttachAnswer":{"disk":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4f324b33-7d7d-3a06-9332-d0ebc71774d1","name":"ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","id":"1","poolType":"NetworkFilesystem","host":"10.0.32.4","path":"/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","port":"2049","url":"NetworkFilesystem://10.0.32.4/acs/primary/ref-trl-9139-v-Mol8-rositsa-kyuchukova/ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1/?ROLE=Primary&STOREUUID=4f324b33-7d7d-3a06-9332-d0ebc71774d1","isManaged":"false"}},"name":"test-data-volume","size":"(5.00 GB) 5368709120","path":"1f8244f6911b4a9d8aeb472483ccc8b2","volumeId":"4","accountId":"2","chainInfo":"{"diskDeviceBusName":"scsi0:0","diskChain":["[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk"]}","format":"OVA","provisioningType":"THIN","poolId":"1","id":"4","bytesReadRate":"(0 bytes) 0","bytesWriteRate":"(0 bytes) 0","iopsReadRate":"(0 bytes) 0","iopsWriteRate":"(0 bytes) 0","hypervisorType":"VMware","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"false"}},"diskSeq":"1","path":"[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk","type":"DATADISK","details":{"storageHost":"10.0.32.4","managed":"false","storagePort":"2049","storage.pool.disk.wait":"60","volumeSize":"(5.00 GB) 5368709120"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-08-04 07:58:53,875 DEBUG [c.c.s.VolumeApiServiceImpl] (Work-Job-Executor-12:[ctx-0f8e7d8f, job-60/job-61, ctx-b021f9a9]) (logid:73720bf2) Volume: Volume {"id":4,"instanceId":3,"name":"test-data-volume","uuid":"b861fb24-ea21-4a5b-b460-1cb4447352fc","volumeType":"DATADISK"} successfully attached to VM: VM instance {"id":3,"instanceName":"i-2-3-VM","state":"Running","type":"User","uuid":"c36f5f65-b56e-4967-9cd1-0102bc393d66"}
2025-08-04 07:58:53,895 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-42:[ctx-9a06b0e0, job-60, ctx-524b0fd3]) (logid:73720bf2) Skipping delivering event Event {"description":"{\"details\":\"Volume Id: b861fb24-ea21-4a5b-b460-1cb4447352fc VmId: c36f5f65-b56e-4967-9cd1-0102bc393d66\",\"event\":\"VOLUME.ATTACH\",\"status\":\"Completed\"}","eventId":null,"eventType":"VOLUME.ATTACH","eventUuid":null,"resourceType":"Volume","resourceUUID":null} to any webhook as account ID is missing
2025-08-04 07:58:53,910 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-42:[ctx-9a06b0e0, job-60, ctx-524b0fd3]) (logid:73720bf2) Complete async job-60, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.VolumeResponse/volume/{"id":"b861fb24-ea21-4a5b-b460-1cb4447352fc","name":"test-data-volume","zoneid":"dca8a8fc-c751-4be2-b2b6-2807260294c6","zonename":"ref-trl-9139-v-Mol8-rositsa-kyuchukova","type":"DATADISK","deviceid":"1","virtualmachineid":"c36f5f65-b56e-4967-9cd1-0102bc393d66","vmname":"VM-c36f5f65-b56e-4967-9cd1-0102bc393d66","vmdisplayname":"VM-c36f5f65-b56e-4967-9cd1-0102bc393d66","vmstate":"Running","vmtype":"User","provisioningtype":"thin","size":"(5.00 GB) 5368709120","created":"2025-08-04T07:32:59+0000","state":"Ready","account":"admin","domainid":"bdb72b93-6ec2-11f0-b236-1e003c00027b","domain":"ROOT","domainpath":"/","storagetype":"shared","diskkbsread":"(0 bytes) 0","diskkbswrite":"(0 bytes) 0","diskioread":"(0 bytes) 0","diskiowrite":"(0 bytes) 0","hypervisor":"VMware","diskofferingid":"7ad37a9c-f0dc-4a26-822d-8348609f6d24","diskofferingname":"Small","diskofferingdisplaytext":"Small Disk, 5 GB","storage":"ref-trl-9139-v-Mol8-rositsa-kyuchukova-esxi-pri1","attached":"2025-08-04T07:58:53+0000","destroyed":"false","isextractable":"true","displayvolume":"true","path":"1f8244f6911b4a9d8aeb472483ccc8b2","storageid":"4f324b33-7d7d-3a06-9332-d0ebc71774d1","chaininfo":"{"diskDeviceBusName":"scsi0:0","diskChain":["[4f324b337d7d3a069332d0ebc71774d1] i-2-3-VM/1f8244f6911b4a9d8aeb472483ccc8b2.vmdk"]}","quiescevm":"false","supportsstoragesnapshot":"false","deleteprotection":"false","physicalsize":"0","virtualsize":"5368709120","utilization":"0.0%","clusterid":"873dbf23-6a29-41a4-9771-0c96761a0693","clustername":"p1-c1","podid":"ded01417-72b6-4e6a-8c2f-03fdec041dbb","podname":"Pod1","externaluuid":"6000C295-0feb-3e21-e1b0-6c2f68eea7fe","tags":[],"hasannotations":"false","jobid":"73720bf2-cfbf-49f1-b40b-63a497c31829","jobstatus":"0"}

@DaanHoogland
Copy link
Contributor

@blueorangutan package

@blueorangutan
Copy link

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14534

@DaanHoogland
Copy link
Contributor

@blueorangutan test

@blueorangutan
Copy link

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

rosi-shapeblue
rosi-shapeblue approved these changes Aug 4, 2025
View reviewed changes
Copy link
Collaborator

@rosi-shapeblue rosi-shapeblue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Re-tested with latest packages. Test results:

Detach Volume if Host is disconnected

  • Management server logs:
[root@ref-trl-9156-v-Mol8-rositsa-kyuchukova-mgmt1 ~]# tail -f /var/log/cloudstack/management/management-server.log | grep -E "detach|13f5d52a|VolumeApiServiceImpl"

2025-08-04 13:20:14,236 DEBUG [c.c.a.ApiServlet] (qtp253011924-19:[ctx-10c7ad91]) (logid:d3859861) ===START===  10.0.35.189 -- GET  apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=detachVolume&id=13f5d52a-1d4b-4aa2-b0a5-caa6714aca16&response=json&signature=742wwAd7Je47yIQC%2BjbPUiy3lmY%3D
2025-08-04 13:20:14,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp253011924-19:[ctx-10c7ad91, ctx-7e43ad08, ctx-c48d6af5]) (logid:d3859861) submit async job-63, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.volume.DetachVolumeCmdByAdmin","cmdInfo":"{\"apiKey\":\"LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q\",\"signature\":\"742wwAd7Je47yIQC+jbPUiy3lmY\\u003d\",\"response\":\"json\",\"ctxUserId\":\"2\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"184\",\"id\":\"13f5d52a-1d4b-4aa2-b0a5-caa6714aca16\",\"ctxDetails\":\"{\\\"interface com.cloud.storage.Volume\\\":\\\"13f5d52a-1d4b-4aa2-b0a5-caa6714aca16\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"13f5d52a-1d4b-4aa2-b0a5-caa6714aca16\",\"cmdEventType\":\"VOLUME.DETACH\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":63,"initMsid":32989291480144,"instanceId":4,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"491de186-fd30-46ea-833c-bab79f380a46"}
2025-08-04 13:20:14,265 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-46:[ctx-701c0b9f, job-63]) (logid:491de186) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.volume.DetachVolumeCmdByAdmin","cmdInfo":"{\"apiKey\":\"LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q\",\"signature\":\"742wwAd7Je47yIQC+jbPUiy3lmY\\u003d\",\"response\":\"json\",\"ctxUserId\":\"2\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"184\",\"id\":\"13f5d52a-1d4b-4aa2-b0a5-caa6714aca16\",\"ctxDetails\":\"{\\\"interface com.cloud.storage.Volume\\\":\\\"13f5d52a-1d4b-4aa2-b0a5-caa6714aca16\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"13f5d52a-1d4b-4aa2-b0a5-caa6714aca16\",\"cmdEventType\":\"VOLUME.DETACH\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":63,"initMsid":32989291480144,"instanceId":4,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"491de186-fd30-46ea-833c-bab79f380a46"}
2025-08-04 13:20:14,267 DEBUG [c.c.a.ApiServlet] (qtp253011924-19:[ctx-10c7ad91, ctx-7e43ad08, ctx-c48d6af5]) (logid:d3859861) ===END===  10.0.35.189 -- GET  apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=detachVolume&id=13f5d52a-1d4b-4aa2-b0a5-caa6714aca16&response=json&signature=742wwAd7Je47yIQC%2BjbPUiy3lmY%3D
2025-08-04 13:20:14,278 INFO  [c.c.s.VolumeApiServiceImpl] (API-Job-Executor-46:[ctx-701c0b9f, job-63, ctx-9b3511d2]) (logid:491de186) Trying to attach volume {"id":4,"name":"test-volumes","uuid":"13f5d52a-1d4b-4aa2-b0a5-caa6714aca16"} to VM instance {"id":3,"uuid":"f1236e9a-7ce0-4887-be93-b3e0ed89b605"}, update async job-63 progress status
2025-08-04 13:20:14,799 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-14:[ctx-1c45eac1, job-63/job-64, ctx-5f1ce139]) (logid:491de186) Execute VM work job: com.cloud.vm.VmWorkDetachVolume{"volumeId":4,"userId":2,"accountId":2,"vmId":3,"handlerName":"VolumeApiServiceImpl"}
2025-08-04 13:20:14,812 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-14:[ctx-1c45eac1, job-63/job-64, ctx-5f1ce139]) (logid:491de186) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume test-volumes from VM VM-f1236e9a-7ce0-4887-be93-b3e0ed89b605. Please contact your system administrator.
2025-08-04 13:20:14,813 INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-14:[ctx-1c45eac1, job-63/job-64, ctx-5f1ce139]) (logid:491de186) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume test-volumes from VM VM-f1236e9a-7ce0-4887-be93-b3e0ed89b605. Please contact your system administrator.
2025-08-04 13:20:14,813 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-14:[ctx-1c45eac1, job-63/job-64]) (logid:491de186) Unable to complete AsyncJob {"accountId":2,"cmd":"com.cloud.vm.VmWorkDetachVolume","cmdInfo":"rO0ABXNyAB9jb20uY2xvdWQudm0uVm1Xb3JrRGV0YWNoVm9sdW1lhvReA-s81AMCAAFMAAh2b2x1bWVJZHQAEExqYXZhL2xhbmcvTG9uZzt4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1ldAASTGphdmEvbGFuZy9TdHJpbmc7eHAAAAAAAAAAAgAAAAAAAAACAAAAAAAAAAN0ABRWb2x1bWVBcGlTZXJ2aWNlSW1wbHNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAABA","cmdVersion":0,"completeMsid":null,"created":"Mon Aug 04 13:20:14 UTC 2025","id":64,"initMsid":32989291480144,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"b9006d2e-4f04-4918-8c47-48947a7b034d"}, job origin: 63 com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume test-volumes from VM VM-f1236e9a-7ce0-4887-be93-b3e0ed89b605. Please contact your system administrator.
	at com.cloud.storage.VolumeApiServiceImpl.orchestrateDetachVolumeFromVM(VolumeApiServiceImpl.java:3207)
	at com.cloud.storage.VolumeApiServiceImpl.orchestrateDetachVolumeFromVM(VolumeApiServiceImpl.java:5231)
	at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:5270)
2025-08-04 13:20:14,841 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-46:[ctx-701c0b9f, job-63, ctx-9b3511d2]) (logid:491de186) Skipping delivering event Event {"description":"{\"details\":\"detaching volume: 13f5d52a-1d4b-4aa2-b0a5-caa6714aca16\",\"event\":\"VOLUME.DETACH\",\"status\":\"Completed\"}","eventId":null,"eventType":"VOLUME.DETACH","eventUuid":null,"resourceType":"Volume","resourceUUID":null} to any webhook as account ID is missing
2025-08-04 13:20:14,853 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-46:[ctx-701c0b9f, job-63]) (logid:491de186) Unexpected exception while executing org.apache.cloudstack.api.command.admin.volume.DetachVolumeCmdByAdmin com.cloud.utils.exception.CloudRuntimeException: Failed to detach volume test-volumes from VM VM-f1236e9a-7ce0-4887-be93-b3e0ed89b605. Please contact your system administrator.
	at com.cloud.storage.VolumeApiServiceImpl.orchestrateDetachVolumeFromVM(VolumeApiServiceImpl.java:3207)
	at com.cloud.storage.VolumeApiServiceImpl.orchestrateDetachVolumeFromVM(VolumeApiServiceImpl.java:5231)
	at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:5270)
2025-08-04 13:20:14,853 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-46:[ctx-701c0b9f, job-63]) (logid:491de186) Complete async job-63, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to detach volume test-volumes from VM VM-f1236e9a-7ce0-4887-be93-b3e0ed89b605. Please contact your system administrator."}
  • API response:
(localcloud) 🐱 > detach volume id=13f5d52a-1d4b-4aa2-b0a5-caa6714aca16
{
  "account": "admin",
  "accountid": "59d95932-7129-11f0-a882-1e00eb000450",
  "cmd": "org.apache.cloudstack.api.command.admin.volume.DetachVolumeCmdByAdmin",
  "completed": "2025-08-04T13:20:14+0000",
  "created": "2025-08-04T13:20:14+0000",
  "domainid": "122ca75d-7129-11f0-a882-1e00eb000450",
  "domainpath": "ROOT",
  "jobid": "491de186-fd30-46ea-833c-bab79f380a46",
  "jobinstanceid": "13f5d52a-1d4b-4aa2-b0a5-caa6714aca16",
  "jobinstancetype": "Volume",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "Failed to detach volume test-volumes from VM VM-f1236e9a-7ce0-4887-be93-b3e0ed89b605. Please contact your system administrator."
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "59da0482-7129-11f0-a882-1e00eb000450"
}
🙈 Error: async API failed for job 491de186-fd30-46ea-833c-bab79f380a46

Attach Volume if Host is disconnected

  • Management server logs:
[root@ref-trl-9156-v-Mol8-rositsa-kyuchukova-mgmt1 ~]# tail -f /var/log/cloudstack/management/management-server.log | grep -E "detach|13f5d52a|VolumeApiServiceImpl"
2025-08-04 13:22:06,775 INFO  [c.c.s.VolumeApiServiceImpl] (API-Job-Executor-47:[ctx-07fbc1e4, job-65, ctx-4ee40204]) (logid:97a52199) Trying to attach volume [VolumeObject {"dataStore":"StoragePool {\"id\":2,\"name\":\"ref-trl-9156-v-Mol8-rositsa-kyuchukova-esxi-pri2\",\"poolType\":\"NetworkFilesystem\",\"uuid\":\"44a66b66-23ea-3449-8196-dbbfd7792673\"}","volumeVO":"Volume {\"id\":5,\"instanceId\":null,\"name\":\"test-attach\",\"uuid\":\"972d57ce-0819-4fa2-b2e6-34ffc10c8e85\",\"volumeType\":\"DATADISK\"}"}] to VM instance [VM instance {"id":3,"instanceName":"i-2-3-VM","state":"Running","type":"User","uuid":"f1236e9a-7ce0-4887-be93-b3e0ed89b605"}], update async job-65 [AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.volume.AttachVolumeCmdByAdmin","cmdInfo":"{\"apiKey\":\"LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q\",\"virtualmachineid\":\"f1236e9a-7ce0-4887-be93-b3e0ed89b605\",\"signature\":\"6QPL4dV8lTLRsunmrD+D+CcKmCo\\u003d\",\"response\":\"json\",\"ctxUserId\":\"2\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"187\",\"id\":\"972d57ce-0819-4fa2-b2e6-34ffc10c8e85\",\"ctxDetails\":\"{\\\"interface com.cloud.storage.Volume\\\":\\\"972d57ce-0819-4fa2-b2e6-34ffc10c8e85\\\",\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"f1236e9a-7ce0-4887-be93-b3e0ed89b605\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"972d57ce-0819-4fa2-b2e6-34ffc10c8e85\",\"cmdEventType\":\"VOLUME.ATTACH\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":65,"initMsid":32989291480144,"instanceId":5,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"97a52199-c935-4702-bc78-a6a8b22ef736"}] progress status
2025-08-04 13:22:06,808 DEBUG [c.c.s.VolumeApiServiceImpl] (API-Job-Executor-47:[ctx-07fbc1e4, job-65, ctx-4ee40204]) (logid:97a52199) New job VmWorkJobVO : {, step: Starting, vmType: Instance, vmInstanceId: 3, secondaryObjectIdentifier: nullAsyncJob {"accountId":2,"cmd":"com.cloud.vm.VmWorkAttachVolume","cmdInfo":"rO0ABXNyAB9jb20uY2xvdWQudm0uVm1Xb3JrQXR0YWNoVm9sdW1lB62v-WGH4hwCAAJMAAhkZXZpY2VJZHQAEExqYXZhL2xhbmcvTG9uZztMAAh2b2x1bWVJZHEAfgABeHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAADdAAUVm9sdW1lQXBpU2VydmljZUltcGxwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAAF","cmdVersion":0,"completeMsid":null,"created":null,"id":66,"initMsid":32989291480144,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"067efddd-6d2e-42ef-b3f5-fb8e73911857"}}, result field: null
2025-08-04 13:22:08,795 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-15:[ctx-581381b3, job-65/job-66, ctx-a99affef]) (logid:97a52199) Execute VM work job: com.cloud.vm.VmWorkAttachVolume{"volumeId":5,"userId":2,"accountId":2,"vmId":3,"handlerName":"VolumeApiServiceImpl"}
	at com.cloud.storage.VolumeApiServiceImpl.createVolumeOnPrimaryForAttachIfNeeded(VolumeApiServiceImpl.java:2567)
	at com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:2585)
	at com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:5224)
	at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:5270)
	at com.cloud.storage.VolumeApiServiceImpl.createVolumeOnPrimaryForAttachIfNeeded(VolumeApiServiceImpl.java:2567)
	at com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:2585)
	at com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:5224)
	at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:5270)
  • API response:
(localcloud) 🐱 > attach volume id=972d57ce-0819-4fa2-b2e6-34ffc10c8e85 virtualmachineid=f1236e9a-7ce0-4887-be93-b3e0ed89b605
{
  "account": "admin",
  "accountid": "59d95932-7129-11f0-a882-1e00eb000450",
  "cmd": "org.apache.cloudstack.api.command.admin.volume.AttachVolumeCmdByAdmin",
  "completed": "2025-08-04T13:22:08+0000",
  "created": "2025-08-04T13:22:06+0000",
  "domainid": "122ca75d-7129-11f0-a882-1e00eb000450",
  "domainpath": "ROOT",
  "jobid": "97a52199-c935-4702-bc78-a6a8b22ef736",
  "jobinstanceid": "972d57ce-0819-4fa2-b2e6-34ffc10c8e85",
  "jobinstancetype": "Volume",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "Unable to find suitable primary storage when creating volume [{\"name\":\"test-attach\",\"uuid\":\"972d57ce-0819-4fa2-b2e6-34ffc10c8e85\"}]."
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "59da0482-7129-11f0-a882-1e00eb000450"
}
🙈 Error: async API failed for job 97a52199-c935-4702-bc78-a6a8b22ef736

@DaanHoogland
Copy link
Contributor

@sureshanaparti , I am not sure if smoke tests are going to give us any extra data above @rosi-shapeblue ’s testing so I think we can merge...

@sureshanaparti sureshanaparti merged commit d7b7bd5 into apache:main Aug 4, 2025
24 of 26 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in Apache CloudStack 4.21.0 Aug 4, 2025
@blueorangutan
Copy link

[SF] Trillian test result (tid-14023)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 51234 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10860-t14023-kvm-ol8.zip
Smoke tests completed. 146 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test Result Time (s) Test File

dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Aug 6, 2025
@erikbocks @dhslove
Fix infrastructure leak on exception while attaching/detaching volume…
d1043f0 
…s in VMware (apache#10860)

* Handled exception separately to prevent infrastructure data leak
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sureshanaparti sureshanaparti

@DaanHoogland DaanHoogland

+1 more reviewer

@rosi-shapeblue rosi-shapeblue

Reviewers whose approvals may not affect merge requirements
Assignees

@rosi-shapeblue rosi-shapeblue

Labels
component:storage
Projects
No open projects
Apache CloudStack 4.21.0
Status: Done
Milestone
4.21.0
Development

Successfully merging this pull request may close these issues.
5 participants
@erikbocks @DaanHoogland @blueorangutan @sureshanaparti @rosi-shapeblue