CleanUp Async Jobs after mgmt server maintenance #8394

kishankavala · 2023-12-21T06:24:39Z

Description

This PR fixes moves resources stuck in transition state during async job cleanup

Problem:
During maintenance of the management server, other servers in the cluster or the same server after a restart initiate async job cleanup. However, this process leaves resources in a transitional state. The only recovery option currently available is to make direct database changes.

Solution:
This PR introduces a resolution by changing Volume, Virtual Machine, and Network resources from their transitional states. This adjustment enables the reattempt of failed operations without the need for manual database modifications.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tested manually and with unit tests

codecov · 2023-12-21T06:33:35Z

Codecov Report

Attention: 45 lines in your changes are missing coverage. Please review.

Comparison is base (1411da1) 30.88% compared to head (14b5967) 30.75%.
Report is 14 commits behind head on main.

Files	Patch %	Lines
...stack/framework/jobs/impl/AsyncJobManagerImpl.java	32.30%	35 Missing and 9 partials ⚠️
...apache/cloudstack/storage/volume/VolumeObject.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8394      +/-   ##
============================================
- Coverage     30.88%   30.75%   -0.13%     
+ Complexity    34079    33941     -138     
============================================
  Files          5341     5341              
  Lines        374861   374922      +61     
  Branches      54518    54529      +11     
============================================
- Hits         115769   115323     -446     
- Misses       243825   244347     +522     
+ Partials      15267    15252      -15

Flag	Coverage Δ
simulator-marvin-tests	`24.64% <28.08%> (-0.18%)`	⬇️
uitests	`4.39% <ø> (+<0.01%)`	⬆️
unit-tests	`16.47% <46.06%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

harikrishna-patnala · 2023-12-21T06:46:26Z

@blueorangutan package

blueorangutan · 2023-12-21T06:48:03Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

harikrishna-patnala

Code LGTM

framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java

blueorangutan · 2023-12-21T07:51:30Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8111

rohityadavcloud · 2023-12-21T14:50:18Z

@blueorangutan test

blueorangutan · 2023-12-21T14:52:05Z

@rohityadavcloud a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

shwstppr · 2023-12-21T15:30:32Z

Moving this to 4.19.1 milestone for now cc @rohityadavcloud
If we are not able to cut RC this week and tests look good we can move it back and merge

framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java

blueorangutan · 2023-12-22T04:28:36Z

[SF] Trillian test result (tid-8653)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47435 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8394-t8653-kvm-centos7.zip
Smoke tests completed. 126 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_05_list_volumes_isrecursive	`Failure`	0.03	test_list_volumes.py
test_07_list_volumes_listall	`Failure`	0.02	test_list_volumes.py
test_02_upgrade_kubernetes_cluster	`Failure`	436.85	test_kubernetes_clusters.py

JoaoJandre

CLGTM, didn't test it

…reated from snapshots, and some code improvements

sureshanaparti · 2024-01-05T06:02:15Z

@blueorangutan package

blueorangutan · 2024-01-05T06:04:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-01-05T07:03:07Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8205

…created from snapshots

sureshanaparti · 2024-01-08T10:26:39Z

@blueorangutan package

blueorangutan · 2024-01-08T10:34:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-01-08T11:39:35Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8228

sureshanaparti · 2024-01-08T12:14:40Z

@blueorangutan test

blueorangutan · 2024-01-08T12:16:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2024-01-09T03:56:31Z

[SF] Trillian test result (tid-8754)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 54871 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8394-t8754-kvm-centos7.zip
Smoke tests completed. 121 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

kiranchavala

LGTM, tested the changes manually.

Stopped the management server during the following scenario's

while a attachvolume is in progress
while create volume from snapshot is in progress
While vm deployment along with a new network is in progress

After the management server is started

Volume is marked Ready and removed column date is kept as NULL
Volume is marked Allocated and removed column date is populated, the entries in volume_details table is removed for the snapshot
Vm state is stopped, volume is set in allocated and the network state is implementing

rohityadavcloud · 2024-01-18T11:43:22Z

@shwstppr can we get this in 4.19.0.0 ? Thanks.

shwstppr · 2024-01-18T12:34:55Z

@blueorangutan package

blueorangutan · 2024-01-18T12:36:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

shwstppr · 2024-01-18T15:11:28Z

@blueorangutan package

blueorangutan · 2024-01-18T15:12:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-01-18T16:18:02Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8370

shwstppr · 2024-01-18T16:33:49Z

@blueorangutan test matrix

blueorangutan · 2024-01-18T16:34:03Z

@shwstppr a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2024-01-19T04:35:15Z

[SF] Trillian test result (tid-8875)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41654 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8394-t8875-kvm-centos7.zip
Smoke tests completed. 121 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2024-01-19T06:26:04Z

[SF] Trillian test result (tid-8873)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 48423 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8394-t8873-xenserver-71.zip
Smoke tests completed. 120 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestSharedNetwork>:setup	`Error`	64.62	test_network.py

sureshanaparti · 2024-01-19T07:11:45Z

[SF] Trillian test result (tid-8873) Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7 Total time taken: 48423 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8394-t8873-xenserver-71.zip Smoke tests completed. 120 look OK, 1 have errors, 0 did not run Only failed and skipped tests results shown below:

Test Result Time (s) Test File
ContextSuite context=TestSharedNetwork>:setup Error 64.62 test_network.py

Error here while creating network, is not related to this PR changes. Changes here would reset/cleanup any Volume, VM & Network(in implementing state) resources for the pending jobs on MS start & is good to go. cc @shwstppr

Execute cmd: createnetwork failed, due to: errorCode: 431, errorText:The VLAN tag to use for new guest network, 2625 is already being used for dynamic vlan allocation for the guest network in zone pr8394-t8873-xenserver-71

This PR fixes moves resources stuck in transition state during async job cleanup Problem: During maintenance of the management server, other servers in the cluster or the same server after a restart initiate async job cleanup. However, this process leaves resources in a transitional state. The only recovery option currently available is to make direct database changes. Solution: This PR introduces a resolution by changing Volume, Virtual Machine, and Network resources from their transitional states. This adjustment enables the reattempt of failed operations without the need for manual database modifications.

Move resources from transition state during async job cleanup

c4cb6be

boring-cyborg bot added component:orchestration component:storage component:vpc labels Dec 21, 2023

harikrishna-patnala approved these changes Dec 21, 2023

View reviewed changes

framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java Outdated Show resolved Hide resolved

rohityadavcloud requested review from shwstppr and sureshanaparti December 21, 2023 07:41

rohityadavcloud modified the milestones: 4.18.2.0, 4.19.0.0 Dec 21, 2023

shwstppr modified the milestones: 4.19.0.0, 4.19.1.0 Dec 21, 2023

JoaoJandre reviewed Dec 21, 2023

View reviewed changes

framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java Outdated Show resolved Hide resolved

JoaoJandre approved these changes Dec 27, 2023

View reviewed changes

kiranchavala mentioned this pull request Jan 1, 2024

Unable to delete the VolumeSnapshots in Backing up state #8423

Closed

sureshanaparti added 2 commits January 5, 2024 11:09

Added transitional flag in Volume state

4a1de94

Updated network event for failed job, (re)added cleanup for volumes c…

790a503

…reated from snapshots, and some code improvements

boring-cyborg bot added the component:api label Jan 5, 2024

Added java doc for volume state constructor

04c636d

Fixed cleanup SNAPSHOT_ID entry in volume details for failed volumes …

14b5967

…created from snapshots

sureshanaparti self-assigned this Jan 8, 2024

kiranchavala approved these changes Jan 9, 2024

View reviewed changes

shwstppr modified the milestones: 4.19.1.0, 4.19.0.0 Jan 18, 2024

shwstppr merged commit 80bbb29 into apache:main Jan 19, 2024
25 of 26 checks passed

shwstppr deleted the async_cleanup branch January 19, 2024 07:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CleanUp Async Jobs after mgmt server maintenance #8394

CleanUp Async Jobs after mgmt server maintenance #8394

kishankavala commented Dec 21, 2023

codecov bot commented Dec 21, 2023 •

edited

harikrishna-patnala commented Dec 21, 2023

blueorangutan commented Dec 21, 2023

harikrishna-patnala left a comment

blueorangutan commented Dec 21, 2023

rohityadavcloud commented Dec 21, 2023

blueorangutan commented Dec 21, 2023

shwstppr commented Dec 21, 2023

blueorangutan commented Dec 22, 2023

JoaoJandre left a comment

sureshanaparti commented Jan 5, 2024

blueorangutan commented Jan 5, 2024

blueorangutan commented Jan 5, 2024

sureshanaparti commented Jan 8, 2024

blueorangutan commented Jan 8, 2024

blueorangutan commented Jan 8, 2024

sureshanaparti commented Jan 8, 2024

blueorangutan commented Jan 8, 2024

blueorangutan commented Jan 9, 2024

kiranchavala left a comment

rohityadavcloud commented Jan 18, 2024

shwstppr commented Jan 18, 2024

blueorangutan commented Jan 18, 2024

shwstppr commented Jan 18, 2024

blueorangutan commented Jan 18, 2024

blueorangutan commented Jan 18, 2024

shwstppr commented Jan 18, 2024

blueorangutan commented Jan 18, 2024

blueorangutan commented Jan 19, 2024

blueorangutan commented Jan 19, 2024

sureshanaparti commented Jan 19, 2024

CleanUp Async Jobs after mgmt server maintenance #8394

CleanUp Async Jobs after mgmt server maintenance #8394

Conversation

kishankavala commented Dec 21, 2023

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

codecov bot commented Dec 21, 2023 • edited

Codecov Report

harikrishna-patnala commented Dec 21, 2023

blueorangutan commented Dec 21, 2023

harikrishna-patnala left a comment

Choose a reason for hiding this comment

blueorangutan commented Dec 21, 2023

rohityadavcloud commented Dec 21, 2023

blueorangutan commented Dec 21, 2023

shwstppr commented Dec 21, 2023

blueorangutan commented Dec 22, 2023

JoaoJandre left a comment

Choose a reason for hiding this comment

sureshanaparti commented Jan 5, 2024

blueorangutan commented Jan 5, 2024

blueorangutan commented Jan 5, 2024

sureshanaparti commented Jan 8, 2024

blueorangutan commented Jan 8, 2024

blueorangutan commented Jan 8, 2024

sureshanaparti commented Jan 8, 2024

blueorangutan commented Jan 8, 2024

blueorangutan commented Jan 9, 2024

kiranchavala left a comment

Choose a reason for hiding this comment

rohityadavcloud commented Jan 18, 2024

shwstppr commented Jan 18, 2024

blueorangutan commented Jan 18, 2024

shwstppr commented Jan 18, 2024

blueorangutan commented Jan 18, 2024

blueorangutan commented Jan 18, 2024

shwstppr commented Jan 18, 2024

blueorangutan commented Jan 18, 2024

blueorangutan commented Jan 19, 2024

blueorangutan commented Jan 19, 2024

sureshanaparti commented Jan 19, 2024

codecov bot commented Dec 21, 2023 •

edited