Feature: Safely shutdown cloudstack #6755

davidjumani · 2022-09-21T11:15:26Z

Description

Adds the feature to safely shutdown CloudStack
It does the following :

Prevents new Async jobs from being added
Waits for existing jobs to finish before shutting down

Contains 4 new apis :

TriggerShutdown - Prevents new jobs and shuts down once all pending jobs are completed
ReadyForShutdown - Returns whether a shutdown has been triggered and the no of pending jobs
PrepareForShutdown - Prevents new jobs from being added but does not shutdown when there are zero pending jobs : Useful when admin changes are required in ACS which can impact operations
CancelShutdown - Cancel a shutdown if possible

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

How Has This Been Tested?

TODO

acs-robot · 2022-09-21T11:16:36Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

acs-robot · 2022-09-21T11:19:30Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-09-21T11:20:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-09-21T11:25:07Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6755 (SL-JID-2390)

acs-robot · 2022-09-22T05:23:25Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-09-22T05:24:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

acs-robot · 2022-09-22T05:29:03Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-09-22T05:30:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-09-22T05:31:31Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6755 (SL-JID-2396)

blueorangutan · 2022-09-22T05:36:49Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6755 (SL-JID-2397)

acs-robot · 2022-09-22T06:47:52Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-09-22T06:49:02Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-09-22T06:54:31Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6755 (SL-JID-2399)

sonarqubecloud · 2022-09-22T08:13:47Z

SonarCloud Quality Gate failed.

1 Bug
0 Vulnerabilities
0 Security Hotspots
6 Code Smells

29.5% Coverage
9.0% Duplication

DaanHoogland

looks generally good.
A functional question though, there is a UI component, but will the API let all clustered MS shutdown? (I didn´t see code for that) It seems only the MS that happens to handle the API will shut down.

plugins/shutdown/pom.xml

DaanHoogland · 2022-09-27T12:56:59Z

framework/jobs/src/main/java/org/apache/cloudstack/framework/jobs/impl/AsyncJobManagerImpl.java

    private static final int HEARTBEAT_INTERVAL = 2000;
    private static final int GC_INTERVAL = 10000;                // 10 seconds

+    private boolean allowAsyncJobs = true ;


seems like this should be called shutdownTriggered, or else the messages of the exceptions are to specific and we don´t know the reason the async jobs are disallowed is a shutdown instead of some other maintenance task.

I'm guessing this can be reused later, hence the more generic naming. I'll look into polishing the message accordingly

plugins/shutdown/src/main/java/org/apache/cloudstack/api/command/CancelShutdownCmd.java

plugins/shutdown/src/main/java/org/apache/cloudstack/api/command/PrepareForShutdownCmd.java

plugins/shutdown/src/main/java/org/apache/cloudstack/api/command/TriggerShutdownCmd.java

plugins/shutdown/src/main/java/org/apache/cloudstack/shutdown/ShutdownManager.java

plugins/shutdown/src/main/java/org/apache/cloudstack/shutdown/ShutdownManagerImpl.java

DaanHoogland · 2022-09-27T13:07:10Z

plugins/shutdown/src/main/java/org/apache/cloudstack/shutdown/ShutdownManagerImpl.java

+                    }
+                    if (shutdownManager.countPendingJobs() == 0) {
+                        s_logger.info("Shutting down now");
+                        System.exit(0);


are other threads allowed to shutdown as well (by maybe setting a flag on ManagedContextRunnable and adding a check for it there? (<== genuine question)

When System.exit() is called, all shutdown hooks are triggered and only when complete does the program exit

https://docs.oracle.com/javase/7/docs/api/java/lang/Runtime.html#exit(int)

plugins/shutdown/src/test/java/org/apache/cloudstack/shutdown/ShutdownManagerImplTest.java

acs-robot · 2022-10-31T05:52:42Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-10-31T05:53:02Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-10-31T06:04:10Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6755 (SL-JID-2596)

davidjumani · 2022-10-31T06:11:20Z

@blueorangutan package

blueorangutan · 2022-10-31T06:12:02Z

@davidjumani a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

DaanHoogland · 2022-10-31T06:22:39Z

looks generally good. A functional question though, there is a UI component, but will the API let all clustered MS shutdown? (I didn´t see code for that) It seems only the MS that happens to handle the API will shut down.

@davidjumani can you explain this?

davidjumani · 2023-03-28T09:09:50Z

@blueorangutan package

blueorangutan · 2023-03-28T09:10:03Z

@davidjumani a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-03-28T10:07:25Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 5789

davidjumani · 2023-03-28T10:13:59Z

@blueorangutan test

blueorangutan · 2023-03-28T10:15:03Z

@davidjumani a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-03-28T16:40:32Z

Trillian test result (tid-6328)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 21691 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6755-t6328-kvm-centos7.zip
Smoke tests completed. 84 look OK, 0 have errors, 25 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
all_test_safe_shutdown	`Skipped`	---	test_safe_shutdown.py
all_test_login	`Skipped`	---	test_login.py
all_test_scale_vm	`Skipped`	---	test_scale_vm.py
all_test_metrics_api	`Skipped`	---	test_metrics_api.py
all_test_outofbandmanagement	`Skipped`	---	test_outofbandmanagement.py
all_test_outofbandmanagement_nestedplugin	`Skipped`	---	test_outofbandmanagement_nestedplugin.py
all_test_routers_iptables_default_policy	`Skipped`	---	test_routers_iptables_default_policy.py
all_test_secondary_storage	`Skipped`	---	test_secondary_storage.py
all_test_service_offerings	`Skipped`	---	test_service_offerings.py
all_test_storage_policy	`Skipped`	---	test_storage_policy.py
all_test_update_security_group	`Skipped`	---	test_update_security_group.py
all_test_usage_events	`Skipped`	---	test_usage_events.py
all_test_vm_autoscaling	`Skipped`	---	test_vm_autoscaling.py
all_test_vm_deployment_planner	`Skipped`	---	test_vm_deployment_planner.py
all_test_vm_life_cycle	`Skipped`	---	test_vm_life_cycle.py
all_test_vm_lifecycle_unmanage_import	`Skipped`	---	test_vm_lifecycle_unmanage_import.py
all_test_vm_snapshot_kvm	`Skipped`	---	test_vm_snapshot_kvm.py
all_test_vm_snapshots	`Skipped`	---	test_vm_snapshots.py
all_test_volumes	`Skipped`	---	test_volumes.py
all_test_vpc_ipv6	`Skipped`	---	test_vpc_ipv6.py
all_test_vpc_redundant	`Skipped`	---	test_vpc_redundant.py
all_test_vpc_router_nics	`Skipped`	---	test_vpc_router_nics.py
all_test_vpc_vpn	`Skipped`	---	test_vpc_vpn.py
all_test_host_maintenance	`Skipped`	---	test_host_maintenance.py
all_test_hostha_kvm	`Skipped`	---	test_hostha_kvm.py

davidjumani · 2023-03-31T04:18:46Z

@blueorangutan package

blueorangutan · 2023-03-31T04:19:03Z

@davidjumani a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-03-31T05:21:45Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 5818

davidjumani · 2023-03-31T05:26:44Z

@blueorangutan test

blueorangutan · 2023-03-31T05:27:03Z

@davidjumani a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-03-31T17:53:23Z

Trillian test result (tid-6345)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43283 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6755-t6345-kvm-centos7.zip
Smoke tests completed. 109 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

kiranchavala

Tested the feature of Safe shutdown, LGTM

Scenarios I had tested

safeshutdown-testcases.xlsx

DaanHoogland

clgtm

github-actions · 2023-04-11T09:05:16Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

sonarqubecloud · 2023-04-12T10:34:39Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
5 Code Smells

13.6% Coverage
0.0% Duplication

rohityadavcloud · 2023-04-12T14:53:16Z

Thanks, great feature. The banner background colour and text colour aren't in-line with our current AntD theme. Should this be warning (yellow) or error (red) for the message? https://antdv.com/components/alert/#Alert (see banner, to be used on top of the page)

davidjumani · 2023-04-13T03:59:52Z

Thanks @rohityadavcloud I'll create a PR to address this

* Safely shutdown feature (ref: apache#6755) * Updated version and some improvements * Management Server Maintenance - Prepare and Cancel Maintenance changes This is supported for the Cloudstack deployments with multiple management servers. - During preparing for maintenance, MS waits for pending jobs to finish, and then Transfer/Migrate the agents to other available MS - New APIs: prepareForMaintenance, cancelMaintenance - New MS States: PreparingToMaintenance, Maintenance * check for single active management server * refactoring plugin name * updated version, and cleanup * code improvements * support list hosts by management server id * update ui with ms maintenance apis * code improvements * ui changes * ui icons update * ui fixes * cond checks for maintenance and shutdown * fix for management server not down issue on service stop * continue with other components on error * agent transfer fixes * maintenance window timeout and fixes * ui changes - added connected agents tab, and updated hosts & management servers fields * marvin test update * keep maintenance after shutdown/restart, do not update last_updated time in cluster heartbeat during maintenance (notifies node inactive/down after heartbeat threshold) * listener for ms maintenance updates * cleanup * keep last msid in host table * review comments * allow only one mgmt server to prepare for maintenance * added ms uuid in logs * minor code improvements * ui fields update * fix systemvm navigation in connected agents * algorithm check and input from ui * check for active ms from host setting * agent migration code improvements * minor ui label fix * fixes & code improvements * agent reconnect fixes, consider avoid list * ui fixes * direct agents transfer and pending jobs timer task fixes * close unclosed socket channels if any * Updated pending jobs check timer task with ScheduledExecutorService * fixes * keep maintenance state on trigger shutdown call when ms is in maintenance * direct agent transfer fixes * add pending jobs count to ms response * during ms heartbeat, update state to up only when it's down * allow vm work jobs of async job created before prepare for maintenance * Revert "keep maintenance state on trigger shutdown call when ms is in maintenance" This reverts commit 4ebbea71ef20a65286bed41a517f03e253a8fe90. * removed duplicate schema changes from schema-41800to41810.sql (already defined at schema-41811to41812.sql)

boring-cyborg bot added component:api component:build component:UI Python Warning... Python code Ahead! labels Sep 21, 2022

davidjumani force-pushed the safe-shutdown branch from 4d7f30f to dc36104 Compare September 21, 2022 11:16

davidjumani force-pushed the safe-shutdown branch from dc36104 to 3f07449 Compare September 22, 2022 05:21

davidjumani force-pushed the safe-shutdown branch from 3f07449 to 7719141 Compare September 22, 2022 05:26

davidjumani force-pushed the safe-shutdown branch from 7719141 to db8287a Compare September 22, 2022 06:46

DaanHoogland reviewed Sep 27, 2022

View reviewed changes

davidjumani force-pushed the safe-shutdown branch from db8287a to f851091 Compare October 31, 2022 05:51

Fix line endings

89b2d4f

davidjumani marked this pull request as draft March 28, 2023 10:14

kiranchavala approved these changes Apr 4, 2023

View reviewed changes

DaanHoogland approved these changes Apr 6, 2023

View reviewed changes

DaanHoogland marked this pull request as ready for review April 6, 2023 11:21

github-actions bot added the status:has-conflicts label Apr 11, 2023

Merge branch 'main' of github.com:apache/cloudstack into safe-shutdown

79cc970

github-actions bot removed the status:has-conflicts label Apr 12, 2023

DaanHoogland merged commit 941cc83 into apache:main Apr 12, 2023

davidjumani deleted the safe-shutdown branch April 12, 2023 10:55

rohityadavcloud added this to the 4.19.0.0 milestone Apr 12, 2023

rohityadavcloud mentioned this pull request Apr 19, 2023

Feature x #6662

Closed

12 tasks

sureshanaparti mentioned this pull request Oct 25, 2024

Support for Management Server Maintenance Mode #9854

Merged

14 tasks

Feature: Safely shutdown cloudstack #6755

Feature: Safely shutdown cloudstack #6755

Uh oh!

Conversation

davidjumani commented Sep 21, 2022

Description

Types of changes

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

acs-robot commented Sep 21, 2022

Uh oh!

acs-robot commented Sep 21, 2022

Uh oh!

blueorangutan commented Sep 21, 2022

Uh oh!

blueorangutan commented Sep 21, 2022

Uh oh!

acs-robot commented Sep 22, 2022

Uh oh!

blueorangutan commented Sep 22, 2022

Uh oh!

acs-robot commented Sep 22, 2022

Uh oh!

blueorangutan commented Sep 22, 2022

Uh oh!

blueorangutan commented Sep 22, 2022

Uh oh!

blueorangutan commented Sep 22, 2022

Uh oh!

acs-robot commented Sep 22, 2022

Uh oh!

blueorangutan commented Sep 22, 2022

Uh oh!

blueorangutan commented Sep 22, 2022

Uh oh!

sonarqubecloud bot commented Sep 22, 2022

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DaanHoogland Sep 27, 2022

Choose a reason for hiding this comment

Uh oh!

davidjumani Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DaanHoogland Sep 27, 2022

Choose a reason for hiding this comment

Uh oh!

davidjumani Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

acs-robot commented Oct 31, 2022

Uh oh!

blueorangutan commented Oct 31, 2022

Uh oh!

blueorangutan commented Oct 31, 2022

Uh oh!

davidjumani commented Oct 31, 2022

Uh oh!

blueorangutan commented Oct 31, 2022

Uh oh!

DaanHoogland commented Oct 31, 2022

Uh oh!

davidjumani commented Mar 28, 2023

Uh oh!

blueorangutan commented Mar 28, 2023

Uh oh!

blueorangutan commented Mar 28, 2023

Uh oh!

kiranchavala left a comment •

edited

Loading

rohityadavcloud commented Apr 12, 2023 •

edited

Loading