alert: Send alert for ha'ed vm's #5664

ravening · 2021-11-04T11:42:51Z

Description

When ha is performed on vm's send the alert for it so that
its for admins to know which vm's got ha'ed else its time
consuming to get those details from logs

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

DaanHoogland

clgtm, but would the subject exactly the same as the content be helpful? can we add more info to the mail content?

ravening · 2021-11-04T12:32:17Z

clgtm, but would the subject exactly the same as the content be helpful? can we add more info to the mail content?

@DaanHoogland if you have any content then let me know.. i can add that

DaanHoogland · 2021-11-04T14:15:08Z

clgtm, but would the subject exactly the same as the content be helpful? can we add more info to the mail content?

@DaanHoogland if you have any content then let me know.. i can add that

source and target host? time down? time up? down time duration?
just suggestions @ravening . What would you need to know as operator. As said, clgtm

rohityadavcloud · 2021-11-18T08:22:36Z

@blueorangutan package

blueorangutan · 2021-11-18T08:23:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-11-18T10:28:42Z

Packaging result: ✖️ el7 ✔️ el8 ✖️ debian ✖️ suse15. SL-JID 1723

rohityadavcloud · 2021-11-23T11:20:21Z

@blueorangutan package

blueorangutan · 2021-11-23T11:21:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-11-23T13:25:58Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✔️ suse15. SL-JID 1771

rohityadavcloud · 2021-12-30T10:50:54Z

@blueorangutan package

blueorangutan · 2021-12-30T10:53:08Z

@rohityadavcloud a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-12-30T12:42:21Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✖️ suse15. SL-JID 2039

GabrielBrascher

Thanks for the PR @ravening.
I've proposed a few changes that, in my opinion, can make this PR even more helpful :-)

GabrielBrascher · 2021-12-30T16:05:19Z

server/src/main/java/com/cloud/ha/HighAvailabilityManagerImpl.java

@@ -608,7 +608,9 @@ protected Long restart(final HaWorkVO work) {

            VMInstanceVO started = _instanceDao.findById(vm.getId());
            if (started != null && started.getState() == VirtualMachine.State.Running) {
+                String message = String.format("HA on VM: %s", started.getHostName());


I like the idea of enhancing the alerts.
What do you think of changing from Ha on VM: VM Name to Ha Starting VM: VM Name (i-2-13477-VM).

Thus, making it String.format("HA on VM: %s (%s)", started.getHostName(), started.getInstanceName());.

@ravening Do you want to add instance name to the alert message (as advised)?

GabrielBrascher · 2021-12-30T16:13:51Z

server/src/main/java/com/cloud/ha/HighAvailabilityManagerImpl.java

@@ -608,7 +608,9 @@ protected Long restart(final HaWorkVO work) {

            VMInstanceVO started = _instanceDao.findById(vm.getId());
            if (started != null && started.getState() == VirtualMachine.State.Running) {
+                String message = String.format("HA on VM: %s", started.getHostName());
                s_logger.info("VM is now restarted: " + vmId + " on " + started.getHostId());


Taking the opportunity, what do you think of enhancing also the INFO log message?

As a suggestion, it could be logged something as the following:

HA is now restarting VM instance {id: "123", name: "VM Name", uuid: "1234.....abc", type="User"} on Host [{id: "1", name: "host.name", uuid: "ABC...123", type="Routing"}]

With the following code changes:

Suggested change

s_logger.info("VM is now restarted: " + vmId + " on " + started.getHostId());

HostVO hostVmHasStarted = _hostDao.findById(started.getHostId());

s_logger.info(String.format("HA is now restarting %s on %s", started, hostVmHasStarted));

DaanHoogland · 2022-01-10T10:09:13Z

@blueorangutan package

blueorangutan · 2022-01-10T10:10:03Z

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-01-10T11:45:58Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✖️ suse15. SL-JID 2157

DaanHoogland · 2022-01-10T13:35:37Z

@blueorangutan test

sureshanaparti · 2022-01-10T14:16:01Z

@blueorangutan test

blueorangutan · 2022-01-10T14:19:03Z

@sureshanaparti a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-01-10T23:28:38Z

Trillian test result (tid-2845)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31418 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5664-t2845-kvm-centos7.zip
Smoke tests completed. 92 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

sureshanaparti · 2022-01-11T06:56:50Z

HI @ravening Can you address outstanding comments. Thanks.

nvazquez · 2022-02-06T02:41:35Z

Ping @ravening

When ha is performed on vm's send the alert for it so that its for admins to know which vm's got ha'ed else its time consuming to get those details from logs

ravening · 2022-02-09T13:45:05Z

Ping @ravening

@nvazquez done

nvazquez · 2022-02-09T18:27:30Z

@blueorangutan package

blueorangutan · 2022-02-09T18:28:03Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-02-09T18:57:09Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✔️ suse15. SL-JID 2545

GabrielBrascher

Code LGTM

nvazquez · 2022-02-18T02:20:48Z

@blueorangutan package

blueorangutan · 2022-02-18T02:22:03Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-02-18T03:03:42Z

Packaging result: ✔️ el7 ✖️ el8 ✔️ debian ✔️ suse15. SL-JID 2647

nvazquez · 2022-03-16T03:14:20Z

@blueorangutan package

blueorangutan · 2022-03-16T03:15:03Z

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-03-16T04:28:47Z

Packaging result: ✖️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2895

nvazquez · 2022-03-16T04:30:18Z

@blueorangutan package

blueorangutan · 2022-03-16T04:31:03Z

@nvazquez a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-03-16T05:08:31Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2898

nvazquez · 2022-03-16T05:15:53Z

@blueorangutan test

blueorangutan · 2022-03-16T05:17:03Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-03-16T14:46:53Z

Trillian test result (tid-3628)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32812 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5664-t3628-kvm-centos7.zip
Smoke tests completed. 92 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland approved these changes Nov 4, 2021

View reviewed changes

rohityadavcloud approved these changes Nov 18, 2021

View reviewed changes

rohityadavcloud approved these changes Dec 30, 2021

View reviewed changes

rohityadavcloud added this to the 4.17.0.0 milestone Dec 30, 2021

GabrielBrascher reviewed Dec 30, 2021

View reviewed changes

apache deleted a comment from rohityadavcloud Jan 10, 2022

apache deleted a comment from blueorangutan Jan 10, 2022

apache deleted a comment from sureshanaparti Jan 10, 2022

apache deleted a comment from blueorangutan Jan 10, 2022

nvazquez assigned ravening Feb 6, 2022

nvazquez added the type:enhancement label Feb 6, 2022

Rakesh Venkatesh added 2 commits February 9, 2022 14:37

alert: Send alert for ha'ed vm's

165e3b2

When ha is performed on vm's send the alert for it so that its for admins to know which vm's got ha'ed else its time consuming to get those details from logs

feedback change

b905d57

ravening force-pushed the ha_alerts branch from ae80614 to b905d57 Compare February 9, 2022 13:44

GabrielBrascher approved these changes Feb 9, 2022

View reviewed changes

nvazquez approved these changes Mar 17, 2022

View reviewed changes

nvazquez merged commit 6f3c18f into apache:main Mar 17, 2022

	s_logger.info("VM is now restarted: " + vmId + " on " + started.getHostId());
	HostVO hostVmHasStarted = _hostDao.findById(started.getHostId());
	s_logger.info(String.format("HA is now restarting %s on %s", started, hostVmHasStarted));

alert: Send alert for ha'ed vm's #5664

alert: Send alert for ha'ed vm's #5664

Conversation

ravening commented Nov 4, 2021

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

DaanHoogland left a comment

Choose a reason for hiding this comment

ravening commented Nov 4, 2021

DaanHoogland commented Nov 4, 2021

rohityadavcloud commented Nov 18, 2021

blueorangutan commented Nov 18, 2021

blueorangutan commented Nov 18, 2021

rohityadavcloud commented Nov 23, 2021

blueorangutan commented Nov 23, 2021

blueorangutan commented Nov 23, 2021

rohityadavcloud commented Dec 30, 2021

blueorangutan commented Dec 30, 2021

blueorangutan commented Dec 30, 2021

GabrielBrascher left a comment

Choose a reason for hiding this comment

GabrielBrascher Dec 30, 2021

Choose a reason for hiding this comment

sureshanaparti Jan 11, 2022

Choose a reason for hiding this comment

GabrielBrascher Dec 30, 2021

Choose a reason for hiding this comment

DaanHoogland commented Jan 10, 2022

blueorangutan commented Jan 10, 2022

blueorangutan commented Jan 10, 2022

DaanHoogland commented Jan 10, 2022

sureshanaparti commented Jan 10, 2022

blueorangutan commented Jan 10, 2022

blueorangutan commented Jan 10, 2022

sureshanaparti commented Jan 11, 2022

nvazquez commented Feb 6, 2022

ravening commented Feb 9, 2022

nvazquez commented Feb 9, 2022

blueorangutan commented Feb 9, 2022

blueorangutan commented Feb 9, 2022

GabrielBrascher left a comment

Choose a reason for hiding this comment

nvazquez commented Feb 18, 2022

blueorangutan commented Feb 18, 2022

blueorangutan commented Feb 18, 2022

nvazquez commented Mar 16, 2022

blueorangutan commented Mar 16, 2022

blueorangutan commented Mar 16, 2022

nvazquez commented Mar 16, 2022

blueorangutan commented Mar 16, 2022

blueorangutan commented Mar 16, 2022

nvazquez commented Mar 16, 2022

blueorangutan commented Mar 16, 2022

blueorangutan commented Mar 16, 2022