[FEATURE REQUEST] Cluster upgrade monitor notifications support

[craftyhouse]

Is your feature request related to a problem? Please describe.
As an operator I would like to know when cluster upgrades are occurring and their status. Today SFRP only supports notification if there is a failure. See https://docs.microsoft.com/en-gb/azure/service-fabric/service-fabric-cluster-upgrade-version-azure#register-for-notifications

Describe the solution you'd like
A clear and concise description of what you want to happen.

Able to easily configure an email notification to stay informed
https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-diagnostics-event-generation-operational#cluster-events

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Alternatives are described in https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-diagnostics-event-generation-operational#cluster-events , but not prescriptive. It's up to the customer to figure out and may incur more charges.

Additional context
Add any other context or screenshots about the feature request here.

[GitTorre]

This would make sense in ClusterObserver, which already reads and transmits cluster, node, and app level health information via etw events and telemetry (AppInsights, LogAnalytics) as a single instance service. FO runs on each node, so this type of monitoring would be redundant across instances. Will modify ClusterObserver to support this.

[andyrdean]

+1 please on behalf of my customer.

[GitTorre]

You can track progress in the develop branch. Initial impl is in place there.

[GitTorre]

This will ship in the next release, which should ship this week along with FO 3.1.24.

Explanation of this feature.

CO will query for Upgrade status across Cluster and Application scope. Each time it detects some change in upgrade state, it will emit telemetry and ETW that contains the details. In the case when you want to know if some upgrade failed, you will see a FailureReason field. This will contain the name of the upgrade step where the failure happened (e.g., it failed in HealthCheck step). Here is an example of CO LogAnalytics telemetry where a cluster upgrade was moving forward across UDs, but then RolledBack due to a failure in HealthCheck step (SF decided it was not safe to progress with the upgrade and began rolling back to previous version).

[GitTorre]

More details will be in CO Readme upon shipping.

[GitTorre]

This is now shipped.

[andyrdean]

Awesome! Thanks Charles 👍.