Merge branch 'eichhorl/monitor-last-cup' into 'master'

feat(consensus): CON-1063 Persist ECDSA key monitoring metric across upgrades

Erroneous threshold public key changes are monitored in the orchestrator by comparing the keys of a CUP with those of the previous CUP. If a key was changed or deleted, the corresponding metric is incremented, thus raising an alert. 

However, this doesn't work for the last CUP before an upgrade:

As the orchestrator typically shuts down immediately after inspecting the last CUP before an upgrade, the error metric is reset before it can be scraped.

In this MR we persist the result of comparing the last two CUPs in the orchestrator's data directory. After restart, we initialize the metric with the persisted result and thus guarantee that the alert is raised correctly.

Metrics of keys that haven't changed in the last CUP are reset to 0, as we assume that the alert was already raised.

In order to not raise false alarms after membership changes, we reset the metric once a node is unassigned. 

See merge request dfinity-lab/public/ic!13126

rs/orchestrator/image_upgrader/src/image_upgrader.rs

@@ -101,8 +101,8 @@ pub trait ImageUpgrader<V: Clone + Debug + PartialEq + Eq + Send + Sync, R: Send
     /// Path to the image image download and unpacking destination.
     fn image_path(&self) -> &PathBuf;
     /// Optional data path, used for storing latest reboot time. Default is None.
-    fn data_dir(&self) -> &Option<PathBuf> {
-        &None
+    fn data_dir(&self) -> Option<&PathBuf> {
+        None
     }
     /// Return the logger to be passed to the upgrade functions.
     fn log(&self) -> &ReplicaLogger;

rs/orchestrator/src/args.rs

@@ -59,7 +59,7 @@ pub struct OrchestratorArgs {
     /// The path to directory that is dedicated to data specific to the orchstrator.
     /// If not provided, the relevant data are not persisted to the disk.
     #[clap(long, parse(from_os_str))]
-    pub(crate) orchestrator_data_directory: Option<PathBuf>,
+    pub(crate) orchestrator_data_directory: PathBuf,
 }
 
 impl OrchestratorArgs {

rs/orchestrator/src/error.rs

@@ -51,6 +51,9 @@ pub enum OrchestratorError {
 
     /// Generic error while handling reboot time
     RebootTimeError(String),
+
+    /// Generic error while monitoring key changes
+    ThresholdKeyMonitoringError(String),
 }
 
 impl OrchestratorError {
@@ -61,6 +64,10 @@ impl OrchestratorError {
     pub(crate) fn invalid_configuration_error(msg: impl ToString) -> Self {
         OrchestratorError::InvalidConfigurationError(msg.to_string())
     }
+
+    pub(crate) fn key_monitoring_error(msg: impl ToString) -> Self {
+        OrchestratorError::ThresholdKeyMonitoringError(msg.to_string())
+    }
 }
 
 impl fmt::Display for OrchestratorError {
@@ -94,6 +101,13 @@ impl fmt::Display for OrchestratorError {
             OrchestratorError::RebootTimeError(msg) => {
                 write!(f, "Failed to read or write reboot time: {}", msg)
             }
+            OrchestratorError::ThresholdKeyMonitoringError(msg) => {
+                write!(
+                    f,
+                    "Failed to read or write threshold key changed metric: {}",
+                    msg
+                )
+            }
             OrchestratorError::SubnetMissingError(subnet_id, registry_version) => write!(
                 f,
                 "Subnet ID {:?} does not exist in the Registry at registry version {:?}",